Hi everyone, may I know if it's possible to stream...
# ask-metaflow
a
Hi everyone, may I know if it's possible to stream the data artifacts using the API? I am aware that we can obtain the artifacts directly using the API. However, I wish to stream it and I'm wondering if the API supports this? If not, I can try streaming the s3 file instead, but I'm not sure how to deserialise the file back into it's original object. Any help would be much appreciated, thanks!
d
This is not yet implemented but we have a feature in progress which will enable this. No ETA yet but hopefully H1.
a
Thanks for the reply! In that case I might try to stream the pickled files from s3, and do the unpickling on the client end. May I know if the unpickling tools are available through the API?
a
@adamant-psychiatrist-48924 what is the use case for streaming the object?
a
I have a service that I intend to stream the artifacts back to the client upon request. Since the artifacts size could be large, we plan to stream it instead of returning the whole artifact at once from the service.