Hi everyone, may I know if it's possible to stream the data artifacts using the API? I am aware that we can obtain the artifacts directly using the API. However, I wish to stream it and I'm wondering if the API supports this? If not, I can try streaming the s3 file instead, but I'm not sure how to deserialise the file back into it's original object. Any help would be much appreciated, thanks!
d
dry-beach-38304
02/17/2025, 5:58 PM
This is not yet implemented but we have a feature in progress which will enable this. No ETA yet but hopefully H1.
a
adamant-psychiatrist-48924
02/18/2025, 2:06 AM
Thanks for the reply! In that case I might try to stream the pickled files from s3, and do the unpickling on the client end. May I know if the unpickling tools are available through the API?
a
ancient-application-36103
02/18/2025, 4:32 PM
@adamant-psychiatrist-48924 what is the use case for streaming the object?
a
adamant-psychiatrist-48924
02/19/2025, 1:48 AM
I have a service that I intend to stream the artifacts back to the client upon request. Since the artifacts size could be large, we plan to stream it instead of returning the whole artifact at once from the service.