https://outerbounds.com/ logo
Join Slack
Powered by
Does the Metaflow S3 client support async puts? Th...
# ask-metaflow
c
Does the Metaflow S3 client support async puts? The scenario is a step performing expensive GPU training, and want to minimize GPU idle time and costs.
v
great question! Async puts or gets? Python async isn't a great match for CPU-intensive work but if you want to parallelize data loading (or uploading) alongside CPU/GPU-intensive training, it should be possible
are you thinking of implementing a data loader for PyTorch or some other framework or what's the use case you have in mind?
c
Short term quick solution would be to see if S3 puts can be async (for the checkpointing scenario).
We’re not looking into implementing a data loader for PyTorch yet… I’ve found Lightning to come with a lot of amazing features such as AsyncCheckpointIO, but not all training is in Lightning!
v
boto
, which is required to talk to S3 is not async-compatible and won't be in the near future. There's an unofficial version that supports async but it's not without issues even if it was async-compatible, a safer approach is to isolate it in a separate process which can truly run in parallel, which is mostly how 
metaflow.S3
works today
let me create an example for you to give an idea. I will follow up later today or tomorrow
thanks ty 1
Open in Slack
PreviousNext
Powered by