The human-centric platform for production ML & AI

Outerbounds

Hi all,

Curious about something - if I enable <https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html|checkpoints> in Sagemaker training, and I prematurely stop the training tasks, would Metaflow's `resume` command be able to carry on the training task and load the model from the previous saved checkpoint? Or would it start a new sagemaker training instance? Thanks in advance!

I don’t know if it works out of the box but if you persist Sagemaker checkpoints in S3, it could/should work

Yup I think the checkpoints are persisted to s3. I'll give it a try and let you know what I find. Thanks Ville :slightly_smiling_face:

great! I’m really curious to hear if you do (not) get it working :slightly_smiling_face: 

we have tested checkpointing with vanilla PyTorch and it can work fine - it’s actually a super useful feature when combined with `resume` 