Hi, does Metaflow have any plugins to run Sagemaker Training jobs?

no special integration but you can use their Python API to launch Sagemaker Training Jobs from Metaflow steps

Ah yeah that'd work, but I was thinking it'd be cool if we had a

@sagemaker

decorator similar to the

@batch

one

right, interesting. What features of SM Training Jobs you find most exciting over plain

@batch

?

There are a few goodies like hyperparameter tuning, distributed training, automated publishing of training metrics. I've been told that SM is under active development while Batch has kinda stopped

yep, Batch is easy to get started with but quite limited, which is why we have support for EKS (Kubernetes) now. One gotcha with Sagemaker is that it’s about 20% more expensive than the same instances outside Sagemaker, so the goodies come with a (high) cost.

we have deep pockets 🙂

haha, you can certainly use it via the Python API today. You could even wrap the calls in a custom

@sagemaker

decorator (or a convenience library) of your own

i'm poking around the implementation of the

@batch

decorator, and it seems like i can follow that to implement a

@sagemaker

decorator if I replace the

batch.launch_job()

with a

sagemaker.launch_job()

? am i on the right track? https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/aws/batch/batch_cli.py#L277

yep! That's the right track. There are probably many details and differences but that's a key piece. See an overview here

I see in the Batch

create_job

code it has

command

and

image

. How does it actually override the

CMD

in the Dockerfile to run the given command, which seems to be

CMD ["python3"]

by default in the image?