👋 Hey Folks! Need some advice on the best way to get setup for training models on multiple-gpu (not multi node yet). Here are the use cases we are currently targeting: • Run several experiments and track metrics on WandB • We’d like to monitor the progress of the jobs on the Metaflow UI • Develop using notebooks and submit longer running training jobs to the multi GPU node. • [Nice to have] Ideally, we’d want to seamlessly switch between AWS and GCP We looked at this https://outerbounds.com/engineering/deployment/aws-managed/cloudformation/ Curious how much ops overhead we would have for deploying all these services and if there is a simpler stack that we can deploy to achieve these use cases.

@sparse-dress-4861 ops overhead for the cloudformation stack at somewhat reasonable scale should be very minimal. if you are looking for the minimum possible overhead, at the expense of losing some metaflow functionality - you can skip deploying the metadata service and the UI - just the S3 bucket and AWS Batch compute environment (that scales down to 0) should have as close to 0 overhead as is feasible in the cloud. metaflow will store all the tracking information in your local filesystem - so your results wouldn't be shareable across users. cards would work and can perhaps stand-in as a very lightweight UI. the next frontier would be to deploy the metadata service to get tracking capabilities across users. an enhancement on top of that would be to deploy the ui service.

hey @ancient-application-36103! Thanks!

so your results wouldn’t be shareable across users.

ah, we definitely would like to make it shareable across the team. I am assuming it is not too much of an overhead for the metadata service and the UI ? The compute footprint for these services is also fairly low (IIRC)?

correct!

awesome! We ll give this a try and get back here if we run into any issues

Yep! You would want to list the instances from that you would like to use instead of the defaults that the template setup

perfect!

It’s totally up to you - you can configure your batch compute environment as you wish

I see. would you recommend getting dedicated instances in that case? Also, is GCP better for such use cases?

Depending on the instance type, on-demand could work well

we ll try with the on demand. Spot would have been awesome for our wallet though 😃

You could create multiple compute environments prioritising spot instances ahead of on-demand when you attach them to the job queue

I think on demand is okay for now. But we were to add in this policy of prioritizing spot over on-demand, how can we do that in this CloudFormation template?

The template is a starter template, it doesn’t have all the bells and whistles - it might be easier to set the compute environment manually on AWS Batch’s end

got it. we ll use this template to set up the stack and the customize batch as needed 👍

hey @sparse-dress-4861! I only ran metaflow-torchrun locally but we plan on kicking off some trail runs on argo + k8 in the next two weeks (if things go according to plan). ill will let you know how that goes from the looks of the error though it seems like your batch nodes might not have internet connection, probably a security setting from the template. IIRC when metaflow was open-sourced the free sandbox environment also had outside connections disabled one possible workaround is to just bake a docker image with all your required dependencies and push the image to your internal ECR and use that as the base/default image for your metaflow tasks

thanks @stale-eve-11739! let me know how your trials go. Makes sense on the lack of an internet connection for security reasons. We ll look into baking everything in to a docker image but that seems a bit inconvenient. We want some flexibility in terms of being able to install packages not in the image for different experiments.

Yeah that would only be a temporary solution just to see if that resolves your problem. You should update the permissions attached to whatever role you are using

Yes been struggling with the permissions bit for the role that we use for this stack. AWS doesn’t make this easy at all. 😕

Hey @sparse-dress-4861 happy to help you debug this on a call. We should be able to use the base CUDA or PyTorch images for Metaflow tasks while allowing you to flexibly layer packages on top. Would you like to schedule a time for tomorrow or Thursday to pair program?

@crooked-jordan-29960 that would be awesome! Could you share your calendar? I also DM’ed you my calendar for convenience