Hi there! First of all, thanks for a fantastic product❤️metaflow I’ve got two questions, the answers to which I could not find by searching through the chat-to-date: 1. Is there a way for Metaflow to tag AWS EC2 instances used during a run based on information from that individual run, e.g. with key

metaflow.user

and value

$METAFLOW_USER

? a. I know that I can select an AWS Job queue through the

@batch

decorator, that this Job queue points to AWS Compute environments, and that these Compute environments can in turn propagate tags to the corresponding EC2 instances. This way, the tags are however fixed based on the Job Queue used, and do not change dynamically based on e.g.

METAFLOW_USER

. I also hesitate to use the hacky solution of setting up individual Job queues/Compute environments for individual users, as I seem to remember there is a relatively low limit on the maximum number of Job queses/Compute environments allowed. b. I also know that I can set

METAFLOW_BATCH_EMIT_TAGS

to

True

, and that this propagates e.g. the tag

metaflow.user

to the corresponding AWS Batch Job, but these tags do not get further propagated to the corresponding EC2 instances. 2. If the answer to the above question is no, is there a good way to identify e.g. the

METAFLOW_USER

of a Metaflow run from a given AWS EC2 instance id which is spawned as part of that Metaflow run? a. Running

aws ec2 describe-instances --instance-ids <instance-id>

, I find no information which would allow for going

EC2 instance

-->

Batch job

(which would in turn allow to find the corresponding

metaflow.run_id

and

metaflow.user

), but maybe I am missing something🙂 All help very much appreciated! Thanks in advance!🙂

@steep-army-18888 What is the use case to tag ec2 instances from AWS Batch? A single ec2 instance could be used my multiple jobs (often from different users) during it's lifetime.

Thanks for the lightning-speed reply @square-wire-39606 and apologies for moving at a snail’s pace on my side 🐌 1. Use case would be monitoring and cost overview. Currently, we’re a team of ~25 training ML models interactively on individual EC2 instances. These are automatically tagged with the user name, and so I can see at any given time who uses which resources and we’ve set up an AWS lambda function which sends everyone detailed weekly cost reports of the resources they’ve been using. As we’re now moving to train using Metaflow (on AWS) instead, I would ideally like to be able to have the same level of usage overview and cost insights (on a user level). But yeah, great point that the EC2 instances used as part of the Batch jobs might be used by several different jobs/users during their lifetime, and so this level of granularity perhaps simply isn’t possible.

this is great! yes - in metadata_dict we can potentially start emitting a field that identifies the ec2 instance if the instance metadata is accessible. but it seems that you have able to hit the batch apis directly to collect all the necessary information.

Hi Sam and @square-wire-39606 thanks for the discussion! I was wondering do we have some updates about this topic? My team is faced with similar need: we are going to create a Cloudwatch dashboard to monitor the utilization of GPU launched by

@batch

, and is having problem in linking the specific GPU to the user's batch job (since multiple users may launch batch job in the same queue)

@fast-dog-88545 if you are only interested in monitoring the GPU utilization rates, you can use this metaflow card instead - https://github.com/outerbounds/monitorbench/blob/main/gpu_profile.py

Thank you @square-wire-39606 , yes metaflow card is very useful and my team use it a lot. But we also want a dashboard to monitor dynamic real-time GPU utilization during training. IIRC card can only be seen after that step finish right?

yes - we are also looking into making these cards realtime