Hi there! First of all, thanks for a fantastic pro...
# ask-metaflow
s
Hi there! First of all, thanks for a fantastic product❤️metaflow I’ve got two questions, the answers to which I could not find by searching through the chat-to-date: 1. Is there a way for Metaflow to tag AWS EC2 instances used during a run based on information from that individual run, e.g. with key
metaflow.user
and value
$METAFLOW_USER
? a. I know that I can select an AWS Job queue through the
@batch
decorator, that this Job queue points to AWS Compute environments, and that these Compute environments can in turn propagate tags to the corresponding EC2 instances. This way, the tags are however fixed based on the Job Queue used, and do not change dynamically based on e.g.
METAFLOW_USER
. I also hesitate to use the hacky solution of setting up individual Job queues/Compute environments for individual users, as I seem to remember there is a relatively low limit on the maximum number of Job queses/Compute environments allowed. b. I also know that I can set
METAFLOW_BATCH_EMIT_TAGS
to
True
, and that this propagates e.g. the tag
metaflow.user
to the corresponding AWS Batch Job, but these tags do not get further propagated to the corresponding EC2 instances. 2. If the answer to the above question is no, is there a good way to identify e.g. the
METAFLOW_USER
of a Metaflow run from a given AWS EC2 instance id which is spawned as part of that Metaflow run? a. Running
aws ec2 describe-instances --instance-ids <instance-id>
, I find no information which would allow for going
EC2 instance
-->
Batch job
(which would in turn allow to find the corresponding
metaflow.run_id
and
metaflow.user
), but maybe I am missing something🙂 All help very much appreciated! Thanks in advance!🙂
1