steep-army-18888
02/23/2023, 5:46 PMmetaflow.user
and value $METAFLOW_USER
?
a. I know that I can select an AWS Job queue through the @batch
decorator, that this Job queue points to AWS Compute environments, and that these Compute environments can in turn propagate tags to the corresponding EC2 instances. This way, the tags are however fixed based on the Job Queue used, and do not change dynamically based on e.g. METAFLOW_USER
. I also hesitate to use the hacky solution of setting up individual Job queues/Compute environments for individual users, as I seem to remember there is a relatively low limit on the maximum number of Job queses/Compute environments allowed.
b. I also know that I can set METAFLOW_BATCH_EMIT_TAGS
to True
, and that this propagates e.g. the tag metaflow.user
to the corresponding AWS Batch Job, but these tags do not get further propagated to the corresponding EC2 instances.
2. If the answer to the above question is no, is there a good way to identify e.g. the METAFLOW_USER
of a Metaflow run from a given AWS EC2 instance id which is spawned as part of that Metaflow run?
a. Running aws ec2 describe-instances --instance-ids <instance-id>
, I find no information which would allow for going EC2 instance
--> Batch job
(which would in turn allow to find the corresponding metaflow.run_id
and metaflow.user
), but maybe I am missing something🙂
All help very much appreciated! Thanks in advance!🙂