Hi all, I wanted to raise a new (minor) feature for discussion that I think could have a lot of utility (especially for Metaflow operators/admins). Here's a link to the issue, but in a nutshell: what if Metaflow took advantage of the ability for Batch to propagate tags from jobs to the actual ECS tasks in order to inject Metaflow metadata like run_id, flow_name, etc, as resource tags on the ECS tasks that actually run the job? This would be beneficial for monitoring systems, and also for cost attribution tools.

Makes sense. I have opened a PR to address this - https://github.com/Netflix/metaflow/pull/631

That's a good point. When I proposed it

namespace

originally, I guess I was thinking of the production token for SFN executions.

(since in our setup. we set that to the branch name and frequently use it as a results namespace)

I can add branch name directly as well.

The release is in flight - you should have the package available within the next hour - https://github.com/Netflix/metaflow/actions/runs/1080541216

I'm not sure that this behavior will actually provide per-job cost allocation when using EC2 (vs. Fargate). We (AWS Batch) need to do work here to make EC2 cost allocation better.

@worried-machine-92008 Yes, this unfortunately a sore point. Having the tags lets you (if you enable the tags as cost allocation tags) get tagged metering data in Cost Explorer (vCPU Hours and GB Hours for cpu and memory, respectively); there's still a lot of machinations required by the end user to convert that back into an accurate-ish chargeback based on the instances utilized.

We are working on it, is all i can say at this point 🙂 Also Fargate can do per-job cost tracking but to my knowledge this is still not supported in MetaFlow?

I believe Fargate compute environments actually are supported by Metaflow, though it won't work well in our case since the flows we're most interested in tracking cost for have resource requirements well in excess of what Fargate allows 😞