https://outerbounds.com/ logo
Join Slack
Powered by
Hello team, We need the ability to populate our M...
# ask-metaflow
a
Hello team, We need the ability to populate our Metaflow ec2 instances with custom tags via environment variable for cost attribution purposes. Could we please revisit this issue here: https://github.com/Netflix/metaflow/issues/1627 Or if you have any suggestions, that would be fantastic. Thanks in advance!
✅ 1
a
@acoustic-van-30942 can you provide examples of tags that would be used here?
And how are you planning on track AWS Batch usage through tags?
a
CC - @most-leather-78733 We basically have multiple sagemaker domains in an AWS account, where each domain represents a different ML project. Each domain then corresponds/maps to a specific tenant id that is used for cost attribution.
a
Because the same ec2 instance can be used by different tasks at the same time - I am not sure if AWS exposes any mechanism to attribute costs accrued by a single AWS Batch job
The question of exposing tags for AWS Batch jobs has come up before but it would be good to understand the use case so that we can ensure that any change in this direction actually solves for the use case
a
Oh that's actually a good point. Usually though, we don't keep residual ec2 instances around. The min is set to 0 in the batch compute environment
So as soon as jobs are completed, the ec2 instances are terminated
a
Even in that case - what happens if you only request a tiny share of an EC2 instance and no other workload is running at that moment on that instance - is the entire instance cost attributed to that workload?
It can surprisingly non trivial to do cost attribution - we have spent many cycles thinking and implementing it 😊
a
These are all good questions. @most-leather-78733 feel free to share your thoughts on this. How does the 
BATCH_EMIT_TAGS
work though? Does user B overwrite user A's 
METAFLOW_USER
tag?
My initial guess is that if two jobs (job A + B) share the same resources, then job B would just appends its tags to the ec2 instances of job A. (for our use case anyway)
a
No those tags are only attached to the AWS Batch job. Your AWS bill has line items for AWS Batch jobs but for EC2 instances. To understand overall cost, you need to track which job ran on which instance for what duration and then attribute a share of the EC2 instance back to that job - what/how you choose to attribute is a deep topic by itself.
The tags are not appended to the EC2 instance
There is no mechanism to do that - and the same EC2 instance can serve millions of tasks over a long enough stretch of time
a
Oh my. That does sound very non-trivial! Would you be available to sync on this sometime this week? We are getting pressured on cost attribution and it's probably one of our highest priority items right now. We can dive deeper into our particular use case and explain our requirements.
a
I am at a conference this week unfortunately so my hours are limited. Happy to chat right now, or next week or continue async.
a
Let's chat asynchronously for now.
👍 1
👍🏼 1
a
meanwhile can you help me with what these tags will look like and what the overall cardinality of these tags will likely be?
m
the requirement is to add a new tag 
adsk:team=tenant-uuid
on EC2 such that a cost dashboard team can produce reports. Multiple teams are using the same Metaflow infra stack today in a single AWS account, hence why we wanted to initially push https://github.com/Netflix/metaflow/issues/1627 where the batch decorator or metaflow/config.json can be used. It doesn't have to be exact to start.
a
do you want to attach the tag to AWS Batch job or the EC2 instance?
m
It sounds like attaching to EC2 is not feasible, so I'm willing to work with Batch job tags and then get our cost teams to improve their AWS cost filters.
a
AWS Batch Jobs are not a line item in AWS bill. What you get billed for is EC2 spend so there is some manual processing of bills that you will have to do to attribute EC2 spend to individual batch jobs. Given that Metaflow already generates some tags (metaflow.flow_name, metaflow.username etc.) that are attached to AWS Batch jobs, can you confirm that if any other tags are published, those will definitely solve for your issue? One way to figure that out will be to use say 
metaflow.flow_name
tag and see if you are able to generate flow level cost attribution reports. Then it becomes a bit easier to support the proposal of introducing custom tags, otherwise if there is nothing to be gained, we would rather avoid the overhead of maintaining a feature of limited utility.
d
Hey hey, jumping on this thread as this has been something that our teams are currently finding workarounds as well. As an enterprise company, we have teams who do the cost reporting for all our AWS resources that we review monthly. The way that get's used is we tag with product, service, etc and those show up in the right org. I thought AWS released a feature for cost attribution for ecs here. Once you tick this, AWS exposes ECS tasks as a line item in the AWS bill. Once we have the tags exposed on the batch task, we should be able to attribute the bills properly. With that usability in mind, can we revisit this issue to be prioritised. My teams would be happy to contribute to this too if needed because this is something we need
thankyou 2
👍 1
a
As workaround until we support arbitrary cost allocation AWS tags, can I have a Lambda that subscribe to events that get emit in change of steps? In the Architecture, runtime stream events to a Metadata so am I right to say that I can use a metadata implementation to capture those events and to manually tag the AWS resources accordingly?
4 Views
Open in Slack
PreviousNext
Powered by