Hi all, following up on a question I asked in the ...
# dev-metaflowc
cool-father-45885
08/12/2021, 6:30 PMHi all, following up on a question I asked in the Metaflow gitter.
Copy code
Hello, I am interested in overriding the default log driver for the Batch job definition so that we can ingest logs from the Metaflow Batch jobs running in ECS into Datadog as opposed to Cloudwatch logs. How would I go about overriding the default job definition which Metaflow creates on my behalf when sending tasks to Batch job queues?✅ 1
cool-father-45885
08/12/2021, 6:30 PMCurrently, we are using the default logdriver which is awslogs
cool-father-45885
08/12/2021, 6:31 PMBut we are looking for the best way to ingest logs into Datadog from the individual Metaflow tasks orchestrated by Batch
cool-father-45885
08/12/2021, 6:31 PMcc: @ancient-application-36103
s
square-wire-39606
08/12/2021, 6:32 PMLet me do some digging and get back to you in a few minutes?
❤️ 1
c
cool-father-45885
08/12/2021, 6:32 PMSounds great, thank you @ancient-application-36103
s
square-wire-39606
08/12/2021, 6:33 PMA simple way would be to set a bridge between cloudwatch logs and datadog to forward all Metaflow logs to datadog.
square-wire-39606
08/12/2021, 6:35 PMAll logs for Metaflow AWS Batch jobs go into
/aws/batch/jobmetaflow_square-wire-39606
08/12/2021, 6:39 PMUsing the client API you can get access to the cloudwatch log group and stream name for each Metaflow task as well -
Copy code
Run('MyFlow/42')['start'].task.metadata_dict['aws-batch-awslogs-stream'] Copy code
Run('MyFlow/4')['start'].task.metadata_dict['aws-batch-awslogs-group']square-wire-39606
08/12/2021, 6:41 PMIt seems the only way to forward Cloudwatch Logs to Datadog is via a
forwardersquare-wire-39606
08/12/2021, 6:42 PMAlso, Metaflow currently stores all the logs in S3 as well, but configuring the Datadog
forwardersquare-wire-39606
08/12/2021, 6:42 PM@cool-father-45885 Would something like this work or were you hoping for a more native integration where Metaflow forwards the logs to Datadog?
square-wire-39606
08/12/2021, 6:47 PMIt should also be possible to run a custom docker image with a datadog agent that automatically forwards all the logs to datadog - I haven't tried it myself though.
c
cool-father-45885
08/12/2021, 6:49 PMA more native integration would certainly be ideal. Currently, it sounds like the log forwarding solution is our best option requiring the least amount of added maintenance.
👍 1
r
rapid-garden-26667
08/12/2021, 6:49 PMWe were hoping for something more native but the lambda solution will work. Thank you @square-wire-39606!
Follow up question, does Metaflow have a mechanism to add tags that will propagate to the underlying components of aws batch?
s
square-wire-39606
08/12/2021, 6:52 PMVery timely question! In our recent release, we have started to tag the underlying instances with metadata about the AWS Batch job - https://github.com/Netflix/metaflow/releases/tag/2.3.3
🙌 2
square-wire-39606
08/12/2021, 6:53 PMWould this suffice or were you interested in also forwarding custom user tags?
r
rapid-garden-26667
08/12/2021, 6:55 PMThis looks great, we will try and get this setup. Thank you for all the help!
👍 1
s
square-wire-39606
08/12/2021, 6:55 PMI have also opened a Github issue - https://github.com/Netflix/metaflow/issues/643 to track Datadog integration. At minimum, we should document the lambda bridge pattern.
👍 2
square-wire-39606
08/12/2021, 6:56 PM@rapid-garden-26667 Great! Let us know any other feedback/questions you may have!
c
cool-father-45885
08/12/2021, 6:56 PMThank you very much @ancient-application-36103
🙇 1
p
plain-baker-2104
10/26/2021, 2:30 PM@rapid-garden-26667 what were you hoping to tag? the batch jobs or the ec2 instances (which have costs). tagging batch jobs will not show up in cost explorer...(at least with ec2 instances, maybe fargate)