Hi, we would like to add sumologic to meteflow. We added it to the ecs task definition but only Http calls were logged. What is the right way to implement it?

thanks @rough-sundown-12921

Got it working by setting the OTEL_ENDPOINT with the environment decorator. But Metaflow returned this error :

"Overriding of current TracerProvider is not allowed"

Is there another way to change the tracerprovider for aws batch?

@bulky-afternoon-92433 any ideas here?

first for some context on the internals: When setting

OTEL_ENDPOINT

it will enable opentelemetry for Metaflow internals. This includes a bunch of predefined spans. The error you see might stem from trying to redefine a span when inside one. Are you setting up something custom for the telemetry on the flow/container side as well, or relying completely on the Metaflow provided setup?

Thanks @bulky-afternoon-92433. Not a custom telemetry setting. I just set OTEL_ENDPOINT with the @environment decorator before the first task. Ideally we would like to capture batch related telemetry as well. What is the proper way to set OTEL_ENDPOINT? Thank you.

ah, there seems to be an issue here as well. One of the intended ways is that you simply set the

METAFLOW_OTEL_ENDPOINT

in your metaflow profile and it carries over when deploying/running remotely. I just noticed that this does not seem to be working for Batch/SFN. Will be pushing a fix for it. In the mean time, as an alternative approach you could look into a way to provide the environment variable for all jobs on Batch. I know that at least on the Kubernetes side some teams are opting to set the otel configuration as a default for all containers instead of configuring through Metaflow profiles.

@bulky-afternoon-92433, Thanks! We tested this yesterday and it worked.