How are people handling logging with Metaflow? We would like to capture fully structured logs into Loki, which requires them to be in JSON format, which in turn makes them unreadable in the Metaflow UI. We ended up writing our own logging library that has 2 outputs: One that pushes JSON directly to Loki, skpping the stdout/stderr, and one that logs to stdout/stderr so stuff gets surfaced in the Metaflow UI. Our only problem with this approach is that now we’re getting double logs into Loki because we are unable to prevent Metaflow from logging into CloudWatch (all CloudWatch logs automatically get pushed to Loki in our case). Does anyone know how to have stdout/stderr only go into the Metaflow UI, but not into CloudWatch?

Perhaps you could explicitly deny logs from being created + pushed to Cloudwatch via IAM policies? I’m not sure about the implications about doing that though.

Yes, we tried a

DENY

policy to do that, however, this causes our flows to fail

Was having this issue earlier today. This is a issue I came across, though I haven't tried the workaround myself or verified if it would fix the double logging issue https://github.com/Netflix/metaflow/issues/180#issuecomment-653547161