https://outerbounds.com/ logo
Join Slack
Powered by
We've started to run into an intermittent issue wh...
# ask-metaflow
q
We've started to run into an intermittent issue where Metaflow's S3 client (running on Kubernetes/Argo) hangs indefinitely when persisting artifacts to S3. The issue first appeared when we upgraded MLflow (not Metaflow) from 2 to 3, but the call stack on the hung pods is always in Metaflow's S3 client. Has anyone seen anything like this before? Or does anyone have any thoughts on why this would happen or how to prevent it?
✅ 1
v
that’s odd. They get stuck forever? Retries can take minutes 
@timeout
and 
@retry
help with intermittently hanging tasks but of course it’d be good to identify and fix the root cause
would it be easy to try with the older versions (since you said MLFlow upgrade coincides with the issue) to confirm if its related to the upgrade?
q
yeah, I mean, we obviously don't wait forever, but we've seen them get stuck persisting artifacts for hours or even days
we seem to have fixed it for now by setting 
MLFLOW_DISABLE_TELEMETRY=true
which turns off the sharing of anonymized usage data with MLflow devs, but I don't understand how that would break Metaflow's S3 client
v
could be resource contention, unless they do really aggressive instrumentation, in which case it could break something deeper
👍 1
good to hear that the issue got fixed
13 Views
Open in Slack
PreviousNext
Powered by