Has anybody experienced this metaflow-ray error du...
# ask-metaflowi
important-london-94970
02/10/2025, 7:41 PMHas anybody experienced this metaflow-ray error during training? It's entirely possible I am running out of memory and this is a side-effect but I'm not sure
Copy code
File "/metaflow/metaflow_extensions/ray/plugins/status_notifier.py", line 106, in wait_for_task_completion
if time.time() - status.timestamp > heartbeat_timeout:
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'✅ 1
s
square-wire-39606
02/10/2025, 7:55 PM@hallowed-glass-14538 any thoughts here?
i
important-london-94970
02/10/2025, 8:00 PMadding
@retryh
hallowed-glass-14538
02/10/2025, 8:25 PMhey! can you share more context on when this starts happening ?
s
square-wire-39606
02/10/2025, 8:26 PMreading through the code - status.timestamp can be
Nonei
important-london-94970
02/10/2025, 8:27 PMIt is happening sporadically while training. Sometimes it happens when my data is downloading, sometimes it happens in the middle of a training epoch.
h
hallowed-glass-14538
02/10/2025, 8:30 PMfew more questions for more context:
• are you running this on K8s / aws-batch ?
• what is the metaflow datastore you are running this on?
• what is the size of
num_paralleli
important-london-94970
02/10/2025, 8:32 PMRunning on k8s, on GCP so using the “gs” datastore, and num_parallel==3
h
hallowed-glass-14538
02/10/2025, 8:37 PMlemme take a look and reach out! thanks for the bug report
hallowed-glass-14538
02/11/2025, 1:42 AMhey! so we have cut a patch release (0.1.4) that handles the failure case for this condition gracefully. The patch also adds an environment variable for debugging whats happening behind the scenes. you can set the
@environment(vars={"METAFLOW_RAY_DEBUG_MODE":"true"})@step@metaflow_rayi
important-london-94970
02/11/2025, 1:43 PMUpgraded to
metaflow-ray==0.1.4 Copy code
[@metaflow_ray] Task 0 status: running with timestamp 1739281146.9187553
23
08:39:12
[@metaflow_ray] Task 0 status: unreachable with timestamp None
24
08:39:12
[@metaflow_ray] Task 0 unreachable
25
08:39:12
[@metaflow_ray] Task 0 still unreachable after 0.0 seconds
26
08:39:13
[@metaflow_ray] Task 0 status: running with timestamp 1739281152.0030744
27
08:39:14
[@metaflow_ray] Task 0 status: running with timestamp 1739281152.0030744important-london-94970
02/11/2025, 1:43 PMThanks for the quick support!