Has anyone seen this error when trying to resume from a specific step

python3 seg_infer.py --environment=conda resume tile_inference_join  --origin-run-id 1455

Metaflow 2.12.10+netflix-ext(1.2.1) executing SegInferenceFlow for user:mikebentleymills
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
2024-08-08 13:11:26.222     All packages already cached in s3.
2024-08-08 13:11:26.222     All environments already cached in s3.
2024-08-08 13:11:26.646 Gathering required information to resume run (this may take a bit of time)...
2024-08-08 13:11:31.587 Workflow starting (run-id 1960):
2024-08-08 13:11:32.721 Start cloning original run: SegInferenceFlow/1455
2024-08-08 13:11:33.067 [1960/tile_inference/26543] Cloning results of a previously run task 1455/tile_inference/18951
2024-08-08 13:11:33.078 [1960/end/26544] Cloning results of a previously run task 1455/end/18957
2024-08-08 13:11:33.105 [1960/slides_to_tilesets/26545] Cloning results of a previously run task 1455/slides_to_tilesets/18946
2024-08-08 13:11:33.162 [1960/tile_inference_process/26547] Cloning results of a previously run task 1455/tile_inference_process/18948
2024-08-08 13:11:33.199 [1960/tiles_to_slide_join/26548] Cloning results of a previously run task 1455/tiles_to_slide_join/18947
2024-08-08 13:11:33.203 [1960/tile_inference/26550] Cloning results of a previously run task 1455/tile_inference/18952
2024-08-08 13:11:33.210 [1960/start/26549] Cloning results of a previously run task 1455/start/18945
2024-08-08 13:11:33.214 [1960/tile_inference/26551] Cloning results of a previously run task 1455/tile_inference/18955
2024-08-08 13:11:33.221 [1960/tile_inference/26552] Cloning results of a previously run task 1455/tile_inference/18954
2024-08-08 13:11:33.232 [1960/tile_inference/26553] Cloning results of a previously run task 1455/tile_inference/18953
2024-08-08 13:11:33.237 [1960/tile_inference/26554] Cloning results of a previously run task 1455/tile_inference/18949
2024-08-08 13:11:33.263 [1960/tile_inference/26555] Cloning results of a previously run task 1455/tile_inference/18950
2024-08-08 13:11:33.333 Cloning task from 1455/tile_inference_join/18956 failed with error: No completed attempts of the task was found for task 'SegInferenceFlow/1960/tile_inference_join/26546'
2024-08-08 13:11:34.566 Cloning original run is done
2024-08-08 13:11:34.822 Workflow failed.
2024-08-08 13:11:34.822 Terminating 0 active tasks...
2024-08-08 13:11:34.822 Flushing logs...
    Data store error:
    No completed attempts of the task was found for task 'SegInferenceFlow/1960/tile_inference_join/26546'

This seems to have started when I updated to 2.12.10 but not sure. It happens even for versions of the command that had previously worked.

we did change some of the resume stuff in 2.12.9.

The previous run (1455 I'm resuming from) of tile_inference_join completed successfully in that case.

got it