Hello, we have recently seen an increase in runs failing due to the following error:

File "/venv/lib/python3.10/site-packages/metaflow/datastore/task_datastore.py", line 364, in load_artifacts
    for (key, blob) in self._ca_store.load_blobs(to_load.keys()):
  File "/venv/lib/python3.10/site-packages/metaflow/datastore/content_addressed_store.py", line 140, in load_blobs
    with open(file_path, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

These happen sporadically and I can't deterministically reproduce the error, restarting the run will sometimes fix it. Is there a way to enable debug logging for metaflow? I'm trying to see what blob it's trying to load and if a parent step is not serializing the artifact correctly?

peculiar. This is with S3 datastore, right?

Hi @victorious-lawyer-58417, yes we're running in ECS and use S3 as the datastore

change

run

here to point at the run that failed and execute this snippet that tries to access all artifacts

from metaflow import Flow
run = Flow('HelloFlow').latest_run

for step in run:
    for task in step:
        for artifact in task:
            try:
                artifact.data
                print(f'Task {task.pathspec}: Artifact {artifact.id} is ok')
            except:
                print(f'Task {task.pathspec}: Artifact {artifact.id} failed')

I ran the snippet you mentioned and it does fail to load a few artifacts from a previous step. In the meantime, I can try re-running the flow at the previous step

@User - can we do a quick screen share to get to the bottom of this issue?

Thanks @ancient-application-36103 let me check something at our end and get back to you. We have a custom extension that checkpoints and copies artifacts when we resume failed flows at a particular step. I'll get back to you once I verify if the problem happens with the copying there

for sure!

Thanks for checking in @ancient-application-36103 I was just chatting with @User about this, who figured out it might be because of expiring artifacts (we delete certain artifacts for compliance reasons).

return_missing

is set to

True

which ends up swallowing the exception and returning a

None

path when the file doesn't exist https://github.com/Netflix/metaflow/blob/2.7.19/metaflow/plugins/datastores/s3_storage.py#L117-L143

good to hear that you figured it out. A proper support for artifact lifecycles / garbage collection is on the roadmap, so eventually there will be a better solution for you which addresses the issue Meanwhile one option you can look into (if you haven't already), is S3 Intelligent Tiering which moves artifacts automatically to other storage tiers if they are not being used. This way you can expire artifacts if they haven't been used in N days, which should much reduce the likelihood for this issue popping up.

Thanks Ville, I'll take a look at it. Also looking forward to the artifact lifecycle work, it would be useful to make error handling more configurable so users can decide what exceptions to swallow / raise

interesting. Can you elaborate that point re: exception handling?

return_missing

is a variable that controls whether an exception is bubbled up the call stack or not, ideally this behavior is something that the user can choose to enable/disable. However the

load_bytes

method doesn't allow us to configure this and always sets it to

True

I'm not sure if this pattern of exception handling exists in other parts of the metaflow codebase, but if it does then perhaps it's useful to have a shared global config that disables all exception swallowing for errors raised in external systems

interesting, good feedback

I was just chatting with @User about this, who figured out it might be because of expiring artifacts (we delete certain artifacts for compliance reasons).

@User, which artifacts do you delete? We are hitting this same error at Zillow. However we have an s3 expiry policy of data older than 28 days, which should not be hit in this case (unless I’m wrong).

Internal error
Traceback (most recent call last):
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/task.py", line 548, in run_step
    self._exec_step_function(step_func)
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/task.py", line 53, in _exec_step_function
    step_function()
  File "/opt/zillow/user_embedding_generation_pipeline/user_embedding_generation_flow.py", line 421, in publish_guids_to_odp
    env=self.env
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/task.py", line 87, in property_setter
    v = param.load_parameter(parameter_ds[var])
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/datastore/task_datastore.py", line 43, in method
    return f(self, args, kwargs)
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/datastore/task_datastore.py", line 826, in __getitem__
    _, obj = next(self.load_artifacts([name]))
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/datastore/task_datastore.py", line 377, in load_artifacts
    for (key, blob) in self._ca_store.load_blobs(to_load.keys()):
  File "/opt/zillow/.venv/lib/python3.9/site-packages/metaflow/datastore/content_addressed_store.py", line 140, in load_blobs
    with open(file_path, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not NoneType

I think this is an edge case that happens at step transitions. We have observed that sometimes there's a lot of delay between one step finishing and the next one being started. For ex: it's possible for an artifact to be near it's expiry time at step A and hence not get written out, but then expire before it's loaded in step B, resulting in this error message.