Hey, I’m running into the recurring problem during triggering flows on AWS Step Functions.

TypeError: rename: src should be string, bytes or os.PathLike, not NoneType

is error that happens during bootstrapping conda environment. Deleting local folder

.metaflow

and deploying manually the flow again resolves the issue for a couple of days. But afterwards, the execution throws the same error again. Any ideas what can cause this behaviour?

This happened to me when I deleted remote metaflow store on

S3

, but didn't delete the local store

.metaflow

directory. Did you also delete the remote

S3

store?

What exactly is the remote S3 store? Is it the location of env variable:

METAFLOW_DATASTORE_SYSROOT_S3

?

@steep-traffic-55869 Yes. My terminology might not be the best, but generally it's the place where metaflow stores its files. It can be either locally (if you don't configure AWS cloud), or in some cloud (like S3). When you deploy with Step, then it uses S3. You can explore it on the path

METAFLOW_DATASTORE_SYSROOT_S3

, it contains the flows and info about them, the artifacts.

Do you think that when re-deploying flow there can be some interference between runs?

@steep-traffic-55869 I don't know. I managed to replicate the error like this: • Run with aws batch:

python flow.py --environment conda --datastore s3 run --with batch

• Delete S3 metaflow folder. • Run again:

python flow.py --environment conda --datastore s3 run --with batch

- this throws the error you mentioned. The error is solved by deleting

.metaflow

folder. I hope someone more knowledgeable has a better insight. Can you replicate it deterministically?

And after you deleted

.metaflow

, you haven’t got this error anymore?

Yes. The problem is probably hidden in that

.metaflow/conda.dependencies

does not correspond to the conda packages cached on S3 anymore because they were deleted.