Hey Metaflow team, we're PoCing Metaflow at my company and we're running into the following issue when running workflows on azure kubernetes using

Metaflow 2.10.8

and the

pypi

environment:

2023-12-13 13:34:50.754 [93/start/508 (pid 13121)] Kubernetes error:
2023-12-13 13:34:50.755 [93/start/508 (pid 13121)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2023-12-13 13:34:50.755 [93/start/508 (pid 13121)]
2023-12-13 13:34:47.298 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "<frozen runpy>", line 198, in _run_module_as_main
2023-12-13 13:34:47.298 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "<frozen runpy>", line 88, in _run_code
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "/metaflow/metaflow/plugins/pypi/bootstrap.py", line 119, in <module>
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]     shutil.move(tmpfile, dest)
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "/usr/local/lib/python3.11/shutil.py", line 825, in move
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]     os.rename(src, real_dst)
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg] TypeError: rename: src should be string, bytes or os.PathLike, not NoneType

We've traced the error and found that in the

conda.manifest

file the path for the

cryptography

package is the following:

{
          "url": "<https://files.pythonhosted.org/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>",
          "path": "<http://files.pythonhosted.org/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl|files.pythonhosted.org/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
        },

but in our Azure blob root container, this path does not exist (note the lack of the

14

sub dir ) We've tried deleting the

.metaflow

directory locally as well as deleting everything under

<http://files.pythonhosted.org/packages|files.pythonhosted.org/packages>

in our blob container but we still get the same error. Any ideas/thoughts?

looking...

You can try the “bleeding edge” decorators as well to unblock and continue PoCing until this is resolved. See here: https://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators. For this case, the stock decorators should work fine but both versions use slightly different ways of caching files so it’s possible that the stock don’t handle your corner case and the others do (but also possible it also doesn’t work). Either way, if you try it out, let us know.

Thanks will give that a try! In my venv I can also confirm the version of cryptography is 41.0.7 which is the same as the

conda.manifest

can you help me with the arguments to the

@pypi

and

@conda

decorator in your flow as well as the version of python?

Sure thing! In my venv the python version is 3.11.6 This is my

pypi_base

definition:

@pypi_base(
    packages={
        "pandas": "2.1.3",
        "azure-storage-blob": "12.19.0",
        "scikit-learn": "1.3.2",
        "xgboost": "2.0.2",
    }
)

and then I have one

pypi

definition

@pypi(packages={"azure-ai-ml": "1.12.1"})