Hey Metaflow team, we're PoCing Metaflow at my company and we're running into the following issue when running workflows on azure kubernetes using

Metaflow 2.10.8

and the

pypi

environment:

2023-12-13 13:34:50.754 [93/start/508 (pid 13121)] Kubernetes error:
2023-12-13 13:34:50.755 [93/start/508 (pid 13121)] Error (exit code 1). This could be a transient error. Use @retry to retry.
2023-12-13 13:34:50.755 [93/start/508 (pid 13121)]
2023-12-13 13:34:47.298 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "<frozen runpy>", line 198, in _run_module_as_main
2023-12-13 13:34:47.298 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "<frozen runpy>", line 88, in _run_code
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "/metaflow/metaflow/plugins/pypi/bootstrap.py", line 119, in <module>
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]     shutil.move(tmpfile, dest)
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]   File "/usr/local/lib/python3.11/shutil.py", line 825, in move
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg]     os.rename(src, real_dst)
2023-12-13 13:34:47.299 [93/start/508 (pid 13121)] [pod t-3e5c3889-7w24v-m6zxg] TypeError: rename: src should be string, bytes or os.PathLike, not NoneType

We've traced the error and found that in the

conda.manifest

file the path for the

cryptography

package is the following:

{
          "url": "<https://files.pythonhosted.org/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>",
          "path": "<http://files.pythonhosted.org/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl|files.pythonhosted.org/packages/14/fd/dd5bd6ab0d12476ebca579cbfd48d31bd90fa28fa257b209df585dcf62a0/cryptography-41.0.7-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
        },

but in our Azure blob root container, this path does not exist (note the lack of the

14

sub dir ) We've tried deleting the

.metaflow

directory locally as well as deleting everything under

<http://files.pythonhosted.org/packages|files.pythonhosted.org/packages>

in our blob container but we still get the same error. Any ideas/thoughts?