Hey there. I'm running into problem when trying to use metaflow-netflixext that I haven't seen before. It's a new deployment using the kubernetes/argo stack, which we haven't used before, so it might have to do with some faulty configuration. I'm testing it with a very simple flow like this one When I don't have the extensions package installed, everything works as expected. I can run it using

--with kubernetes

and I can deploy it as an argo-workflow and trigger it without problems. When I install the metaflow-netflixext, I can still run it locally, however when I try to run it remotely in a container I get the error that some 20 packages were not found in the cache. All of these look like they are required for Metaflow itself, not for the specific flow I'm running. E.g.

'ld_impl_linux-64-2.44-h1423503_1': not found at packages/conda/conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.44-h1423503_1.conda/ld_impl_linux-64-2.44-h1423503_1.conda/0be7c6e070c19105f966d3758448d018/ld_impl_linux-64-2.44-h1423503_1.conda
'libgomp-15.1.0-h767d61c_4': not found at packages/conda/conda.anaconda.org/conda-forge/linux-64/libgomp-15.1.0-h767d61c_4.conda/libgomp-15.1.0-h767d61c_4.conda/3baf8976c96134738bba224e9ef6b1e5/libgomp-15.1.0-h767d61c_4.conda
...
'uv-0.7.8-h2f11bb8_0': not found at packages/conda/conda.anaconda.org/conda-forge/linux-64/uv-0.7.8-h2f11bb8_0.conda/uv-0.7.8-h2f11bb8_0.conda/aff01745ebc7e711904866ee2e762a42/uv-0.7.8-h2f11bb8_0.conda

When I look at our datastore location on S3, this is true. They packages are indeed not there. Why are they not there and what do I have to do to make them available?

I am not 100% sure because I don’t have the full trace (you can send me one using

METAFLOW_DEBUG_CONDA=1 …

but it may mean that Metaflow thinks it already cached those packages but they were actually wiped (maybe a TTL or something?) To re-upload, you have to make Metaflow forget about them in a way. The reference to these files is stored in a

conda.cnd

(or

condav2.cnd

or

condav2-1.cnd

(I get confused between internal and external)) in the

.metaflow

directory that should be either in your current working directory or in one of its parents. You can move that file aside or remove it and the next time you run, Metaflow will re-resolve the environment and re-upload things that are not cached. Note that it does expect things to persist so if your buckets have a TTL you may have further problems down the line.

Hello, sorry I've been held up by other tasks.

To re-upload, you have to make Metaflow forget about them in a way. The reference to these files is stored in a

conda.cnd

(or

condav2.cnd

or

condav2-1.cnd

(I get confused between internal and external)) in the

.metaflow

directory that should be either in your current working directory or in

one of its parents. You can move that file aside or remove it and the

next time you run, Metaflow will re-resolve the environment and

re-upload things that are not cached. Note that it does expect things to persist so if your buckets have a TTL you may have further problems down the line.

This was indeed the reason. Removing it solved my issue. So, I guess it is a bit of an anti-pattern to use the same code location for multiple different metaflow deployments, right?

not really — the issue here is that the metaflow cache was “cleaned” and it was unexpected.