hey Outerbounds team, running into an odd error on a secondary instance of metaflow with libuuid failing during step execution. The environment has been cached and pushed to s3, but for some reason when setting up the environment on the 1st step after

start

, we hit the below error:

2023-12-18 23:48:06.811 [42175/load_data/1091563 (pid 122589)] [117ea501-bf00-4ffd-b557-d418e9b67e60] Pretty-printed STDERR:
2023-12-18 23:48:06.811 [42175/load_data/1091563 (pid 122589)] [117ea501-bf00-4ffd-b557-d418e9b67e60] critical libmamba Download error (1) Unsupported protocol [<s3://metaflow-dev-us-east-1/metaflow/conda_env/packages/conda/packages/conda/_transmut/conda.anaconda.org/conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.tar.bz2>]
2023-12-18 23:48:06.811 [42175/load_data/1091563 (pid 122589)] [117ea501-bf00-4ffd-b557-d418e9b67e60]     Protocol "s3" not supported or disabled in libcurl

Hey @bulky-portugal-95315 — based on the logs, it looks like you are using the Netflix extension decorators. Could you run setting METAFLOW_DEBUG_CONDA=1 (ie: set that environment variable). If you could then point me to the instance of thr log thst works and the one that doesn’t, I can try to see what the difference between the two may be. It’s really weird that it does this only once. I’ve seen this error in the past but typically with conda and not mamba/micromamba.

yeah i’ve searched all over trying to figure out where the error is coming from. so to give you the full picture, we actually created our own extensions library to integrate with our artifactory to cache packages from internal libraries. let me set the debug flag and see what it logs

sure. debug flag should help me track it down a bit (hopefully).

@dry-beach-38304 i was able to switch instances to another EC2 instance and do a fresh setup and this error seems to have gone away, i cannot explain it

For glibc, the latest version of the extension actually allows you to cap the version of glibc. I ran into this issue lately too as we were working across different Ubuntu versions. You can run using CONDA_OVERRIDE_GLIBC=2.31 which should make sure that it only resolves with packages compatible with that version of glibc.

oh so it does have something to do with the ubuntu version that you are submitting the flow run from?

If you are resolving on Linux it should work without any changes but you can always try with https://github.com/Netflix/metaflow-nflx-extensions/pull/32 as well. Could you also run with thr debug flag so I can verify the glibc version is being pulled in and respected?

yep let me run it with the debug flag on and DM you the log