Hi, In the past 2 days we have started getting an issue when running our custom images. It seems like azure has pushed a faulty version of azure-storage-blob and we got that version when metaflow is installing it here Is there a way to fixate that? (it's over today when they pushed another version that fixed the last one) The error we got originated from this file in their package:

azure/storage/blob/_shared/base_client.py

getting KeyError: "sdk_moniker" problematic version:

azure-storage-blob==12.18.0

(12.18.1 solved our issue)

One option is to build a custom docker image with all of those pip packages installed. If the package already exists during the bootstrap script, then that should be a no-op and be pinned to your working version. This is a better option in the long-run, because runtime installation of unpinned pip packages has caused problems in the past Some docs on using a custom image: https://outerbounds.com/docs/use-custom-image/ Other option in the short term is to fork metaflow and pin the version in that metaflow_environment file

We already have a version of azure-storage-blob + azure-identity installed on our docker images and it installed the newest version of that package (we validated by printing the version inside the flow + ssh the pod and checked the version) pining the version in a fork was my short term solution while debugging the issue. (until azure fix)

My guess is that you haven’t installed all 3 package

azure-identity azure-storage-blob simple-azure-blob-downloader

in pip. If one of those isn’t installed, then the pip installer may trigger a re-solving of dependencies and update it Here’s an issue covering this topic that may be helpful: https://github.com/Netflix/metaflow/pull/1490

Yeah, we haven’t installed the simple-azure-blob-downloader. Great, thanks! I’ll give it a go with the env var and install manually with fixate version 🙏 🙏 Thank you both! @elegant-beach-10818 @square-wire-39606

Worth noting that the env var flag for that pull request was never approved because of the option of pip installing all packages beforehand which results in a no-op

O, will try that, thanks!