Hi,
I work for a gov. contractor out in northern ...
# ask-metaflowd
damp-doctor-13263
02/11/2024, 1:46 AMHi,
I work for a gov. contractor out in northern virginia serving a very large client. The client is new to ML and modeling so we're trying to do a proof of concept for them. We have built a few models and now we're trying to deploy to production. I have been advocating for Metaflow based on the very minimal documentation I have come across online. The requirements are very strict. The MLOps stack needs to support deploying to airgapped, on-prem environments without any access to AWS.
We have our own K8s cluster and absolutely no access to the internet and no access to anything AWS.
How does @conda and @pypi manage packages that are imported by the individual steps in the flow? Does @pypi need access to https://pypi.org/?
f
fast-vr-44972
02/11/2024, 2:07 AMYou can configure a private repository. From the docs
Both the decorators support private packages published in private software repositories, in case you want to access internally published libraries in Metaflow.
d
dry-beach-38304
02/11/2024, 2:09 AMHey. Yes and no. You need access to the packages at least once (to get them). It can be from a private mirror of course. After the initial fetch of packages, they are typically cached to something like S3. Metaflow works better if there is some storage medium. It doesn’t have to be S3 but something.
d
damp-doctor-13263
02/11/2024, 2:13 AMThank you for the response. I have already set up the Metaflow Service to use MinIO for data storage.
damp-doctor-13263
02/11/2024, 2:14 AMSo, all I need is pypiserver or Nexus Repository to register the Python packages?
damp-doctor-13263
02/11/2024, 2:17 AMAnd have the @pypi decorator to point at the package registry?
d
dry-beach-38304
02/11/2024, 2:37 AMYou will also need some conda repo too. At least once for packages but you do need access to the repodata when resolving. Depending on what you want to do you can definitely reduce access to this particularly with the nflx-extension which allows you to save environments. I am a bit unclear as to when you need things to be airgapped.
dry-beach-38304
02/11/2024, 8:20 PMOK. I can type a bit of a longer response now:
• there are two stages with the pypi/conda decorators:
◦ resolving the environment
◦ executing a step/task within that environment
• For the first stage, you would need access to:
◦ a pypi repo of some kind (it can be your own internal registry or whatever but it needs to be able to find the packages requested as well as their dependencies
◦ a conda repo of some kind. Same thing as for pypi. This is even if you only use pypi packages since the base environment is always setup using conda.
• For the second stage, you would need access to:
◦ your storage backend (you mention MinIO). The packages that have been found in the first stage would be saved in that backend and so there would be no need to access any repo at this time.
I’ll add that with the bleeding edge decorators, you can actually “pre-resolve” environments (ie: the first stage) and then give them names. You can then use those names to execute things and would skip the first stage entirely for that.
Hope this helps. Let us know if you have more questions.
d
damp-doctor-13263
02/13/2024, 2:58 PMThis is really great. Thank you so much. I'll be on a lookout for decorators' support for pre-resolve environments.