Hi colleagues !
How can I add in conda_base/conda...
# ask-metaflowg
gifted-helicopter-98777
12/22/2023, 2:50 PMHi colleagues !
How can I add in conda_base/conda/pypi decorator channel to install private libraries?
Or, how can I “merge” libs in image with conda env i’m creating in a flow?
Thank you in advance!
👍 1
d
dry-beach-38304
12/22/2023, 6:14 PMAre you looking to create an environment that has both conda libraries and pypi packages from a private repo?
dry-beach-38304
12/22/2023, 6:15 PMIf so, the base OSs decorators do not provide that functionality but you can use the bleeding edge ones to allow a mix of packages from both conda and pypi. Those decorators rely on conda-lock to make that functionality work.
b
bitter-school-29575
12/22/2023, 6:43 PMI think the question is how to install from a private repo?
I also had the same problem. We're installing it using
pip install git+https://.....d
dry-beach-38304
12/22/2023, 6:59 PMAh. I see. You mean a private git repo. I thought you meant a pypi repository. For the former, I believe the OSs version will support it very soon. There is a PR for it here: https://github.com/Netflix/metaflow/pull/1559. The bleeding edge decorators have support for it already. I can send the syntax if not clear but it is either something like “mypackage @ git+https://…”: “” or without the mypackage @.
b
bitter-school-29575
12/22/2023, 8:06 PMThanks for the answer.
Can you please send the code because I'm not familiar with 'bleeding edge' decorators.
Sorry
d
dry-beach-38304
12/23/2023, 8:18 AMYes, I’ll test it out and send you something today. If you don’t hear from me feel free to ping me again
❤️ 1
b
bitter-school-29575
12/23/2023, 9:37 AMI'm curious is there any way to build a Docker image with all dependencies installed so they won't be downloaded every time?
d
dry-beach-38304
12/23/2023, 7:16 PMok, to answer your second question first: yes, it is absolutely possible to build your own custom docker image with all the packages you need and then you don’t need to use conda/pypi at all.
dry-beach-38304
12/23/2023, 7:16 PMThis is typically a good solution if you are always running the same flow over and over and can maintain the docker image yourself.
dry-beach-38304
12/23/2023, 7:17 PMMetaflow hasn’t chosen this approach (yet) because typically docker is higher overhead and, lots of times, does not work readily on cloud workstations.
dry-beach-38304
12/23/2023, 7:17 PMbut it is entirely possible.
dry-beach-38304
12/23/2023, 7:17 PMas of now, you can do it entirely separately from metaflow. Metaflow supports the specification of your own docker image in most decorators like @batch
dry-beach-38304
12/23/2023, 7:21 PMFor the other question, I need to look a little bit more. For pure pypi, you can do this:
Copy code
@pypi(packages={"<https://github.com/openai/CLIP@main>": ""})
@step
def start(self):
self.next(self.end)dry-beach-38304
12/23/2023, 7:21 PMThat works well
dry-beach-38304
12/23/2023, 7:22 PMFor mixed mode (so conda and pypi packages), I may have a bug. I will try to have a look in a bit (but maybe not right now unless super urgent)
b
bitter-school-29575
12/24/2023, 7:55 AMAnd seems I have mixed mode because this doesn't work.
I would like to create a separate question regarding the Docker image.
Thanks you very much!
d
dry-beach-38304
12/24/2023, 8:10 AMOkie. I’ll check mixed mode and see what I can do. It should be doable.
dry-beach-38304
01/11/2024, 10:13 AMSorry for the delay — it hit me afterwards that there is a problem in mixed mode due to a bug in poetry: https://github.com/python-poetry/poetry/issues/8774. I filed this a while ago but haven’t had time to propose a fix for it yet (and it doesn’t look like the poetry community jumped on it 🙂 ). It may take me a bit longer to fix due to that.
6 Views