Hi, i'm liking the pypi decorator that's being introduced THANKS AGAIN !!! . I saw that there is a pip.py in development https://github.com/Netflix/metaflow/blob/chore/pip-deco-fixes/metaflow/plugins/pypi/pip.py. I presume soon a pip decorator will be introduced ? Will the pip decorator has the capability to configure to load python packages from private python repo (example: AWS CodeArtifact) ?

the

@pypi

decorator is the file you mention (see here: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/pip.py). I believe these are old branches that were merged into master.

Oh, okay. The bleeding edge version https://github.com/Netflix/metaflow-nflx-extensions, only works with --environment=conda at the moment right ?

--environment=pypi and --environment=conda are strict aliases (in the main release) so yes, the extension only calls it --environment=conda but in all cases, there is conda involved 🙂

ok, will give it a try to see if i can get the authentication to CodeArtifact going ...

if you have your credentials in your pip.conf, it should just work (hopefully). I know @wooden-dusk-90720 did get it to work with CodeArtifact (iirc). If you have questions, let me know.

@adventurous-gigabyte-81428 I am very interested in hearing whether you did manage to get it to work. We do also use CodeArtifact at the place where I work and I would really appreciate the decorator to work with it.

@billowy-receptionist-31718 are u using AWS Batch or Kubernetes for ur Metaflow compute ?

Kubernetes

Will keep this thread updated with our findings. There is an unconventional way to get it to work :) which is to initially created the pip.conf via the aws cli all in the userdata when the ec2 is launched. Then via the kubernetes decorator mount the host volume to the pod to replace the pip.conf A bit convoluted, but that’s one work around we got working with AWS Batch

the resolution of the environment (both in the mainline and in the extension) should only happen on the machine doing the launching. Not sure why you would need it on the compute nodes themselves but I may be missing something.

Thank you Samuel. I am looking forward to hear more about your findings. We would love to be able to instruct the decorators to use another PYPI source.

@dry-beach-38304 yeah the unconventional approach we were using was when we didn’t even use conda or pypi. And we were using our own hacked pip decorator. It’s a dirty approach just to get something out there first …. :) But yeah, I’m revisiting a better approach with the conda + Metaflow extension

ah ok 🙂. That makes sense now.

@billowy-receptionist-31718 here's a simple example. Install with python library of metaflow version 2.10.2, metaflow-netflixext 1.0.5. With the code snippet like so. We got our private repo going

@pypi_base(python="3.8.0",
           packages={"boto3": "1.25.1",
                     "torch": "1.11.0",
                     "YOUR_PRIVATE_PACKAGE": "1.0.0"},
           extra_indices=[f"<https://aws>:{AWS_CA_TOKEN}@XXXX.d.codeartifact.YYYY.amazonaws.com/pypi/ZZZS/simple"])

Out of curiosity, did it not work if the indice was in the pip.conf?

erm.. i've not tried, let give it a go a let you know 🙂

It should ideally. If it doesn’t let me know as it may be a bug :).

I can confirm it works

it works also when the indice was in the pip.conf 🙂

well, the custom

pip

wasn’t wrong per say … nothing better existed officially 🙂.

with this new understanding, we don't need to worry about having the remote docker to be able to login to codeartifact to individually install our private python packages anymore, since all the environment is "pre-build" as a package to be uploaded to s3

correct. With a small caveat: the environment is already resolved so it has a lock file of sorts and all packages are all pushed to S3 (each package is separately pushed so they can be reused across environments). WIth the extension, the definition of the environment itself is also pushed to S3. But in all cases (non extension or extension), your understanding is correct that remotely, the packages are directly fetched from S3 without the need to go through a resolution or to query your private codeartifact or anything like that.

honest i felt relief now with a belated lightbulb moment, been running metaflow the "wrong" way for about a year lol.

@adventurous-gigabyte-81428 That looks really nice and simple, and thank you for posting the example. I will try the same here today at our setup. Another question though: How are you dealing with the AWS_CA_TOKEN? Are you manually running the

aws codeartifact get-authorization-token

before you run your flows? @dry-beach-38304 are the metaflow pypi/conda decorators designed to always read from the pip.conf if present? Hence it will maybe be cleaner to have the indices etc. in there, as you also suggests. I guess I am just trying to abstract as much hassle away from our datascientists as possible

@billowy-receptionist-31718 : • for the decorators in the main repository, they should read from pip.conf but don’t support auth yet. • for the ones in the extension, you have a few options: ◦ pass them as

extra_indices

as shown above ◦ have them in the pip.conf ◦ have it specified in

METAFLOW_CONDA_DEFAULT_PYPI_SOURCE

(specifying it here effectively replaces the default index, for example pypi). You can set this directly in your

~/.metaflowconfig/

json config just like all the other metaflow configuration values you may have.

@billowy-receptionist-31718 yes we manually run the logging and output it to an environment variable

@adventurous-gigabyte-81428 @dry-beach-38304 after i installed the extension metaflow tells me to run my flow with

--environment=conda

as opposed to before where i were running with

--environment=pypi

also i am getting following error:

(metaflow-dummy-py3.11) ➜  metaflow-dummy git:(main) ✗ python pypi-decorator-flow.py --environment=conda run                                                                                                                   
Metaflow 2.10.2+netflix-ext(1.0.5) executing PandasFlow for user:jesper.rix
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping Conda environment... (this could take a few minutes)
    Incompatible environment:
    No mamba binary found

Yes. In the mainline pypi and conda environment are aliases but in the extension it’s just called conda. It has no functional difference. For the solver, yes the extension uses mamba by default (I found it was more accurate at reporting issues) but you can switch to micromamba if you want by setting METAFLOW_CONDA_DEPENDENCY_RESOLVER to micromamba

Hi again I just wanted to tell you that i managed to get it work by modifying our existing configuration a little bit. I got it working with the `pypi`step decorator like here:

@pypi(python='3.11.3', packages={'pandas': '2.1.0', 'topicalityfilter': '0.1.3'})
    @card
    @step
    def start(self):

topicality filter are our internal package, which are located in our own AWS CodeArtifact repo. For this example i just held the authorization token in my `pip.conf`file. Thanks for your help and time guys, much appreciated 🙏

cool. Glad you got it working. If you have any further question, don’t hesitate to hit us up.