Hi, now that we have Pip support, what's the best way to pass additional index to the @pypi decorator? More specifically: 1. How do I pass args to install private package hosted on our aws codeartifact? I see a line mentioned in the documentation that it can be installed, but no example there unfortunately 2. How do I pass index url per package? e.g. I want to replicate:

pip install torch --index-url <https://download.pytorch.org/whl/cu118>

currently the deco pulls configuration from the host systems pip config, so adding the aws codeartifact repo as an index-url should be sufficient. Specifically for codeartifact, the simplest setup was to use the aws cli command provided by its connection instructions

Per package indices was related to this actually: https://github.com/pypa/pip/issues/8606 more of a security concern, but I think for now i'll be able to solve it with having my private package as the default index, and having pypi in the extra index url

hmm, based on the issue you linked and https://github.com/pypa/pip/issues/5045#issuecomment-369521345 it doesn't seem that there is any priority in cases where the same package is available on both indices.

Any idea on how to solve this? One way would be to publish the correct torch to our private package repo itself Any other approaches you could see?

sorry for the delay, some thoughts on the matter, I'll continue with this on monday: Having a fully vetted private package repo is the most reliable solution, but understandably causes a lot of work with managing the packages. I haven't looked too much into AWS CodeArtifact upstream capabilities if there are any pick-and-choose capabilities with that Another alternative would be indeed what you were asking about originally, where you could specify an index-url specifically for a package. This comes with a few drawbacks with the current implementation: • currently all packages are resolved in one command with provided extra configs, so at the very least an overhaul for this would be required • providing the index-url for a package is a bit of a headscratcher as well, as it is only required when initially resolving/downloading the packages on the users machine, before they get cached into S3 in case of remote execution. If the index url contains f.ex. credentials, then preferably this should be kept out of the flow file. A third option which could be explored (and what is somewhat touched upon with the upcoming git support for pypi) would be introducing the ability to add wheels directly as dependencies with their fully qualified url. I'm not really a fan of any of the alternatives, as the pip config provides the easiest set-and-forget approach for defining a package source, but I do understand the concern regarding index priority (especially with the pytorch incident).

For now the pytorch issue is resolved by making it

pytorch==2.1.0+cu118

, this resolves the index correctly, so I'm not blocked right now. Facing another issue with pypi though I'm trying to install a package -

pytorchvideo

(https://pypi.org/project/pytorchvideo/0.1.5/#files)

from metaflow import FlowSpec, pypi, step

class FractalFlow(FlowSpec):

    @step
    def start(self):
        self.next(self.plot)

    @pypi(python='3.9.13',
          packages={'pytorchvideo': '0.1.5'})
    @step
    def plot(self):
        # pylint: disable=import-error,no-member
        import pytorchvideo
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    FractalFlow()

This flow fails, even though lcally

pip install pytorchvideo==0.1.5

works for me Tried it on mac and linux both

Metaflow 2.10.4 executing FractalFlow for user:macuser
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Bootstrapping virtual environment(s) ...
    Pip ran into an error while setting up environment:
    command '/Users/macuser/.metaflowconfig/micromamba/bin/micromamba run --prefix /Users/macuser/micromamba/envs/metaflow/osx-arm64/7590fe4ac935166 pip3 --disable-pip-version-check --no-input --no-color --isolated install --dry-run --only-binary=:all: --upgrade-strategy=only-if-needed --target=/var/folders/z2/1gcnb7c55sg2c7hw53p5j3qh0000gn/T/tmpjy50s7hy --report=/var/folders/z2/1gcnb7c55sg2c7hw53p5j3qh0000gn/T/tmpjy50s7hy/report.json --progress-bar=off --quiet --abi cp39 --abi abi3 --abi none --platform any --platform macosx_13_0_universal2 --platform macosx_10_9_universal2 --platform macosx_12_0_arm64 --platform macosx_10_8_universal2 --platform macosx_10_5_universal2 --platform macosx_10_12_universal2 --platform macosx_10_16_universal2 --platform macosx_13_0_arm64 --platform macosx_10_6_universal2 --platform macosx_10_15_universal2 --platform macosx_10_7_universal2 --platform macosx_10_10_universal2 --platform macosx_10_11_universal2 --platform macosx_12_0_universal2 --platform macosx_11_0_universal2 --platform macosx_10_14_universal2 --platform macosx_11_0_arm64 --platform macosx_10_4_universal2 --platform macosx_10_13_universal2 requests>=2.21.0 boto3>=1.14.0 pytorchvideo==0.1.5' returned error (1) 
    ERROR: Could not find a version that satisfies the requirement pytorchvideo==0.1.5 (from versions: none)
    ERROR: No matching distribution found for pytorchvideo==0.1.5

The package doesn't have any whl file available (https://pypi.org/project/pytorchvideo/0.1.5/#files) Is it because of this?

it would be due to the

--only-binary=:all:

option that we pass to the pip install and there only being a source package available for pytorchvideo. The requirement for this was for cross-platform package resolving (f.ex. deploying from an M2 mac to a linux-x86 kubernetes cluster, how to cache required packages). Arguably source packages that do not require platform specific build steps should work in either case, but there are no checks in place for this at the moment.

Is there a way I can override it? Assuming I build and run on the same platform

for the time being there's no way around it for the pypi deco, but I'll take note on the need as we improve the package resolving further regarding target platforms. In the mean time, would resorting to the @conda decorator work for that specific step, or was there some issue with using conda channels vs. custom pip indices?

We have a private pip package, on aws codeartifact, which we need to install. This package depends on pytorchvideo - that gives the above error on using pypi decorator Since I can't pass pip index to conda, tried using the pypi decorator

for the time being you could look into https://docs.metaflow.org/scaling/dependencies/libraries#bleeding-edge-versions-of-the-decorators as well, as one thing it allows you to do is mix-and-match pypi and conda packages

@salmon-furniture-31733: • if you build and run on the same platform, the bleeding edge version supports non-wheel packages (behind the scene, it will build the wheel to ensure reproducibility) • if you are not on the same platform, you can also, as @bulky-afternoon-92433 mentions, mix and match packages and take pytorch from conda and other packages from pip. In this case, pytorchvideo does not seem to be available on conda So, if you are on the same platform, it will work fine. If you are not though, the bleeding edge version will also fail because it won’t build cross platform for you. There is a workaround for this though which consists of resolving your environment in your target environment using something like this: https://netflix.slack.com/archives/C02116BBNTU/p1697359042901309?thread_ts=1697198947.028709&cid=C02116BBNTU Let me know if you have more questions.

Hey @dry-beach-38304, thanks for the suggestions! For now I'll try building it on the same platform with the extensions package Just wanted to check though, is it in the planned stages to support cross platform, for the bleeding edge version?

I am going to try to build more packages (the ones that are “noarch” basically and that work on all platforms) cross-platforms but I am not sure how to cross-build so that part may not be supported. It’s a bit of a conundrum: • either I delay building the packages till you get on platform (so when you are running): in this case though, you are kind of removing some of the guarantees we provide (ie: the environment isn’t fully locked until the build step happens) • either I prevent cross-platform building. I don’t have a great solution yet. Maybe there is an intermediate one where a “env build” step gets introduced to build the packages once. TL;DR: there may be some improvements there but not sure I can go the full way yet (or maybe I just decide to allow source only packages and build them when running).