A related followup: During solve: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/pip.py#L110

***@pypi*** does not currently support source distributions

But later during download step: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/pip.py#L154C1-L161C33

# build wheels if needed
        with ThreadPoolExecutor() as executor:

            def _build(key, package):
                dest = "{prefix}/.pip/built_wheels/{key}".format(prefix=prefix, key=key)
                cmd = [
                    "wheel",
                    "--no-deps",

I’m not quite following why it still trys to

pip wheel

if it doesn’t intend to support source distribution? 🤔

we build wheels for supporting git repos as dependencies

ah yes that’s what it seems to do. thanks for confirming!

it's a git repo - technically this approach can be extended to work with .tar.gz as well - if you would like to take a stab at it, happy to work together! let me know!

That’ll be awesome! I’ll reach out when I have something more concrete!

Sounds good

Hi @square-wire-39606, 👋 I just put out this pr: https://github.com/Netflix/metaflow/pull/2354 I’ve able to verify that it works within our stack (metaflow/kubenetes/argo)/ But I know metaflow covers more use cases than the ones we currently have. We’d love to hear feedback on this. Could you help kick off the review process to move this forward?

thanks! we will review the change. re: more changes - especially if they are larger ones - the recommendation is to start a thread in #C020U025QJK to align on the approach - after that the review process is pretty straightforward

Got it. thanks!

can you run the pre-commit hook for the PR?

how to trigger the pre-commit hook? is there a button to click on the ui? 🤔 (Or I can make an update to the PR soon to address comments I think update automatically triggers the pre-commit hook? )

yeah you can install pre-commit and just execute that

Hi Savin, thanks for the review. want to follow up on your comment more casually here if ok.

it is tricky because one needs to build the source to generate downstream dependency graph

I wonder why git dependencies doesn’t have this problem. When installing from git repo, it still need to resolve all its dependencies right? why pip can resolve git dependencies with those

--platform

options but not sdist? (E.g. if I unpack a sdist and upload source code to github, what’s the difference does it make to pip?)

There is a possibility to make it work when the local_platform is the same as the target_platform

I think the use case can be larger. If the sdist is pure python, then I’d expect it always to work. E.g. If I build sdist into

thrift-0.20.0-py3-none-any.whl

on my macos arm64, then the same whl should work on my remote linux64 right?

Sorry — busy week. I remember the thread about your fork. I can try to help un fork it. The metaflow extension already supports source pip (and ir was quite annoying — granted I do more than what you are trying to do but still) so it may make sense for you to use it directly. For your fork, I think we can override the boto part and you should be able to override some Argo stuff too. It’s not hugely pretty but should work. I’d have to see your code though.

Thanks Romain! Let’s definitely touch base on this. Let me setup a meeting to kick this off. We may not need to go all the way to unfork it right away. Better understanding of why it may fail and insights from your experience in implementing it would already help us plan and debug better for future. thankyou