Hi team in the pypi plugin impl it only accepts the whl but Outerbounds #ask-metaflow

Hi team, in the pypi plugin impl, it only accepts ...

ripe-alarm-8919

02/18/2025, 8:17 PM

Hi team, in the pypi plugin impl, it only accepts the whl but not the source distribution. https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/pip.py#L81C17-L81C54 I wonder why this is the case 🤔 (why not allow both and prefer binary?) The reason I’m asking is that we do have some dependencies seems to not publish whl, only the source. e.g.

Copy code

[[package]]
name = "daff"
version = "1.3.46"
description = "Diff and patch tables"
optional = false
python-versions = "*"
files = [
    {file = "daff-1.3.46.tar.gz", hash = "sha256:22d0da9fd6a3275b54c926a9c97b180f9258aad65113ea18f3fec52cbadcd818"},
]

Wonder what are your thoughts and whether there’s a better way to approach this instead of forking this behavior. thankyou

dry-beach-38304

02/18/2025, 8:43 PM

You can try the Netflix extension which should accept all.

👀 1

ripe-alarm-8919

02/19/2025, 2:05 AM

Reading about it, I believe the netflix extension CondaV2 should support the

@pypi

decorator for source tarballs. For context, we maintain our own

metaflow-fork

and

metaflow_extensions/css

now we would need the following override precedence:

metaflow_extensions/css

metaflow_extensions/netflix_ext

metaflow-fork

I’m struggling with the best way to organize this. I want to avoid maintaining another fork of netflix_ext. @dry-beach-38304 Wonder if you have any context on whether the extension framework can allow multiple extensions work in chain like this.

dry-beach-38304

02/19/2025, 11:18 AM

You can have as many extensions as you want. You do not need to maintain another fork. Provided your extensions are properly namespace packages (it should complain if they are not), you can install any number you like and it will load all of them.

dry-beach-38304

02/19/2025, 11:19 AM

It does maintain a deterministic order based on the order of the packages (ie: if extA depends on extB, extB gets loaded first and extA after it and can override anything in extB). I am not sure if your metaflow-fork is an extension or an actual fork but: • if it is an extension, if you make it depend on netflix_ext, it will load after and can override anything in there • if it is a fork, the extension will override stuff (or can). Netflix ext does not override much except the pypi and conda decorators which is what you want anyways.

ripe-alarm-8919

02/19/2025, 7:48 PM

I have both 1. a

metaflow-fork

(i.e. forking https://github.com/Netflix/metaflow) 2. and a metaflow extension

metaflow_extensions/css

. (similar to the netflix extension, folder structure is copied from the template) > extA depends on extB, how to make this happen? Currently my extension

metaflow_extensions/css

depends on my fork

metaflow==2.14.0+css.2

. (see screenshot of extension folder) Should I update the

setup.py

of my extension to depend on netflix_ext, then modify netflix_ext to depend on my fork? (how to avoid modifying netflix_ext? )

ripe-alarm-8919

02/27/2025, 1:47 AM

Tried to add the source distribution support of

@pypi

directly in the

metaflow-fork

itself but it led to so many instability that’s beyond my ability to tackle. I’m planning to integrate with

metaflow-nflx-extensions

instead. Since i’ll need to make it point to my metaflow-fork, (not the upstream metaflow), I don’t think I can directly install it. i’ll have to maintain a fork of

metaflow-nflx-extensions

so that: user flow ----> metaflow-{my-org}-extensions ------> metaflow-nflx-extensions ----> metaflow-fork @dry-beach-38304 Please let me know your thoughts on this approach or if you see better way. thankyou

dry-beach-38304

02/27/2025, 7:15 PM

that would work but seems quite heavy weight. I am curious to know what your fork has that you can’t do in an extension? That may be a better way to do it so you wouldn’t have to maintain a fork and you could use the nflx extension wholesale

🌟 1

ripe-alarm-8919

02/27/2025, 8:29 PM

It would be ideal if we can un-fork it! Today we have 2 essential changes on the fork: 1.

metaflow/metaflow_environment.py

to support UNSIGNED call to s3.

Copy code

def _get_download_code_package_cmd(self, code_package_url, datastore_type):
...
boto3.client(...).download_file

here we updated the boto3 client usage so that it makes the request with

Config(signature_version=UNSIGNED)

config. 2.

metaflow/plugins/argo/argo_workflows.py

modifications to support our custom decorator

@rayStep

(where the step should run on Ray)

Copy code

... to invoke the task_finished method for raysteps. This is because the raystep itself runs on a remote ray cluster, while argo workflow expects some steps after the task finishes, which includes writing task_ids to files on the argo pods. This is the reason we cannot perform these operations in the task_finished() method of the RaystepDecorator, and the reason why we need to use this cli wrapper to invoke the task_finished method.

I’m not sure how to achieve above in a plugin. (maybe module override?) Would love to hear your thoughts on this!

9 Views

Open in Slack

Previous Next