Hi, I'm getting a weird issue with authentication while installing private packages from our private AWS code artifact The authentication works if I use the extensions package(2.10.5+netflix-ext(1.0.7)), but fails if I use only the default package (2.10.5) Have added the logs Any idea how I could debug this further? Adding logs as a thread

@bulky-afternoon-92433 can you help here?

looking into this now. Initial thought is if the token contains some characters that break the subprocess command, as there's currently no escaping in place. Is the token correctly passed to the command based on the full logs?

Token doesn't get logged in the full logs, it only shows the url This is the index-url in the config:

<https://aws>:<token>@private.d.codeartifact.eu-west-2.amazonaws.com/pypi/ds/simple/

In the logs, it only shows -

Unauthorized for url: <https://private.d.codeartifact.eu-west-2.amazonaws.com/pypi/ds/simple/pvt-package/1.2.2rc11%2Bds.339/pvt_package-1.2.2rc11%2Bds.339-py3-none-any.whl>

username, token doesn't show up

the verbose mode is still on a todo list unfortunately. Does the extension also use the pip.conf for indices or did you supply those through the decorator? have you verified that the pip.conf works outside of metaflow if you try to directly call

pip install

?

the extension and pip install, both works via the conf and is there a way to pass indices via the decorator? I can try that instead

with the default package its not possible to pass indices via the decorator yet, as the pip config was supposed to be a bit of an easier setup overall (i.e. if indices are configured and work on the host, they should work with Metaflow as well out of the box) one more thing, as the token is redacted from the conf. Are you using the token in cleartext in the config, or using env replacement for it?

It's being passed as cleartext in the config and if I pass wrong credentials while doing pip install, I get the error:

401 Credentials not correct for ...

Here its

401 Unauthorized for url:

just wondering if there's something else going wrong apart from credentials?

definitely. if you want to try out debugging the issue, you can checkout the git repo and

pip install --editable .

Hi @bulky-afternoon-92433, I think I've found the issue Inside pip.py, I get none, empty list for

index_url

,

extra_index_url

https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/pypi/pip.py#L175 Even though I'm passing the config as

PIP_CONFIG_FILE=pip.conf python flow.py --environment=conda run

This way of passing the config however works when the extension is installed The extension looks to be fetching the urls as https://github.com/Netflix/metaflow-nflx-extensions/blob/4c2382160ec04b3920d0f40fa[…]46bad396/metaflow_extensions/netflix_ext/plugins/conda/conda.py

excellent, I should be able to patch it today with this. It's most likely the case that the pip config envvar doesn't get passed to the underlying conda env.

There might be some other issue as well, because even if I fix this and make it return the correct index url, with username, token authentication, when it tries to install the package, the url it prints with 401 doesn't have the username, token in it, and it fails Maybe it is stripping away the auth

wellp, got somewhere with looking into this. compared to the extension the main difference is I am using

pip config get

whereas the extension does

pip config list

and opts to parse the output in python. There is an open issue for pip related to the use of

PIP_CONFIG_FILE

and specifically config get: https://github.com/pypa/pip/issues/11700 Can you try with

PIP_CONFIG

instead to see if that works out of the box?

PIP_CONFIG=pip.conf python flow.py --environment=conda run

Hi, it unfortunately doesn't work with PIP_CONFIG as well For now I'm using the extension itself, but as you suggested it would be good to have similar solution in the main package itself

There's a PR for this under review for now, will hopefully get this in next week and this should at least fix the config reading issue when using the env var for file location: https://github.com/Netflix/metaflow/pull/1626 Interesting though that the

PIP_CONFIG

didn't work for you, I'll have to doublecheck this but thought that I tested it was reading the config correctly with that