https://outerbounds.com/ logo
Join Slack
Powered by
Hello all, we are running a docker container for o...
# ask-metaflow
h
Hello all, we are running a docker container for our dev environment. I am able to use the container shell and use pip to install packages from our private repository, but as soon as we try to run flows that require packages from that same private repo, it doesn't work, we get a 401 Error. It seems like metaflow is building the conda command but the pip keyring configurations do not get passed through from metaflow-->conda-->pip. Any help would be appreciated on how to configure either metaflow or conda to pass those through!
v
that seems like a plausible explanation. How do you auth with your private repo exactly?
h
GCP has the concept of Application Default Credentials, kind of like a "personal service account key" that you can use to authenticate to the different services from your machine. In this case, I am using it as described here, the part where it talks about the python keyring and ADC. I am passing the 
GOOGLE_APPLICATION_CREDENTIALS
environment variable to the container and mounting the json "key" file as a volume. from there, using pip in the container "just works", but it seems like metaflow/conda/pip doesn't respect that same flow
v
thanks, good context. It's possible that you are the first one using this particular flow with 
@pypi
and 
@conda
. I opened an issue here
thankyou 1
@ancient-application-36103 is OOO but he might be able to provide further insights
b
for some additional context on why the keyring auth might not be working with OSS 
@pypi
: What is happening under the hood upon environment resolving is that we call 
pip config list
and parse it for any index-url and extra-index-url entries. The assumption with the implementation is that for private repositories, the url will contain the required credentials. This should be simple to verify if you are able to call the pip config list to see what is included in the index-url strings. In the mean time, the same page lists an option for authenticating with a service account key, which adds auth credentials as part of the url. This should work with the current versions, unless its not an option for your use case.
the bleeding-edge version of the decorator pulls pip config in a similar fashion, but I'm unsure if it had support for multiple auth schemes aside from simple http auth. @dry-beach-38304 can confirm if this is the case
h
@bulky-afternoon-92433, Thank you for that extra context, this is exactly what is happening. The service account key method works fine and this is how we currently were able to make it work, but we are trying to distribute this set up without having to use them!
👌 1
hoping this method can be supported in the future 🙂
b
Glad you a the workaround for the time being. Will definitely look into implementing the keyring support as time permits, unless someone else beats me to it 🙂
🙌 1
d
The bleeding edge decorators have an additional mechanism to pull in auth (outside of the pip config). I haven’t tested with this but there are env vars thst can be used to set additional auth.
h
Hi @dry-beach-38304, @bulky-afternoon-92433, Following-up on this thread with you guys, have you been able to implement this?
b
Hi @hundreds-midnight-75494, I have setup a private PyPI registry on GCP on my side, can you help me with what your flow looks like?
h
Hi @brainy-truck-72938! Basically we would want to remove the requirement for an authentication in our 
pip.conf
and 
.pyirc
files and just let the application default credentials do their magic. I have tested a pip install from our repo directly from the local environment and it works just fine, but when trying to install the same requirements from within metaflow, the auth doesn't get passed through.
b
Hi @hundreds-midnight-75494 please confirm if I understand correctly,
from there, using pip in the container "just works"
In my case i.e. the local setup, I still have to do the following i.e. pass 
--extra-index-url
. This is irrespective of if I use 
GOOGLE_APPLICATION_CREDENTIALS
or not. This is because I am not using any 
.pypirc
file and the output for 
pip config list
is empty for me.. Thus, I always need 
--extra-index-url
, both with and without 
GOOGLE_APPLICATION_CREDENTIALS
How does your docker container get to know about the 
--extra-index-url
/ address of the repository...
or maybe you do use 
pip.conf
or perhaps 
.pypirc
, just that it doesn't have the creds, but it does have the repo URL, is it?
h
yes, we have both 
pip.conf
as well as 
.pypirc
and if we do not pass in the credentials, it just works with ADC locally, but when it is time to run on the cluster, it doesn't "pass through"
(even if the container spins up with the appropriate workload identity, with a service account that has the appropriate permissions)
b
yeah, so you have 
pip.conf
/ 
.pypirc
where the repo URL is kinda fetched from AND you use 
GOOGLE_APPLICATION_CREDENTIALS
which has the file (mounted) so it is using both, correct?
h
correct
b
okay, can you help me with how your 
flow.py
looks like? not the business logic, only the usage of these decorators and stuff..
h
this setup only works if we have the credentials in the URL of the pip repo
b
yeah, but we don't want creds in URL, we want to use 
GOOGLE_APPLICATION_CREDENTIALS
, pointing to the mounted file?
h
correct!
b
thanks, if you could help me with how your 
flow.py
looks, that could help a bit...
h
Copy code
@pypi_base(
    packages={
        'google-cloud-bigquery': '3.11.4',
        'google-cloud-logging': '3.8.0',
        'google-cloud-storage': '2.12.0',
        'dynaconf': '3.2.3',
        'gcsfs': '2023.9.2',
        'pandas': '2.1.1',
        'pandas-gbq': '0.19.2',
        'prophet': '1.1.1',
        'holidays': '0.18.0',
        'internalpackagename1': '1.9.0',
        'internalpackagename2': '0.1.16',
        'internalpackagename3': '0.1.46',
    },
    python="3.9",
)
the relevant part would be this , I replaced the names of the packages by internalpackagename1-2-3 but you get the idea 🙂
b
thanks!
h
if we have the credentials in 
.pypirc
and 
pip.conf
, it works flawlessly across the board, but if we remove the credential part, it will only work for local flow runs, not remote ones
b
yeah, can you also help me with how do you: • set the env variable i.e. 
GOOGLE_APPLICATION_CREDENTIALS
• mount the file further, you run 
remote
runs on kubernetes / argo-workflows / batch, etc.?
I am just trying to produce the exact environment 🙂
also, can you add 
keyrings.google-artifactregistry-auth
in your list of packages in the 
pypi_base
decorator and let me know if that works? @hundreds-midnight-75494
h
I appreciate it! sorry. bouncing between a few things and at the end of my work day so trying to wrap things up 🙂
👍 1
b
no worries, for me locally, I was missing 
keyrings.google-artifactregistry-auth
in one of my conda environments, and thus, even using 
GOOGLE_APPLICATION_CREDENTIALS
was failing..
so we can try adding 
keyring
and 
keyrings.google-artifactregistry-auth
just for a sanity check first
h
so we are not specifically mounting a file nor setting the env variable in the cluster, this happens automatically by setting the appropriate kubernetes service account to use a specific google service account that has the required permissions in IAM
yeah looking at that part now
b
add those two to 
pypi_base
h
yeah I think you're onto something there for sure!
b
it seems like they do need their own backend for keyring i.e. 
keyrings.google-artifactregistry-auth
only then, the variable 
GOOGLE_APPLICATION_CREDENTIALS
seems to work...
alternatively, you can check if your docker container had 
keyrings.google-artifactregistry-auth
installed?
Hi @hundreds-midnight-75494, were you able to verify the above?
h
Hi @brainy-truck-72938, the couple runs I tried last night didn't work out. I was hoping to test it out again this morning, but something else came up. I will try and get back to it this pm!
b
interesting
h
WARNING: 401 Error, Credentials not correct for https://us-python.pkg.dev/ourrepo
is what I was seeing
when not specifying the credentials as part of the URL
b
yes, I get the same if 
keyrings.google-artifactregistry-auth
is not installed
h
yeah but it looks like it's not able to install any package as it is trying to pull from our repo first
let me double check the configurations this pm
b
yeah, 
keyrings.google-artifactregistry-auth
should be installed first...
h
that's where I was going to go 🙂
it was set up to install before our private packages though
b
Screenshot 2024-03-19 at 8.55.32 PM.png
so, I get this prompt
this is when 
keyrings.google-artifactregistry-auth
is not installed
and when I enter info, I get 401
but if 
keyrings.google-artifactregistry-auth
is installed, it works out of the box
aka no prompt
the command I am using is this:
Copy code
GOOGLE_APPLICATION_CREDENTIALS=~/Desktop/outerbounds/mf-af-gcp-exploration-f72606b8ba0c.json pip install swagger-petstore-open-api-3-0-client --extra-index-url <https://us-python.pkg.dev/mf-af-gcp-exploration/demo/simple/>
h
yeah, I have no problem with this on my local box, all is authenticated perfectly, it just won't follow-through on the cluster
b
I really believe having 
keyrings.google-artifactregistry-auth
and 
keyring
should solve it
h
yeah, I hope so, will try different combinations of how to configure so metaflow pulls those packages first
b
thanks, please let me know
d
just following up a bit here (trying to understand as well). On your remote node, why does it need to have access to pip. If the environment is resolved, shouldn’t it no longer need to access pip?
h
Hi @dry-beach-38304, thanks for the great question, I guess my explanation is not precise enough, due to my lack of understanding of how metaflow is resolving the environment. long story short, using 
pip install <privatepackage>
works just fine, but running the flow.py doesn't. I think @brainy-truck-72938 is right that it is because the artifactregistry-auth keyring is missing at the time of execution and installing that package from inside the flow would probably solve it. After a bit more experimentation, it is still failing as it's asking for the authentication for the private repo before even pulling from pypi (see https://stackoverflow.com/questions/67253141/python-pip-priority-order-with-index-url-and-extra-index-url). Is there a way to cheat this with multiple @pypi decorators for the flow and therefore set up a form of "dependency graph"?
so to summarize, the environment never gets resolved when using metaflow in this case due to the fact that it is not able to authenticate to the private repo by using application default credentials from within the script
d
Let me make sure I understand: • you have a flow as you described above • running the flow locally works fine • running the flow with 
--with kubernetes
does not • inside the flow, you are NOT doing any additional pip install is that ocrrect?
h
no, the flow never runs, so I guess local vs remote doesn't make a difference. I am NOT doing any pip install inside the flow. as soon as I try to run the flow without specifying credentials in 
.pypirc
and 
pip.conf
it fails. However, if I do pip install locally on the computer, it will install the private package without an issue.
the keyring packages are installed locally on the computer, and therefore it is using ADC flawlessly via that method. unfortunately, trying to reproduce that keyring installation from within the flow with the @pypi_base decorator doesn't work.
d
got it
(it makes more sense that it doesn’t work for a local execution either)
🙌 1
b
as a side experiment, you can also uninstall 
keyrings.google-artifactregistry-auth
locally and check that it will STOP working
h
I already know it will stop working @brainy-truck-72938, it didn't work before I installed them 😄
🆒 1
d
ok now I get it (sorry I was being slow). So the two packages 
keyrign
and 
keyrings.google-artifactregistry-auth
are the two that enable this thing to work and somehow, when using the 
pypi
decoraotr, they are not there and so it is failing.
h
the more I understand how all of this works, I think you have the right solution, sadly it won't work because there is no easy way for me to enforce a priority index to the pypi repos and therefore it asks for the authentication to the private repo before even installing the keyring packages 😞
correct @dry-beach-38304
kind of like a circular dependency now
b
yeah, maybe we can inject 
keyrings.google-artifactregistry-auth
dependency somewhere when using GCP
h
if there was a way to do @pypi_base and @pypi_overlay of sorts 🙂
d
it should be solveable. If pip runs in the outside environment, it should work (since there the packages are already present). If not, it needs to build an env first and then use that env to resolve the full one.
this 1
h
while testing this, is there a way to add verbosity to the execution? like "resolving 1 environment" is doing something in the background but not displaying until it has timed out
d
If you install the bleeding edge extension, you can do 
METAFLOW_DEBUG_CONDA=1
and it will give you a lot of details. I’d actually bei nterestered in what it says there if you don’t mind trying it out.
h
happy to! can you point me to a doc on how to install the said extension? or just tell me what to do?
d
h
hah, already installed then
wow, I wish I would have known that earlier haha!
just setting that environment variable worked like charm
d
what isthe output? of that (It’s long so you may need a gist or something — make sure it doesn’t contain any private info, I just care about the gist of what it is doing. It should say what is it calling)
(this debug flag comes in very handy trying to help users internally 🙂 )
h
'-i', '<https://us-python.pkg.dev/privaterepo>', '--extra-index-url', '<https://pypi.org/simple>',
that seems to be the issue
it is resolving our private repo as the "main index"
d
the part I am a bit confused about is that that resolve should be running in an enviornment that already has the proper packages no?
if it’s a matter of flipping them, that should be do-able
h
no, I deleted the environment to make sure that flow started fresh
Copy code
[global]

index-url = <https://pypi.org/simple>
extra-index-url = <https://us-python.pkg.dev/privaterepo>
this is my 
pip.conf
.pypirc
is:
Copy code
[distutils]
index-servers =
    general
    privaterepo

[general]
repository: <https://pypi.org/simple>

[privaterepo]
repository: <https://us-python.pkg.dev/privaterepo>
so the run gets locked forever, until I press ctrl+c and this is what comes out after
Copy code
Looking in indexes: <https://us-python.pkg.dev/privaterepo>, <https://us-python.pkg.dev/privaterepo>, <https://pypi.org/simple>
Keyring provider requested: auto
Keyring provider set: subprocess with executable /home/metaflow-user/miniforge-pypy3/bin/keyring
User for us-python.pkg.dev: Collecting keyring==24.3.1
  Obtaining dependency information for keyring==24.3.1 from <https://files.pythonhosted.org/packages/7c/23/d557507915181687e4a613e1c8a01583fd6d7cb7590e1f039e357fe3b304/keyring-24.3.1-py3-none-any.whl.metadata>
  Using cached keyring-24.3.1-py3-none-any.whl.metadata (20 kB)
User for us-python.pkg.dev: Collecting keyrings.google-artifactregistry-auth-gcloud==1.1.4
  Obtaining dependency information for keyrings.google-artifactregistry-auth-gcloud==1.1.4 from <https://files.pythonhosted.org/packages/26/0c/f30848a1d290eda52b8534e8c6409177f5958df0ed5b1d984214fe423cc4/keyrings.google_artifactregistry_auth_gcloud-1.1.4-py3-none-any.whl.metadata>
  Using cached keyrings.google_artifactregistry_auth_gcloud-1.1.4-py3-none-any.whl.metadata (4.6 kB)
User for us-python.pkg.dev:   Link requires a different Python (3.9.0 not in: '>=3.6, <3.9'): <https://files.pythonhosted.org/packages/d5/ac/dda6bfd193975437575cdf170bb371570a802579e50b1b79d46eb12ea350/google-cloud-bigquery-2.6.2.tar.gz> (from <https://pypi.org/simple/google-cloud-bigquery/>) (requires-python:>=3.6, <3.9)
  Link requires a different Python (3.9.0 not in: '>=3.6, <3.9'): <https://files.pythonhosted.org/packages/d9/ac/48b6e72972ae1949774b8bf84935640c1f1a5827780a7160f9b8964bb368/google_cloud_bigquery-2.6.2-py2.py3-none-any.whl> (from <https://pypi.org/simple/google-cloud-bigquery/>) (requires-python:>=3.6, <3.9)
Collecting google-cloud-bigquery==3.11.4
  Obtaining dependency information for google-cloud-bigquery==3.11.4 from <https://files.pythonhosted.org/packages/cc/6a/d0ef792288f2fa2cfea80899a82de302b3332dfda41984fe114e2cfbf700/google_cloud_bigquery-3.11.4-py2.py3-none-any.whl.metadata>
  Using cached google_cloud_bigquery-3.11.4-py2.py3-none-any.whl.metadata (8.5 kB)
User for us-python.pkg.dev: Collecting google-cloud-logging==3.8
  Obtaining dependency information for google-cloud-logging==3.8 from <https://files.pythonhosted.org/packages/b0/90/08669c4471bf9d9cfd51b5533c3b197a2c19f36c65747ee82f522275f9cc/google_cloud_logging-3.8.0-py2.py3-none-any.whl.metadata>
  Using cached google_cloud_logging-3.8.0-py2.py3-none-any.whl.metadata (4.8 kB)
User for us-python.pkg.dev: Collecting google-cloud-storage==2.12,>=2.5.0
  Obtaining dependency information for google-cloud-storage==2.12,>=2.5.0 from <https://files.pythonhosted.org/packages/68/b8/cb00819641313e67b640857ffff2c5afdcdcfb4940def9f1502bf614a6d9/google_cloud_storage-2.12.0-py2.py3-none-any.whl.metadata>
  Using cached google_cloud_storage-2.12.0-py2.py3-none-any.whl.metadata (6.1 kB)

Pretty-printed STDERR:
WARNING: 401 Error, Credentials not correct for <https://us-python.pkg.dev/privaterepo/keyring/>
WARNING: 401 Error, Credentials not correct for <https://us-python.pkg.dev/privaterepo/keyrings-google-artifactregistry-auth-gcloud/>
WARNING: 401 Error, Credentials not correct for <https://us-python.pkg.dev/privaterepo/google-cloud-bigquery/>
WARNING: 401 Error, Credentials not correct for <https://us-python.pkg.dev/privaterepo/google-cloud-logging/>
WARNING: 401 Error, Credentials not correct for <https://us-python.pkg.dev/privaterepo/google-cloud-storage/>
pretty sure that confirms that it is because it is not able to get keyring and the auth packages prior to that?
d
could you paste a bit more of the log. It does seem to do something with keyring early on and it does seem to be finding some packages but not quite sure yet what is fully going on.
h
Untitled
Untitled
lines 45-50 was me pressing enter to see if I would get a prompt of sort, then I pressed ctrl-c as it was stuck there
and to prove the point about priority, the run there doesn't even have private packages in @pypi_base, I removed all to test the theory
so technically all of those packages should be able to get installed
d
okie — this is excellent 🙂. Now it makes sense.
🙌 1
ok, let’s try a hackish experiment to see if this will fix it
in 
/home/metaflow-user/miniforge-pypy3/lib/pypy3.9/site-packages/metaflow_extensions/netflix_ext/plugins/conda/utils.py
, you should see a line (around 73) that says 
_BUILDER_ENVS_PACKAGES
. Could you add the two packages there (keyring and the google auth one)?
👀 1
(it currently has pip, wheel, tomli and setuptools.
it would kind of break this comment I have somewhere lese:
Copy code
# We add a few more packages that we need to support building wheels
        # Conda typically includes pip but no harm adding it there too
        # All packages included are miniscule and have no dependencies.
but if it works, I can see about making this configurable so you can choose to load those other packages if needed.
h
that worked! let me re-add the private packages to make sure
b
so the core issue was indeed missing 
keyrings.google-artifactregistry-auth
, right? irrespective of how we are kinda adding it..
h
I believe the core issue is more that pypi doesn't support index prioritization in the chain of repositories, but in regards to why the code didn't work to build the environment, yes @brainy-truck-72938, 100% was because the 
keyring
and 
keyrings.google-artifactregistry-auth
packages were not part of the flow itself in the first place. My original misconception was that since it was installed locally, metaflow should be able to use it intrinsically. Then after that, it pointed us to the whole prioritization of repos.
👍 1
while I was typing this, the flow completed successfully with the proper private packages installed!!!
🎉 1
among us party 1
🎉 1
b
@hundreds-midnight-75494 can you please comment on https://github.com/Netflix/metaflow/issues/1658 confirming the same
👍 1
d
Just to provide a tad more color to this. Due to the way pip does its resolution (in particular, it says that you can pass it the version of python you want to resolve for but it doesn’t actually work), to resolve the environment, we first build a temporary environment with conda that has the right version of pip/python that the user is requesting (in this case 3.9). That environment is then used to run 
pip …
to do the resolution. That environment doesn’t have the required packages to do the auth. The “fix” I mentioned was simply to add those packages to that builder environment and then install it from there.
h
yeah that totally makes sense, now that I understand more of the functionality behind it all
d
pip indeed doesn’t support index prioritization but I think the core issue was this temporary builder environment. I believe the OSS implementation does something similar to what I do in the Netflix extension that you are using (i can’t recall for sure but am pretty sure since I had told Savin/Sakari that pip doesn’t actually respect the python version you are requesting when resolving).
I’ll try to make a quick fix (or if you want to propose one as well) to make this configurable. I don’t necessarily want to include all those packages in the builder environments by default because they are not lightweight packages but I feel that having some overload from the user would make sense.
h
totally agree that it doesn't make sense to include unless it is required! maybe having some form of @pypi_prereq or something?
or making it a configuration item in metaflow.conf?
d
alternatively, is the presence of an env variable (I saw something about 
GOOGLE_APPLICATION_CREDENTIALS
sufficient to say this is needed)?
this 2
I was thinking in metaflow conf yes
something like 
METAFLOW_CONDA_BUILDER_PACKAGES
?
h
if you find a value assigned to 
GOOGLE_APPLICATION_CREDENTIALS
, it would be a great indicator.
I am also ok with adding 
METAFLOW_CONDA_BUILDER_PACKAGES
, it would give more flexibility but would be potentially harder to discover for the users
b
@dry-beach-38304 maybe we can write those suggestions here: https://github.com/Netflix/metaflow/issues/1658
h
d
Haven’t released but let me know if this fixes it: https://github.com/Netflix/metaflow-nflx-extensions/pull/35
✅ 1
h
sweet, thanks @dry-beach-38304, I just downloaded the code and will test now
b
@hundreds-midnight-75494 https://github.com/Netflix/metaflow/pull/1772 is also merged so it should also work on the OSS side of things..
🙌 1
h
@dry-beach-38304, I confirm this code fixes the problem!
👍 1
Open in Slack
PreviousNext
Powered by