https://outerbounds.com/ logo
Join Slack
Powered by
Hi folks -- we are evaluating Metaflow and having ...
# ask-metaflow
f
Hi folks -- we are evaluating Metaflow and having issues with the dependency management. After much fiddling, we have it working with our private GCP Artifact Repository with Metaflow + Netflix extensions. Would like to know how to get Metaflow to just read from either the user's 
pip.conf
or 
.pypirc
or explicitly point to a 
pip.conf
file. Right now, I have token directly in the decorator (I couldn't seem to get keyring authorization to work). The more blocking issue is that the flow doesn't run when submitted to the Kubernetes cluster: we are getting a 
ModuleNotFoundError: No module named 'metaflow'
error when issuing the following: 
python example-flow.py --environment conda --with kubernetes run
This works locally: 
python example-flow.py --environment conda   run
Running with Metaflow and Netflix extensions: 
Metaflow 2.10.7+netflix-ext(1.0.7)
d
hey @freezing-doctor-60390, it should read the repo info from the pip configuration. how is your token stored in your config file? For the other command, Could you give the full traceback. I also believe that the syntax should be: 
python example-flow.py --environment conda run --with kubernetes
(notice the order).
f
~/.pip/pip.conf
Copy code
[global]

extra-index-url = <https://_json_key_base64:TOKEN@us-python.pkg.dev/gcp_project/repo_name/simple/>
Is there an environment var to set? Currently, it doesn't work if I don't use the 
extra_indices
param with the 
@pypi
decorator.
d
that’s very weird. The code here: https://github.com/Netflix/metaflow-nflx-extensions/blob/main/metaflow_extensions/netflix_ext/plugins/conda/conda.py#L207 will basically run 
pip config list
and then look for all 
index-url
and 
extra-index-url
. If you run that do you get the values you expect?
f
I went ahead for now to put the token in the decorator -- will come back to try and test the above -- is there logging or anything that can be enabled? Here's the output when submitting to kubernetes -- tried it with your order and same results:
Copy code
$ python example-flow.py --environment conda run --with kubernetes

Metaflow 2.10.7+netflix-ext(1.0.7) executing ExampleFlow for user:metaflow-user
Validating your flow...
  The graph looks good!
Running pylint...
  Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
  All packages already cached in gs.
  All environments already cached in gs.
2023-12-01 20:54:00.498 Workflow starting (run-id 65):
2023-12-01 20:54:01.585 [65/start/502 (pid 18761)] Task is starting.
2023-12-01 20:54:01.664 [65/start/502 (pid 18761)] Traceback (most recent call last):
2023-12-01 20:54:01.665 [65/start/502 (pid 18761)] File "/mnt/metaflow-haus-examples/example/example-flow.py", line 1, in <module>
2023-12-01 20:54:01.665 [65/start/502 (pid 18761)] from metaflow import FlowSpec, card, pypi, pypi_base, step, current
2023-12-01 20:54:01.665 [65/start/502 (pid 18761)] ModuleNotFoundError: No module named 'metaflow'
2023-12-01 20:54:01.754 [65/start/502 (pid 18761)] Task failed.
2023-12-01 20:54:01.834 Workflow failed.
2023-12-01 20:54:01.834 Terminating 0 active tasks...
2023-12-01 20:54:01.834 Flushing logs...
  Step failure:
  Step start (task-id 502) failed.
d
to debug: 
METAFLOW_DEBUG_CONDA=1
can be set as an env var and that will give you a lot of detail. I have never seen this particular error but then again, I don’t run with kubernetes. Let me see if there is anything kube specific or if I can get some help on running on kubernetes to debug this.
f
To simplify a bit, I ran the "hello world" from the tutorial and just added the 
--environment conda
switch and got the same thing (It runs successfully without the 
--environment conda
switch when submitted to Kubernetes.):
Copy code
$ python 00-helloworld/helloworld.py --environment conda run --with kubernetes

Metaflow 2.10.7+netflix-ext(1.0.7) executing HelloFlow for user:metaflow-user
Validating your flow...
  The graph looks good!
Running pylint...
  Pylint is happy!
Bootstrapping Conda environment... (this could take a few minutes)
  All packages already cached in gs.
  All environments already cached in gs.
2023-12-01 21:02:49.659 Workflow starting (run-id 69):
2023-12-01 21:02:50.532 [69/start/512 (pid 19066)] Task is starting.
2023-12-01 21:02:50.607 [69/start/512 (pid 19066)] Traceback (most recent call last):
2023-12-01 21:02:50.607 [69/start/512 (pid 19066)] File "/mnt/metaflow-haus-examples/00-helloworld/helloworld.py", line 1, in <module>
2023-12-01 21:02:50.608 [69/start/512 (pid 19066)] from metaflow import FlowSpec, step
2023-12-01 21:02:50.608 [69/start/512 (pid 19066)] ModuleNotFoundError: No module named 'metaflow'
2023-12-01 21:02:50.687 [69/start/512 (pid 19066)] Task failed.
2023-12-01 21:02:50.763 Workflow failed.
2023-12-01 21:02:50.763 Terminating 0 active tasks...
2023-12-01 21:02:50.763 Flushing logs...
  Step failure:
  Step start (task-id 512) failed.
d
ah that does simplify it. I don’t have ready access to a kube cluster but I’ll look at the code and also ask someone to run this for me since this is easily reproducible and see what may be happening there.
f
Thank you -- any help would be appreciated and let me know if you need more details. Thanks for the tip on the debug flag!
d
(it’s verbose but you should get what you want from that)
just to clarify, you are running with vanilla oss kubernetes right?
f
GKE on GCP using your starter setup instructions. from Outerbounds -- didn't fully work as is so tweaked by using official Netflix releases
d
images?
oh, for the metadata service. got it.
this is actually not even leaving your box yet. it’s not even going off to kubernetes yet. OK, I may be albe to try this out myself too. I’ll keep you posted.
🙏 1
f
Quick update here: was able to get this working locally and submitting to k8s with base tutorial example and downloading a custom library from a private repo but without anything happening in the steps. Had to be more careful with how Python environments were being managed (now working off the miniforge environment exclusively for conda and mamba binaries. Now that I have a cleaner environment setup: will try to hone in on why pip.conf isn't being picked up.
d
Hey. I haven’t looked at this yet. What was the change?
I use mini forge for mamba and conda (the one from here which I guess is the same): https://github.com/conda-forge/miniforge
I am very confused as to the “no metaflow” error. Are you saying this stopped happening when you changed conda binary?
Did you run in debug mode to see what values it is getting?
f
For context, doing all of this inside a Docker container on MacOS to ensure a clean environment. Specifically I did the following: I updated PATH to ensure that the miniforge mamba, conda, python, pip were at the front of the path. Prior to making this change, I had the path to the miniforge items last in the PATH, which meant it would pick up the system versions of those binaries (I had also installed conda separately and am no longer doing that). I also make sure that I am running everything in the context of mamba's base environment -- not sure if this is directly related but trying to be as consistent as possible. To resolve the private repo issue, I copied my 
~/.pip/pip.conf
to the following locations (locations specified by pip when running `pip config list)`: 
/etc/pip.conf
, 
/etc/xdg/pip/pip.conf
, 
~/.config/pip/pip.conf
, 
~/miniforge-pypy3/pip.conf
Of the above, I haven't done the process of elimination to find out which one(s) would make the difference but I suspect it's one of the global paths. If I get a chance, I'll try running debug for the original situation but focused on moving forwards now as we are now experiencing other issues that are dependency specific (encountering 
CmdStan failed to install in repackaged directory
when building wheel for a lib).
d
Ok. Are the issues you are facing specific to metaflow’s conda/pypi implementation or external to it. If you face any issues with anything metaflow, feel free to ping this thread. Happy to help.
🙏 1
42 Views
Open in Slack
PreviousNext
Powered by