https://outerbounds.com/ logo
Join Slack
Powered by
Hi, I am currently trying to debug, why our Metaf...
# ask-metaflow
e
Hi, I am currently trying to debug, why our Metaflow steps sometimes are slow during "bootstrap of virtual environment". Is there any debugging variable I can set to get more logging out of the bootstrapping process? I am trying to locate any potential bottleneck in network, or k8s cluster resources. In the attached screenshot the bootstrapping takes minutes, but other times it is closer to 20 seconds We are using the --environment=pypi flag and the pypi decorator in the example
f
Hey Casse! can you share any additional info on what you're bootstrapping to help narrow down the issue? looks like you're using conda, if you can share what your step decorators look like it may help. I don't think there's any verbose setting but there are a few env vars you can set 
CONDA_DEPENDENCY_RESOLVER
and 
CONDA_USE_FAST_INIT
. The former defaults to 
conda
but 
mamba
in my experience has been a lot faster (not sure if this is still true). https://github.com/Netflix/metaflow/blob/1e5ccee/metaflow/metaflow_config.py#L482-L488
There's also https://github.com/Netflix/metaflow-nflx-extensions , which is rock solid and I've used for years across a few different orgs Here's the relevant bits from the README.md:
Copy code
Main improvements over the standard Conda decorator in Metaflow

This decorator improves several aspects of the included Conda decorator:

    it allows you to mix and match Conda packages and Pypi packages.
    it supports a wider range of Pypi package sources (repositories, source tarballs, etc)
    it supports a command line tool allowing you to:
        retrieve and re-hydrate any environment used by any previously executed step thereby enabling an easy way to inspect artifacts created in that environment
        resolve environments using standard requirements.txt of environment.yml files
        inspect packages present in any environment previously resolved
    it supports "named environments" which enables easy environment sharing and saving.
    it is generally more performant and efficient in terms of parallel resolution and downloading of packages
    it supports conda, mamba and micromamba
it's a drop in replacement and gives you a lot more levers to pull to optimize the env bootstrapping, and comes with a nice CLI that gives you the ability to build environments and persist them, the 
named environments
bit above
if you're able to, I'd recommend going the netflix-ext route, Romain knocked it out of the park with this one
e
We are just using something simple as the @pypi_base decorator, where packages are read from requirements.txt. Example extract:
Copy code
reqs = requirements_txt_parser("requirements.txt")

@pypi_base(python="3.12", packages=reqs)
class Exmpleflow(DecisionFlow):

    @step
    def model_start(self):
        self.next(self.end)
Note that bootstrapping is not always multiple minutes long. Seems to be around "peak load", when other flows are running at the same time. I would really like to know if I can improve the infrastructure/networking, if they are causing a bottleneck. And perhaps more logging during the bootstrapping process could highlight any bottleneck
f
is there variance with the kubernetes step resources or is that constant? since you're using pypi, you can try a few things to try to improve bootstrap behavior: • try conda (if packages are available there), then you can tweak 
CONDA_DEPENDENCY_RESOLVER
and 
CONDA_USE_FAST_INIT
to see of that helps things • try uv : https://docs.metaflow.org/scaling/dependencies/uv • install https://pypi.org/project/metaflow-netflixext/ netlixext exposes a ton of things that could help in this case: https://github.com/Netflix/metaflow-nflx-extensions?tab=readme-ov-file#configuration and Romain is pretty responsive on here for that too
you should be able to pop the netflixext in there and use the existing requirements.txt
a
@early-nest-89176 I need to double check, but there might be some log buffering happening which has the implication that the delta between timestamps isn’t necessarily indicating the latency of bootstrap
👍 1
d
If you go the extension route, it does provide METAFLOW_DEBUG_CONDA. If you set this to one (may need to pass it through @environment (forgot for OSS), you will get a lot of debugging messages.
And thanks @flaky-plumber-70709 — glad you find it useful. Should see some new things soon too :).
f
it's amazing, and probably only have used a small subset of features over the years but for every place I've worked where I've stood up OSS metaflow its basically solved dependency issues for data scientists with out them even knowing about it
e
@ancient-application-36103 Did you come to a conclusion on the log deltas? I am really just looking for whatever bottleneck we could have, and could be great to rule the bootstrapping out 🙂
a
sorry haven't found the time yet
👍 1
31 Views
Open in Slack
PreviousNext
Powered by