Similar to a few others here, a flow I could run yesterday isn’t working anymore due to a failure during the environment bootstrap step. Haven’t changed anything on my side. I’m using the @pypi_base decorator in my flow to specify my dependencies.

Metaflow 2.10.8+netflix-ext(1.1.1) executing TestHCC for user:gracemoore
Validating your flow...
    The graph looks good!
Bootstrapping Conda environment... (this could take a few minutes)
    Resolving 1 environment ...    Internal errorTraceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/cli.py", line 1175, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/cli.py", line 692, in wrapper
    return func(args, kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/cli.py", line 830, in run
    before_run(obj, tags, decospecs + obj.environment.decospecs())
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/cli.py", line 888, in before_run
    obj.package = MetaflowPackage(
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/package.py", line 70, in __init__
    environment.init_environment(echo)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda_environment.py", line 139, in init_environment
    resolver.resolve_environments(echo)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py", line 227, in resolve_environments
    self._resolve_environments(echo, need_resolution)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/envsresolver.py", line 603, in _resolve_environments
    cached_resolved_env = self._conda.environment(resolved_env_id)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 579, in environment
    env = self._remote_env_fetch([env_id])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 2069, in _remote_env_fetch
    with self._storage.load_bytes(result.keys()) as loaded:
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/storage_executor.py", line 155, in inner_function
    return func(args, kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/datastores/gs_storage.py", line 261, in load_bytes
    items = [future.result() for future in as_completed(futures)]
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/datastores/gs_storage.py", line 261, in <listcomp>
    items = [future.result() for future in as_completed(futures)]
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(self.args, self.kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/datastores/gs_storage.py", line 129, in load_bytes_single
    blob = self.get_blob_client(key)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/datastores/gs_storage.py", line 62, in get_blob_client
    bucket = self.get_bucket_client()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/datastores/gs_storage.py", line 58, in get_bucket_client
    client = get_gs_storage_client()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/metaflow/plugins/gcp/gs_storage_client_factory.py", line 14, in get_gs_storage_client
    from google.cloud import storage
ModuleNotFoundError: No module named 'google'

I saw others had success by downgrading micromamba, but I’m using conda. I also tried using a local environment, but that apparently isn’t compatible with the @pypi_base decorator. Does anyone have any ideas?

hey. I see you are using the netflix conda extension (which is nice 🙂 — glad to see ppl using it). It seems like it can’t find stuff on GCP. Are you saying your flow ran fine before and now doesn’t anymore with no change to the versions of metaflow, extensions or your code?

Are you saying your flow ran fine before and now doesn’t anymore with no change to the versions of metaflow, extensions or your code?

That’s correct, to the best of my knowledge. I’m running this locally without Docker so no base image to speak of!

so in your outside environment (ie: if just running

python

), what happens if you do

from google.cloud import storage

ModuleNotFoundError 👀

so that’s the issue. Somehow, it’s missing that. Let me see what package should be installed. one sec.

pip install google-cloud-storage

?

looks like you need:

google-cloud-storage>=2.5.0
google-auth>=2.11.0

based on waht I can see in the code.

Weird, it’s working now! Thank you for the quick replies. it was running yesterday just based on the packages I had listed in

pypi_base

:

packages={
        "google-cloud-bigquery": "==3.14.1",
        "google-cloud-storage": "==2.14.0",
        "numpy": "==1.26.3",
        "pandas": "==2.1.4",
        "pyarrow": "==14.0.2",
}

it is working again now after running

pip install google-cloud-storage

and I don’t really understand how that solved it, given the boostrap environment should be distinct from my local environment!

so you are right that the bootstrapped env is different