Has anybody else noticed odd import errors sporadi...
# ask-metaflowi
important-london-94970
02/11/2025, 3:40 PMHas anybody else noticed odd import errors sporadically popping up when launching flows? I've been dealing with this off an on and I'm not entirely sure why, but running
rm -rf micromamba/ Copy code
2025-02-11 15:29:04.020 Bootstrapping virtual environment(s) ...
2025-02-11 15:30:48.921 Virtual environment(s) bootstrapped!
2025-02-11 15:30:49.337 Workflow starting (run-id 89):
2025-02-11 15:30:50.113 [89/start/407 (pid 3854)] Task is starting.
2025-02-11 15:30:50.252 [89/start/407 (pid 3854)] Traceback (most recent call last):
2025-02-11 15:30:50.252 [89/start/407 (pid 3854)] File "/tmp/tmpgzdfkyvw/metaflow/extension_support/plugins.py", line 99, in get_plugin
2025-02-11 15:30:50.253 [89/start/407 (pid 3854)] plugin_module = importlib.import_module(path)
2025-02-11 15:30:50.253 [89/start/407 (pid 3854)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:30:50.253 [89/start/407 (pid 3854)] File "/home/jovyan/micromamba/envs/metaflow/linux-64/d4c8f997b8c5837/lib/python3.12/importlib/__init__.py", line 90, in import_module
2025-02-11 15:30:50.253 [89/start/407 (pid 3854)] return _bootstrap._gcd_import(name[level:], package, level)
2025-02-11 15:30:50.253 [89/start/407 (pid 3854)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "<frozen importlib._bootstrap_external>", line 999, in exec_module
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] File "/tmp/tmpgzdfkyvw/metaflow/plugins/aws/batch/batch_decorator.py", line 4, in <module>
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] import requests
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] ModuleNotFoundError: No module named 'requests'
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)]
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] The above exception was the direct cause of the following exception:
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)]
2025-02-11 15:30:50.254 [89/start/407 (pid 3854)] Traceback (most recent call last):
2025-02-11 15:30:50.255 [89/start/407 (pid 3854)] File "/home/jovyan/mnist/train.py", line 11, in <module>
2025-02-11 15:30:50.255 [89/start/407 (pid 3854)] from metaflow import (
2025-02-11 15:30:50.255 [89/start/407 (pid 3854)] File "/tmp/tmpgzdfkyvw/metaflow/__init__.py", line 114, in <module>
2025-02-11 15:30:50.255 [89/start/407 (pid 3854)] from .plugins.datatools import S3
2025-02-11 15:30:50.280 [89/start/407 (pid 3854)] File "/tmp/tmpgzdfkyvw/metaflow/plugins/__init__.py", line 171, in <module>
2025-02-11 15:30:50.280 [89/start/407 (pid 3854)] STEP_DECORATORS = resolve_plugins("step_decorator")
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] File "/tmp/tmpgzdfkyvw/metaflow/extension_support/plugins.py", line 164, in resolve_plugins
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] to_return.append(get_plugin(category, class_path, name))
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] File "/tmp/tmpgzdfkyvw/metaflow/extension_support/plugins.py", line 101, in get_plugin
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] raise ValueError(
2025-02-11 15:30:50.281 [89/start/407 (pid 3854)] ValueError: Cannot locate step_decorator plugin 'batch' at 'metaflow.plugins.aws.batch.batch_decorator'
2025-02-11 15:30:50.346 [89/start/407 (pid 3854)] Task failed.
2025-02-11 15:30:50.401 Workflow failed.
2025-02-11 15:30:50.402 Terminating 0 active tasks...
2025-02-11 15:30:50.402 Flushing logs...
Step failure:
Step start (task-id 407) failed.✅ 1
s
square-wire-39606
02/11/2025, 3:41 PMinteresting - which version of metaflow are you on? also, are you using any extensions?
i
important-london-94970
02/11/2025, 3:43 PMThis is what I have installed, but I get these errors both when I use the extensions and when I don't. I am running this on a jupyterhub user pod in k8s if that helps. The pod has 20Gi of storage.
Copy code
metaflow==2.13
metaflow-ray==0.1.4
metaflow-torchrun==0.0.9important-london-94970
02/11/2025, 3:50 PMThe import error is usually related to some aws submodule. I just got this different import error:
Copy code
2025-02-11 15:46:29.374 Bootstrapping virtual environment(s) ...
2025-02-11 15:47:39.447 Virtual environment(s) bootstrapped!
2025-02-11 15:47:39.789 Workflow starting (run-id 93):
2025-02-11 15:47:40.551 [93/start/421 (pid 4286)] Task is starting.
2025-02-11 15:47:40.675 [93/start/421 (pid 4286)] Traceback (most recent call last):
2025-02-11 15:47:40.675 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/extension_support/plugins.py", line 99, in get_plugin
2025-02-11 15:47:40.675 [93/start/421 (pid 4286)] plugin_module = importlib.import_module(path)
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] File "/home/jovyan/micromamba/envs/metaflow/linux-64/0516ef7640c9358/lib/python3.12/importlib/__init__.py", line 90, in import_module
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] return _bootstrap._gcd_import(name[level:], package, level)
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
2025-02-11 15:47:40.676 [93/start/421 (pid 4286)] File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] File "<frozen importlib._bootstrap_external>", line 999, in exec_module
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/plugins/aws/step_functions/step_functions_decorator.py", line 8, in <module>
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] from .dynamo_db_client import DynamoDbClient
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/plugins/aws/step_functions/dynamo_db_client.py", line 4, in <module>
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] import requests
2025-02-11 15:47:40.677 [93/start/421 (pid 4286)] ModuleNotFoundError: No module named 'requests'
2025-02-11 15:47:40.700 [93/start/421 (pid 4286)]
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] The above exception was the direct cause of the following exception:
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)]
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] Traceback (most recent call last):
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] File "/home/jovyan/mnist/metaflow_ray_train.py", line 21, in <module>
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] from metaflow import (
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/__init__.py", line 114, in <module>
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] from .plugins.datatools import S3
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/plugins/__init__.py", line 171, in <module>
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] STEP_DECORATORS = resolve_plugins("step_decorator")
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/extension_support/plugins.py", line 164, in resolve_plugins
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] to_return.append(get_plugin(category, class_path, name))
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] File "/tmp/tmpmv58w2ly/metaflow/extension_support/plugins.py", line 101, in get_plugin
2025-02-11 15:47:40.701 [93/start/421 (pid 4286)] raise ValueError(
2025-02-11 15:47:40.702 [93/start/421 (pid 4286)] ValueError: Cannot locate step_decorator plugin 'step_functions_internal' at 'metaflow.plugins.aws.step_functions.step_functions_decorator'
2025-02-11 15:47:40.763 [93/start/421 (pid 4286)] Task failed.
2025-02-11 15:47:40.830 Workflow failed.
2025-02-11 15:47:40.830 Terminating 0 active tasks...
2025-02-11 15:47:40.830 Flushing logs...
Step failure:
Step start (task-id 421) failed.a
ancient-application-36103
02/11/2025, 4:31 PMcan you try upgrading to the latest version of metaflow and give it another shot?
i
important-london-94970
02/11/2025, 4:32 PMYes
important-london-94970
02/11/2025, 4:43 PMThat worked on the first try. I'll reply back here if the issue creeps up again
a
ancient-application-36103
02/11/2025, 4:43 PMthank you! i am happy to jump on a quick call whenever that happens if you are open to it so that we can debug live
🙌🏻 1
5 Views