< U01NGJ0GA0J> < straight shampoo 11124> I have one pipeline Outerbounds #ask-metaflow

<@U01NGJ0GA0J> <@U01UL7YFXDE> I have one pipeline ...

curved-island-17262

06/07/2023, 6:39 AM

@User @straight-shampoo-11124 I have one pipeline that always runs into this issue

bash: line 1: metaflow_SampleFlow_linux-64_2baa982ca7df8a31823ae7a84ba9e28d8b8f2285/bin/python: No such file or directory

even with using mamba on

Metaflow 2.9.3

. I tried pinpointing the issue and it seems that it always happens when I install

scikit-learn

as a dependency. This Flow will always throw the same error at the

start

step:

Copy code

from metaflow import FlowSpec, conda, conda_base, step

@conda_base(
    python="3.9.16",
    libraries={
        "pandas": "2.0.2",
        "pyyaml": "6.0",
        "loguru": "0.6.0",
        "pyarrow": "12.0.0",
        "psutil": "5.9.5",
        "snowflake-connector-python": "3.0.4",
    },
)
class SampleFlow(FlowSpec):
    @conda(libraries={"scikit-learn": "1.2.2"})
    @step
    def start(self):
        self.next(self.end)

    @step
    def end(self):
        ...


if __name__ == "__main__":
    SampleFlow()

This Flow will work which uses `@pip`:

Copy code

from metaflow import FlowSpec, conda, conda_base, project, step

@conda_base(
    python="3.9.16",
    libraries={
        "pandas": "2.0.2",
        "pyyaml": "6.0",
        "loguru": "0.6.0",
        "pyarrow": "12.0.0",
        "psutil": "5.9.5",
        "snowflake-connector-python": "3.0.4",
    },
)
class SampleFlow(FlowSpec):
    @pip(libraries={"scikit-learn": "1.2.2"})
    @step
    def start(self):
        self.next(self.end)

    @step
    def end(self):
        ...


if __name__ == "__main__":
    SampleFlow()

Any idea why this is happening?

square-wire-39606

06/07/2023, 1:51 PM

let me try to reproduce and get back to you

curved-island-17262

06/12/2023, 12:41 AM

@ancient-application-36103 was this reproducible?

curved-island-17262

06/12/2023, 2:09 PM

I ran into a new error based on the same issue with the

env_id

today:

Copy code

2023-06-12 22:04:55.674 [8999/scipy/64991 (pid 10658)] AWS Batch error:
2023-06-12 22:04:55.674 [8999/scipy/64991 (pid 10658)] Essential container in task exited This could be a transient error. Use @retry to retry.
2023-06-12 22:04:55.809 [8999/scipy/64991 (pid 10658)] 
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]     return _run_code(code, main_globals, None,
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]     exec(code, run_globals)
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 104, in <module>
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]     bootstrap_environment(sys.argv[1], sys.argv[2], sys.argv[3])
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 16, in bootstrap_environment
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]     packages = download_conda_packages(flow_name, env_id, datastore_type)
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 41, in download_conda_packages
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40]     env = json.load(f)[env_id]
2023-06-12 22:04:37.428 [8999/scipy/64991 (pid 10658)] [27467ff5-2728-43cb-81a2-9e455bdbec40] KeyError: 'metaflow_SampleFlow_linux-64_41adab099e1e33e5052281a592e81c6a50e6cc44'

✅ 1

curved-island-17262

06/12/2023, 2:16 PM

Changing a library’s version to a lower one fixed it but still baffled as to why is happening as the library version is available on Conda.

ancient-application-36103

06/12/2023, 3:29 PM

interesting. which lib's version did you have to downgrade?

curved-island-17262

06/12/2023, 3:36 PM

scipy

from

1.10.1

1.9.3

ancient-application-36103

06/12/2023, 4:05 PM

interesting..

curved-island-17262

06/16/2023, 12:42 AM

@ancient-application-36103 is there anything I could do to debug this issue? My entire team is facing this issue on a daily basis. It comes and goes sporadically.

5 Views

Open in Slack

Previous Next