Just upgraded metaflow and use:

python SomeFlow.py --with batch:cpu=16,memory=30000,queue=bla-684414486554-55ff636 --environment=conda run

but get:

Metaflow 2.12.19 executing MetaFlow_CDCNSLScorePriceModel for user:me_me_me
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
2024-09-12 14:16:31.555 Bootstrapping virtual environment(s) ...
2024-09-12 14:16:31.730 Virtual environment(s) bootstrapped!
2024-09-12 14:16:32.185 Workflow starting (run-id 11):
2024-09-12 14:16:34.964 [11/start/22 (pid 24323)] Task is starting.
2024-09-12 13:16:36.632 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Task is starting (status SUBMITTED)...
2024-09-12 13:16:39.747 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Task is starting (status RUNNABLE)...
2024-09-12 13:16:40.849 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Task is starting (status STARTING)...
2024-09-12 13:16:44.217 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Task is starting (status RUNNING)...
2024-09-12 13:16:43.043 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Setting up task environment.
2024-09-12 13:16:47.268 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Downloading code package...
2024-09-12 13:16:47.704 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Code package downloaded.
2024-09-12 13:16:47.736 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Task is starting.
2024-09-12 13:16:48.091 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Bootstrapping virtual environment...
2024-09-12 13:16:49.323 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] Traceback (most recent call last):
2024-09-12 13:16:51.889 [11/start/22 (pid 24323)] AWS Batch error:
2024-09-12 13:16:52.026 [11/start/22 (pid 24323)] Essential container in task exited This could be a transient error. Use @retry to retry.
2024-09-12 13:16:52.026 [11/start/22 (pid 24323)] 
2024-09-12 13:16:49.323 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647]   File "<frozen runpy>", line 198, in _run_module_as_main
2024-09-12 13:16:49.323 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647]   File "<frozen runpy>", line 88, in _run_code
2024-09-12 13:16:49.323 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647]   File "/metaflow/metaflow/plugins/pypi/bootstrap.py", line 84, in <module>
2024-09-12 13:16:49.323 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647]     shutil.move(tmpfile, dest)
2024-09-12 13:16:49.323 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647]   File "/usr/local/lib/python3.11/shutil.py", line 853, in move
2024-09-12 13:16:49.324 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647]     os.rename(src, real_dst)
2024-09-12 13:16:49.324 [11/start/22 (pid 24323)] [be79f4fc-4c15-4295-955a-ddd296852647] TypeError: rename: src should be string, bytes or os.PathLike, not NoneType
2024-09-12 13:16:52.308 [11/start/22 (pid 24323)] Task failed.
2024-09-12 14:16:52.515 Workflow failed.
2024-09-12 14:16:52.515 Terminating 0 active tasks...
2024-09-12 14:16:52.515 Flushing logs...

This is my decorator:

@conda_base(
    libraries={
        "pandas": "2.2.2",
        "pyarrow": "10.0.1",
        "numpy": "1.26.4",
        "pandas": "2.2",
        "scikit-learn": "1.5.1",
        "lightgbm": "4.5.0",
        "s3fs": "2024.3.1",
        "xgboost": "2.1.1",
        "snowflake-connector-python": "3.11.0"
    },
python="3.11.9"
)

Any idea what could be wrong? Thanks!

Do you have any lifecycle policies attached to your s3 bucket?

I can ask the devops. which (metaflow?) s3 bucket would it be?

You will find a local .metaflow folder, inside of which there is another folder that bears the name of your flow. Can you delete that folder and try running again?

do you mean in my local home directory? I only have .metaflowconfig there.

In your flow directly

as in the repo folder?

Yeah that or any of the parent directories

thanks. just trying it.

Great!