Hi Metaflow, thanks for your help ^! The next issue we’re running into is related to using

@conda_base

. Is

@conda_base

supported in Airflow? If not do I have to convert step to use

@conda

?

Yes, indeed - all decorators are supported in Airflow.

Interesting… I’m getting this issue from Airflow after copying dag.py to MWAA. The flow I wrote uses

@conda_base

.

[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO -   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 16, in bootstrap_environment
[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO -     packages = download_conda_packages(flow_name, env_id, datastore_type)
[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO -   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 75, in download_conda_packages
[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO -     shutil.move(
[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO -   File "/usr/local/lib/python3.9/shutil.py", line 825, in move
[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO -     os.rename(src, real_dst)
[2022-12-20, 16:56:01 UTC] {{pod_manager.py:197}} INFO - TypeError: rename: src should be string, bytes or os.PathLike, not NoneType

are you able to run a flow using

python flow.py run --with kubernetes

outside of Airflow?

@square-wire-39606 No, same error:

2022-12-20 12:43:19.149 [1671561795448588/start/1 (pid 93851)] Task is starting.
2022-12-20 12:43:20.755 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Task is starting (Pod is pending)...
2022-12-20 12:43:51.165 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Task is starting (Pod is pending)...
2022-12-20 12:44:11.399 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Task is starting (Pod is pending, Container is waiting - ContainerCreating)...
2022-12-20 12:44:30.897 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Setting up task environment.
2022-12-20 12:44:39.672 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Downloading code package...
2022-12-20 12:44:40.608 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Code package downloaded.
2022-12-20 12:44:40.652 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Task is starting.
2022-12-20 12:44:42.235 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Bootstrapping environment...
2022-12-20 12:44:56.880 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] Traceback (most recent call last):
2022-12-20 12:45:05.586 [1671561795448588/start/1 (pid 93851)] Data store error:
2022-12-20 12:45:05.732 [1671561795448588/start/1 (pid 93851)] No completed attempts of the task was found for task 'CapacityManagementFlow/1671561795448588/start/1'
2022-12-20 12:45:05.732 [1671561795448588/start/1 (pid 93851)] 
2022-12-20 12:44:56.880 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]   File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2022-12-20 12:44:56.880 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]     return _run_code(code, main_globals, None,
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]   File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]     exec(code, run_globals)
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 104, in <module>
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]     bootstrap_environment(sys.argv[1], sys.argv[2], sys.argv[3])
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 16, in bootstrap_environment
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]     packages = download_conda_packages(flow_name, env_id, datastore_type)
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]   File "/metaflow/metaflow/plugins/conda/batch_bootstrap.py", line 75, in download_conda_packages
2022-12-20 12:44:56.881 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]     shutil.move(
2022-12-20 12:44:56.882 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]   File "/usr/local/lib/python3.9/shutil.py", line 825, in move
2022-12-20 12:44:56.882 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl]     os.rename(src, real_dst)
2022-12-20 12:44:56.882 [1671561795448588/start/1 (pid 93851)] [pod t-8bcfr-mdhhl] TypeError: rename: src should be string, bytes or os.PathLike, not NoneType
2022-12-20 12:45:05.939 [1671561795448588/start/1 (pid 93851)] Task failed.
2022-12-20 12:45:06.124 Workflow failed.
2022-12-20 12:45:06.124 Terminating 0 active tasks...
2022-12-20 12:45:06.124 Flushing logs...
    Step failure:
    Step start (task-id 1) failed.

are you able to run a flow on kubernetes without using the

@conda

/

@conda_base

decorator? I would like to ensure that nothing is borked from a configuration point of view.

Sure. Do I just try running without conda and see if it at least starts and then fails?

Yep

@square-wire-39606 i removed

@conda_base

and it was able to download the vanilla conda packages in our s3 bucket. The job failed when it tried to import a package that didn’t exist in the next step though.

can we do a quick zoom call in 30 mins?

Can we do 2pm CST/12pm PST? Our plat infra guy @wide-battery-59962 will be in a meeting until then

sure! mind sending an invite at savin@outerbounds.co?

oh @square-wire-39606 i got

--with kubernetes

to run with a fresh reinstall of my conda environment! i also got

flow.py airflow create dag.py

to upload the conda environment to s3 by deleting my local

.metaflow

directory, similar to what @red-fountain-16335 had done here. Running the airflow dag now and will update when done. did we ever look into why we’d have something cached incorrectly in

.metaflow

? just wondering if there is some reproducibility with this issue

@aloof-application-80784 one issue that can cause this is if you happen to make changes to metaflow config that will invalidate the cache. It's on our to-do list to add checks for this scenario.

Thanks @square-wire-39606, that makes sense. We were making a lot of config changes to get the Airflow + Kubernetes implementation set up. Really appreciated your help here and metaflow as a product!

I have added a ticket to keep track of this work - https://github.com/Netflix/metaflow/issues/1219