Hey everyone! We are trying to orchestrate our Metaflow pipelines with AirFlow/Cloud Composer (running on GCP), but when we upload the generated AirFlow DAG into our DAGs folder, we get this parsing error: Does this look familiar to anyone? We are using this command to generate the pipeline:

python flow.py --environment=pypi --with retry airflow create flow_dag.py

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/opt/python3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/airflow/config/airflow_local_settings.py", line 62, in pod_mutation_hook
    and any(env_var.name == "AIRFLOW_IS_K8S_EXECUTOR_POD" for env_var in container.env)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/airflow/config/airflow_local_settings.py", line 62, in <genexpr>
    and any(env_var.name == "AIRFLOW_IS_K8S_EXECUTOR_POD" for env_var in container.env)
            ^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'name'

can you provide some additional info on your setup in order to look into this • which version of Metaflow are you experiencing the error with? • is there a known working version as well?

These are the versions we have tried thus far:

Metaflow 2.15.15 and airflow 2.9.3

thanks! we are investigating. also, worth pointing out that not all features of metaflow are available on airflow owing to limitations inherent in airflow's architecture.

I unfortunately don't have access to a Cloud Composer deployment for exact testing, but I tried out a simple airflow generated dag with the latest metaflow version on a local airflow deployment, and it seems to work as expected. Any luck on your end with this? I'm suspecting there might be some differences with the executors of apache-airflow and cloud composer that are throwing a wrench in here. have you been able to run metaflow generated airflow-dags in the past with your cloud composer setup, or is this a fresh setup?

no worries, could you please point me to the metaflow code that does the "transpiling" to AirFlow syntax? maybe I can poke at it a bit

https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/airflow/airflow.py#L648 this is where all the conversions happen 🙂

Thank you!

anything from

metaflow.metaflow_config

is set through: • a prefixed environment variable, e.g.

METAFLOW_AIRFLOW_KUBERNETES_CONN_ID

• a key-value in the metaflow profile json file usually in ~/.metaflowconfig