Hello! It' appears that something in the

google-api-core

release yesterday is not compatible with Metaflow since we have started seeing massive failure of all our pipelines this morning with this message:

[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/bin/download-gcp-object", line 5, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from simple_gcp_object_downloader.download_gcp_object import main
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/lib/python3.12/site-packages/simple_gcp_object_downloader/download_gcp_object.py", line 2, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from google.cloud import storage
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/lib/python3.12/site-packages/google/cloud/storage/__init__.py", line 35, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from google.cloud.storage.batch import Batch
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/lib/python3.12/site-packages/google/cloud/storage/batch.py", line 43, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from google.cloud import exceptions
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/lib/python3.12/site-packages/google/cloud/exceptions/__init__.py", line 24, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from google.api_core import exceptions
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/lib/python3.12/site-packages/google/api_core/__init__.py", line 20, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from google.api_core import _python_package_support
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]   File "/usr/local/lib/python3.12/site-packages/google/api_core/_python_package_support.py", line 28, in <module>
[2025-10-28, 07:01:50 UTC] {pod_manager.py:471} INFO - [base]     from packaging.version import parse as parse_version
[2025-10-28, 07:02:00 UTC] {pod_manager.py:471} INFO - [base] ModuleNotFoundError: No module named 'packaging'

The failure is coming from this line. I've looked through the metaflow codebase and it looks like none of the GCP dependencies here are pinned to specific versions so they are probably pulling in the latest

google-api-core

version

We’re encountering exactly the same issue today, all of our pipelines are currently failing. @quick-carpet-67110, please let me know if you find an initial workaround or solution to this problem.

@thankful-ambulance-42457 ^

I have tested an extremely hacky workaround and it seems to work: 1. We are orchestrating the metaflow pipelines with airflow 2. So, in the dags folder I take the transformed airflow pipelines and grep for this line:

' && if [ -z $METAFLOW_SKIP_INSTALL_DEPENDENCIES ]; then python -m pip install -qqq --no-compile --no-cache-dir --disable-pip-version-check google-cloud-storage google-auth simple-gcp-object-downloader google-cloud-secret-manager requests;

3. Then I simply pin the

google api core

version to 2.27.0 like this by adding it to the list of deps

&& if [ -z $METAFLOW_SKIP_INSTALL_DEPENDENCIES ]; then python -m pip install -qqq --no-compile --no-cache-dir --disable-pip-version-check google-cloud-storage google-auth simple-gcp-object-downloader google-cloud-secret-manager google-api-core==2.27.0 requests; fi

It looks like it's working now

We are working on shipping a workaround for this unfortunate regression with such a core library https://github.com/googleapis/python-api-core/issues/848

looks like the CI/CD runner, finished, going to give this a try

Hi @quick-carpet-67110 you can upgrade to

2.19.3

with the fix.