hey everyone, getting started with metaflow on GCP. wondering if anyone has experienced this issue - 🧵

can you share what your flow looks like?

@ancient-application-36103 metaflow 2.12.3 here's an

from metaflow import FlowSpec, step, conda, pypi, pypi_base, kubernetes, current, secrets
from metaflow.includefile import IncludeFile
from config import parameterize_flow

@pypi_base(
    python='3.10',
    packages={
        'pyyaml': '6.0.1',
        'setfit': '1.0.3',
        'my-private-package': '1.2.3',
        'pandas': '2.2.2',
        'scikit-learn': '1.3.1',
        'datasets': '2.14.4',
    }
)
class TrainingWorkflow(FlowSpec):
    default_params = parameterize_flow()
    config_dir = default_params.CONFIG_DIR
    config_file = IncludeFile("config_file", default=f"{config_dir}/workflow_configs.yaml")
    
    # uncomment for pypi / conda conflict
    # @conda(disabled=True)
    @step
    def start(self):
        import yaml

        self.conf = yaml.safe_load(self.config_file)
        # boilerplate
        self.bucket = self.default_params.GCS_BUCKET
        self.raw_data_dir = self.default_params.GCS_BUCKET_RAW_DIR
        self.flow_config = self.conf['workflows']['workflow_id']
        self.params = self.flow_config['params']
        self.output_dir = self.flow_config['output_dir']
        self.model_name = self.params['model_config']['model_name']
        self.next(self.get_data)

    @pypi(
        packages={
            "gcsfs": "2024.6.0"
        }
    )
    @step
    def get_data(self):
        import pandas as pd
        from sklearn.model_selection import train_test_split
        from datasets import Dataset, DatasetDict

        self.source_filepath = self.params['source_filepath']
        
        df = pd.read_csv(self.source_filepath)
        self.train_data, self.test_data = train_test_split(df)
        ...
        self.next(self.train_model)
    @step
    def train_model(self):
        ...self.next(self.end)

etc

thanks, we are shipping a release that should address this issue. eta for the new package to be available is roughly ~40 mins

amazing, thanks! 🙂 looking forward to seeing the fix. also, possibly related, what's the state of GCP artifact registry support? a little bit confused based on https://github.com/Netflix/metaflow/pull/1779 in the meantime I installed the bleeding-edge extension and got somewhere else I have GOOGLE_APPLICATION_CREDENTIALS exported and https://us-python.pkg.dev/repo-name/package-name/simple in extra_indices

: 401 Client Error: Unauthorized for url: <https://us-python.pkg.dev/repo-name/package-name/package_name-3.3.0-py3-none-any.whl>

it is supported in mainline