Hi all I m getting the following prompt even though I am pre Outerbounds #ask-metaflow

Hi all, I'm getting the following prompt even tho...

acoustic-van-30942

03/30/2023, 7:48 PM

Hi all, I'm getting the following prompt even though I am pretty sure I am up-to-date with my Metaflow versions. Any insight on how to resolve?

Copy code

You are running a version of the metaflow service that currently doesn't support AWS Step Functions. 
For more information on how to upgrade your service to a compatible version (>= 2.0.2), visit:
    <https://admin-docs.metaflow.org/metaflow-on-aws/operations-guide/metaflow-service-migration-guide>
Once you have upgraded your metadata service, please re-execute your command.
    Incorrect version for metaflow service:
    Try again with a more recent version of metaflow service (>=2.0.2).

acoustic-van-30942

03/30/2023, 7:51 PM

• metaflow client version: 2.8.2 • metadata service api version: 2.3.7

acoustic-van-30942

03/30/2023, 8:37 PM

Okay hmm. I have a lead - if I use the DNS domain URL as

METAFLOW_SERVICE_URL

in the metaflow config file, it rejects the step function creation, but no such problems when using the API Gateway URL directly in the config.

ancient-application-36103

03/30/2023, 8:49 PM

@acoustic-van-30942 how are you determining the metadata service version? It is likely that the DB has a old schema that is pinning your metadata service version to an older version than 2.3.7

ancient-application-36103

03/30/2023, 8:50 PM

here is an article that can help you move to a more recent version of the service - https://outerbounds.com/engineering/operations/migration/

acoustic-van-30942

03/30/2023, 8:52 PM

Hi @square-wire-39606 - I think it might have something to do with my DNS domain URL that we have on top of API Gateway. It's been causing me so many issues, as it seems to be altering the behaviour of the metadata service endpoint in a few ways: • Making running flows appear to be failures in the UI even though they aren't actually failures • Not allowing me to submit step functions BUT, as soon as I revert to the underlying API Gateway URL or the NLB in the metaflow config, these above issues go away.

acoustic-van-30942

03/30/2023, 8:53 PM

Metaflow DB is up to date. I pinged the migration lambda and doesn't require any upgrade.

acoustic-van-30942

03/31/2023, 12:28 AM

Okay so we did a bunch of troubleshooting, including stepping into the code... There's a function in the file

step_functions_cli.py

called

check_metadata_service_version

. When I print out the

metadata_version

, API Gateway returns

2.3.7

, while the DNS we mapped to returns

None

. Looking through the code, there is some sort of

context

created that provides this metadata version. If you could offer me insight into the composition of this

context

and how it is constructed, that would go a long way into determining the delta between the domain URL and the API Gateway URL

square-wire-39606

03/31/2023, 2:06 PM

here is how we get the version - https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/metadata/service.py#L486

thankyou 1

14 Views

Open in Slack

Previous Next