Hello team, I am encountering a bizarre error in my self-hosted metaflow deployment. I am on AWS and using recommended terraform deployment setup with only minor changes. Starting today our data scientists started reporting instances of API calls to metadata service failing out of nowhere. For example, accessing API like

run = Flow('MyFlow')['sfn-d646da04-bf25-43de-9ef1-75c134da89d7']

Results in at least 2 REST API calls to metadata service, specifically:

GET /flows/MyFlow/runs
GET /flows/MyFlow/runs/sfn-d646da04-bf25-43de-9ef1-75c134da89d7

However strangely only the first GET request is resulting in a HTTP 500 “internal server error”. All other API calls are working fine, as well as our UI service is responding ok. I did my own investigation and to my surprise it turns out the service logs for the request actually report a HTTP 200 OK response but clients are receiving a 500 response on their end. The only explanation I can think of is that error must be originating from the API Gateway or any other middleware in the path. This is plausible, our

MyFlow

has more than many thousands of executions in our platform. So it is plausible that the total response size exceeds 10MB, but API Gateway has hard limit of 10MB on payload sizes. 1. Do you guys agree with my assessment? I am unable to prove it. I am able to confirm the hypotheses by directly connecting to the load balancer instead of api gateway, the response size is just over 10MB. 2. Any suggestions on how we can overcome it?

We have had issues with API gateway used by metadata service when deploying flows using argo workflows. We had to remove the API gateway.

Does metaflow support authn without API Gateway?

How did you solve this issue in the end?

We have not really solved it yet, but I manually archived some very old execution rows in the db so that the api response size decreased again. It will however recur if left as-is so I am still open to more perma solutions

> Does metaflow support authn without API Gateway? No. You will have to use something on your own side. Our VPC is only accessible using our company VPN. So we don't need extra level of security

Thanks. Yeah I regret we don’t have a VPN setup here, so I think I’ll have to stuff a nginx or something in between just to get the apikey basic auth 😞

Ideally API gateway should work! We actually tried to make it work first but ran out of ideas. It was quite random when things failed. For us it mostly failed when we would deploy our flow using

argo-workflows create

We use step functions here, so for us the api gateway did work out of the box until we hit this nonsensical payload restriction that comes from AWS’s end!

ah ok! Maybe it's the same issue which manifests in different ways. It used to work for us too until it started failing randomly

Yes that was the immediate suggestion I gave to our data scientists. But with such a large team someone eventually forgets the advice.