Hi team, starting today, I have started getting the following error when I try to run a flow or trying to retrieve the artifacts from a run in my sandbox env. Does anyone know what could potentially be the issue? Thanks.

Metaflow service error:
    Metadata request (/flows/MY_FLOW) failed (code 500): "{\"err_msg\": {\"pgerror\": null, \"pgcode\": null, \"diag\": {\"message_primary\": null, \"severity\": null}}}"

looking..

Thanks Ville, also, if needed, the name of the flow is

eda_sapm_training_dag

.

MY_FLOW

is just a placeholder I used in the message.

@clean-battery-63376 can you DM me the email address you used to sign up for the sandbox, so I can find yours

I asked Hamid this, and turns out it is about an internal Metaflow deployment, instead of the OB-provided sandbox. @average-beach-28850 any general pointers to help Hamid understand their deployment situation and potential causes for the MF service error?

gotcha, then I'd start with

curl -v http://<METADATA_SERVICE_URL>/flows

to see the error (or

curl -H 'x-api-key: <METAFLOW_SERVICE_AUTH_KEY-from-your-metaflow config>' -v http://<METADATA_SERVICE_URL>/flows

and see what error it returns

@average-beach-28850: Thanks for your message. Yes, it used to worked fine until a few days ago. I ran the

curl -v http://<METADATA_SERVICE_URL>/flows

in my Jupyter notebook and the below is what I am seeing. I am assuming the yellow part is the issue, right?

I'd look at service logs to see if there are any clues there

I tried looking at the ECS log events, and it seems the issue is the following. Probably an issue with D connection

ERROR:AsyncPostgresDB:global:Exception occurred

sounds like it. It may be worth restarting the ECS service and then looking at the first few pages of the logs right after it starts, it is usually easier to see what the actual error is

Yes, I can see the following. I have bumped the version of the Python within the flow to see if it is related to it or not. It is bootstrapping the environment for a while, so waiting on that:

hmm so its a DNS error "name or service not known" for the RDS hostname.. means it can't even resolve that host. Can you check if you have RDS database deployed, and if the hostname matches what you see in the log here?

Sounds like the RDS instance is gone for some reason. Trying to see if I can recreate it.