hi friends, we've started seeing this error message out of the blue. any ideas what could've caused it?

Pylint is happy!

Metaflow service error:

Metadata request (/flows/ValidateModelPipeline) failed (code 500): {"message": "Internal server error"}

can't instantiate any

Run()

objects either; get the same 500 error

is the metadata service up and running?

you mean the RDS instance? if so, yes, it's running and there is no errors in the logs @square-wire-39606

could this ECS message be relevant?

task status

Status reason 	CannotPullContainerError: Error response from daemon: Get "<https://registry-1.docker.io/v2/>": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

everything looks ok on https://status.docker.com/ also was able to

docker pull netflixoss/metaflow_metadata_service

ok

the running count for the service is set to 0

cloudformation, 12/02/2020 template

can you try to bounce the task?

should I update the stack with the current template? I already backed up the RDS instance

there haven't been any major updates to the template but yeah why not

the task is gone

to clarify, are you saying to update the stack with the original template, or to replace the current template with the latest cloudformation file?

you can replace the template with a newer template

I'm running into an error about Batch ComputeEnvironment not being found. it's weird, I took screenshots of the Resources tab and it's the same exact ComputeEnvironment as before. however, when I check AWS Batch > Compute environments, indeed, that ComputeEnvironment doesn't exist. I think it's because we also use custom compute environments for different flows edit: I manually created a ComputeEnvironment with the name the template expected, problem solved. solving the next errors...

@ancient-application-36103 the cloudformation stack has been in rollingback for about 40 mins now. specifically, the ECSFargateService has been stuck in UPDATE_IN_PROGRESS

What do you see when you go to the ECS console?

I think the issue is that when I created the new ComputeEnvironment, that the next time I ran cloudformation update it started to create a new ECS cluster. however, we already have an ECS cluster running

metadata-service

. so, I think the solution is to roll back and set the Batch compute environment to the already existing one. seems stuck, though

if I manually delete the new ComputeEnvironment ECS cluster, will it mess up the rollback?

It's hard to say

blob sweat what do you recommend?

it looks like the existing ECS cluster (metaflow-photogrammetry-...) has had it's services updated:

metadata-service

->

metadata-service-v2

, and a new service

metaflow-ui-service

do you want to do a quick screen share?

sure!

issue resolved @ancient-application-36103. the problem was that someone had been fiddling with our IAM permissions while enabling multi-region support. ended up cutting off the VPC from talking to dockerhub

no need to protect the guilty @User, it was me!

tysvm for the support call earlier

to be specific, i was working on allowing our flows to be dispatched to different regions with different resource (e.g. s3) configurations, which involved messing around with VPC peering and routing tables

ok I'll do my worst next time @calm-smartphone-49719 🧌

in the process I accidentally removed the internet egress route from the same vpc the metadata service was in