Huston, we got a problem!!! 🙂 Random our metaflow service stop working, today around 9 UTC we think its was something we deployed but we got this error

2023/12/12 13:03:30 goose run: failed to connect to `host=metaflow-metaflow-579p5yo1.[key].<http://eu-west-1.rds.amazonaws.com|eu-west-1.rds.amazonaws.com> user=metaflow database=metaflow`: server error (FATAL: no pg_hba.conf entry for host "10.10.2.137", user "metaflow", database "metaflow", no encryption (SQLSTATE 28000))

We have trying anything, even destory our stack wtih Terraform, upgrade to lates version but still have this issue, we are image version: netflixoss/metaflow_metadata_service:v2.4.5, before we use latest but get stuck on a 2year old image so now we custom change that image. So even after er full redeploy from ground we still se this problem, the RDS database are Postgres 16.1 before 15.4 with same issue So anyhow have this problem before and whats can be changed our terraform setup look like this and all versions are updated to the latest terraform modules.

# Random suffix for this deployment
resource "random_string" "suffix" {
  length  = 8
  special = false
  upper   = false
}

locals {
  resource_prefix = "metaflow"
  resource_suffix = random_string.suffix.result
}

module "metaflow" {
  source  = "outerbounds/metaflow/aws"
  version = "0.10.1"

  resource_prefix = local.resource_prefix
  resource_suffix = local.resource_suffix

  ui_certificate_arn        = "arn:aws:acm:eu-west-1:<acc-id>:certificate/<uuid>"
  ui_static_container_image = "<acc-id>.<http://dkr.ecr.eu-west-1.amazonaws.com/metaflow-ui-frontend:latest|dkr.ecr.eu-west-1.amazonaws.com/metaflow-ui-frontend:latest>"
  ui_allow_list             = ["range-list"]
  db_instance_type          = "db.m6g.large"
  db_engine_version          = "16.1"
  
  metadata_service_container_image = "netflixoss/metaflow_metadata_service:v2.4.5"

  enable_step_functions = false
  subnet1_id            = data.aws_subnets.public_subnets.ids[0]
  subnet2_id            = data.aws_subnets.public_subnets.ids[1]
  vpc_cidr_blocks       = ["10.10.0.0/16"]
  vpc_id                = data.aws_vpc.vpc.id
  with_public_ip        = true
  force_destroy_s3_bucket = true
  
  tags = {
    "managedBy" = "terraform"
  }
}

# export all outputs from metaflow modules
output "metaflow" {
  value = module.metaflow
}

# The module will generate a Metaflow config in JSON format, write it to a file
resource "local_file" "metaflow_config" {
  content  = module.metaflow.metaflow_profile_json
  filename = "./metaflow_profile.json"
}

since it worked before.. can you try rolling back service image to 2.4.3 and see if it helps

Same error, here are some error the service sending back, remeber its just stop working on image 2.3.0, donno about something inside the app automatic update something or?

Gotcha, I'd ignore UI server for now. If we can make metaflow-service work we can figure out the rest. What supposed to happen is: 1. terraform creates an RDS instance with username/password 2. there is ECS task for metadata service which is supposed to connect to the db using username/password 3. when it first runs with a fresh db, it runs migrations to create tables and whatnot If you still have your RDS db that was used successfully before, I'd try to connect to it using username/password from ECS task definition, using psql. It should already have all the tables. It may not be publicly exposed so for debugging, you might need to connect from inside the VPC and/or make it public, depends on your networking setup. If you can connect and metadata service can't, maybe its using a wrong/old password or something, so that narrows it down. (The original error looks like service can't connect to the db. The last MFGUI backend screenshot looks like backend service can connect but there are no tables. Might be because its a fresh db but metadata service didn't run the initial migrations for some reason. )

After a fresh install its look like as you say its not run the migration, can i manual trigger this migration? I will tomorrow (danish time) redeploy it again with Terraform, i expect the same error will appen on it, so need some way to manually trigger this migration. I will ofc test the database connection to just to besure i can connect to the database for sure 🙂 i will ignore everything about the UI service for now, and conectate for the service first.

in 2.4.4 there was a new option to enable metaflow service to use SSL when connecting MF_METADATA_DB_SSL_MODE, might be that setting this to "prefer" would help https://github.com/Netflix/metaflow-service/releases/tag/v2.4.4

Cool, i have a fully working setup again with lots of fighting ^^ with Terraform and currenct version of metaservice+frontend, thanks for helping 🙂

great! what did you have to do to make it work? did MF_METADATA_DB_SSL_MODE help?

I just turn off the SSL in AWS RDS, but now we have new problems, so we will start again from version 2.0.6 and checkup, we hit some new bugs inside the metaservice container so its have been a "hard fight" after its crash