Hello, I'm running into this issue: `ERROR: relati...
# ask-metaflowu
user
02/07/2023, 8:22 PMHello, I'm running into this issue:
ERROR: relation "public.flows_v3" does not exist✅ 1
h
hallowed-glass-14538
02/07/2023, 11:45 PMhey Ishana! a few questions to understand the scope of the issue :
1. How have you deployed your service?
2. If it is via a container, can you check the image tag of the container
My main hunch is that the metaflow service was not able to finish it's migration of the database to the latest schema (when it starts up). This part of the documetation gives a little more color to this migration aspect.
m
many-computer-12737
02/28/2023, 3:44 AMI’ve been working with @User and can elaborate on the issue. We developed a set of Terraform modules internally to deploy the Metaflow metadata and UI servers as separate services each connected to different AWS RDS instances, such that the UI server RDS instance is a logical replica of the metadata server RDS instance. Our concern was that DB triggers installed by the UI service might induce unacceptable additional load on a single shared DB.
many-computer-12737
02/28/2023, 4:03 AMWe’ve since figured out that we need to create the replicated tables manually in the UI service RDS instance, and then set up logical replication as described by AWS.
many-computer-12737
02/28/2023, 4:04 AMA higher-level question, though: Have you observed metadata DB loads in a production environment when using the UI that are high enough to warrant setting up a logical replica in this way?
d
dry-beach-38304
02/28/2023, 10:28 AMI can give you a bit of context on the logical replica bit. The UI is extremely read heavy and we wanted to ensure a very strong guarantee on the main write path (when a flow runs). That is why we set up logical replication to ensure that whatever the UI does, it wouldn’t get in the way. We have seen cases where the read path does suck up a lot of resources. granted, it was frequently because there was something unindexed but it did serve its purpose of isolating the write path.
dry-beach-38304
02/28/2023, 10:28 AMrecently though, we decided to switch to using aurora which allows for an easier scaling of the reads and is more “integrated”
dry-beach-38304
02/28/2023, 10:29 AMyou can see that work here: https://github.com/Netflix/metaflow-service/pull/344
dry-beach-38304
02/28/2023, 10:30 AMnote that this is what we are running internally but we have a slightly different setup so it has not been validated “in the wild”. @User can provide more context if you are interested in going that route. This has greatly reduced our maintenance burden (we don’t ahve to worry about the logical replica falling behind which did happen in some cases) and will enable us to scale more.
m
many-computer-12737
03/01/2023, 9:17 PMFYI: @User @rich-elephant-44987
👀 2
u
user
03/01/2023, 9:40 PMyes, happy to provide more details here. The PR that @dry-beach-38304 linked allows you to horizontally scale read workloads from the UI without using logical replication. To help us achieve this, we use Aurora Postgres internally. It simplifies our design and operational overhead. Note that there are caveats with this setup so please review the README file in the PR. This still does not solve the problem of data scaling (each node of the DB contains the entire data) but this isn’t a big problem since we have a lot of headroom on the data size.
12 Views