https://outerbounds.com/ logo
Join Slack
Powered by
Hey everyone, I’m in the process of updating our ...
# ask-metaflow
c
Hey everyone, I’m in the process of updating our Metaflow AWS stack with a new version of the Terraform template (previously used this one, now upgrading to this). I am worried about losing all the results and metadata of Flows executions that were run in the previous setup. I assume that I need to make sure that in 
terraform plan
the 
METAFLOW_DATASTORE_SYSROOT_S3
and 
METAFLOW_DATATOOLS_S3ROOT
have to point to the same buckets as before. But I am unsure about the Metadata service? How can I make sure this data is preserved after the update? Thanks a lot for any help
✅ 1
Looking at the output of 
terraform plan
regarding the metadata RDS I see the following two lines:
Copy code
# module.metaflow.module.metaflow-datastore.aws_db_instance.this will be destroyed
and
Copy code
# module.metaflow-datastore.aws_db_instance.this[0] will be created
  + resource "aws_db_instance"
So I assume I will lose my metadata if I run the current template. It looks like it’s mostly because the module/directory structure has been changed between scripts. I am not sure I will be able to manually align that (or whether it’s a good idea).
Do you think it would be an option to take a snapshot from the AWS RDS instance, run the 
terraform apply
and then restore the old RDS from the snapshot? This plus preserving the S3 buckets should allow me to keep the current data and metadata, right?
f
Hey Jakob 👋 Is your current RDS instance using an Aurora cluster? From the linked terraform, it looks like there's a new option to use an Aurora cluster, and if you're currently using a Postgres DB it'd have to be migrated for that flavor first, or you could specify to continue using Postgres
c
Hey Russell, thanks for the hint. It’s both Postgres DBs for the old and new stack.
f
hmm, my guess would be this is caused from the count arg in terraform
Copy code
resource "aws_db_instance" "this" {
  count                     = local.use_aurora ? 0 : 1
it looks like that didn't always exist in the tf template, so that could be what's breaking it for you
e.g. it's a count of 1, rather than a single implicit resource
I'm not too familiar with terraform though, so others might know of a workaround!
c
Thanks again for the pointers Russell. I will look into it
h
I havent tried this migration but my first guess looking at this might be to try a 
terraform import
to put it in your tf state and tell in that you want it to use the existing db
2 Views
Open in Slack
PreviousNext
Powered by