https://outerbounds.com/ logo
Join Slack
Powered by
Hello! I've been using metaflow for the past year ...
# ask-metaflow
c
Hello! I've been using metaflow for the past year and it's been working great 🙂 I'm using the terraform-aws-metaflow module to set up my environment. Earlier this week I changed the database instance type from 
db.t2.small
--> 
db.t3.small
as the support is ending for the t2 instances. With this change I also pulled in some changes to the 
aws_lambda_function.db_migrate_lambda
resource. It went from python runtime "python3.7" -> "python3.12" and what I believe is the latest source code hash. After this change, I'm not able to see any logs in the metaflow UI anymore under stdout. Looking in the console I can see there is a 504 error 
Failed to load resource: the server responded with a status of 504 ()
Is anyone aware of this being a known issue in the latest version?
a
did you also execute the aws lambda?
@bulky-afternoon-92433 / @hallowed-glass-14538 - any thoughts here?
c
tbh, I'm not sure it's related to the lambda at all. It's just that those are the only changes I can see in my terraform plan that I applied just before the logs issue started appearing
a
Is the backend server healthy?
c
yes, all the services look healthy and there are no issues running things in the instance. It's just displaying the logs in the metaflow UI
a
Can you verify that the service instance is able to read logs from the s3 bucket? Likely there might be log lines in the service indicating an issue
c
In the logs of the metadata-service I can't see any errors. Also, I am able to see the artifacts in the UI and I guess that's taken from the s3 bucket? Are the stderr logs also pulled from the bucket?
b
they are also pulled from S3 yes, so if artifacts are showing up then S3 access from the server to the bucket is not an issue. Did you upgrade the image version for the ui_backend during this as well? there were some log fetching related optimizations for the ui_backend in 
2.4.11
 strike that, from the screenshot it seems to be running 2.3.0
the db_migrate lambda simply tries to perform a request against the migration service endpoints (which is part of the metadata-service container) so in order to figure out what is going on with the setup, some things to check: • did the migration lambda execute successfully? • what version of the metadata-service (not ui) are you running? this will directly affect what version your db schema is at • if there's a version mismatch of images between metadata-service/ui-service, I would first try to upgrade/downgrade the ui-service image accordingly, as the best guarantee for functionality is to somewhat match versions.
c
Actually, I can't see any invocations of the lambda. This seems to be the change I'm catching in my latest terraform plan https://github.com/outerbounds/terraform-aws-metaflow/pull/88 Is it really necessary to run the 
db_migrate_lambda
after that? This is the image being pulled for the metadata_service and ui_backend: 
metaflow_metadata_service:v2.3.0
FYI, this resolved itself without any action. Still not sure what the issue was but it's working now 🤷
8 Views
Open in Slack
PreviousNext
Powered by