Hi all,
I'm experiencing some issues with the Met...
# ask-metaflowa
acoustic-van-30942
03/06/2023, 7:15 AMHi all,
I'm experiencing some issues with the Metaflow UI after transitioning to an internal hardened image. I'm unable to drill into the the runs. Seems to be some sort of error with the errorlogger. Any insight into what's causing this error?
✅ 1
acoustic-van-30942
03/06/2023, 7:16 AMIn addition to this, I ran a flow and got this error below once, but haven't been able to recreate this issue in the 10 subsequent runs:
v
victorious-lawyer-58417
03/06/2023, 7:22 AMit seems the service wasn’t available. Maybe it was starting up? 🤔
a
acoustic-van-30942
03/06/2023, 7:22 AMYeah it could be a cold start. Seems ok now.
v
victorious-lawyer-58417
03/06/2023, 7:23 AMit seems like the same 504 in the UI side. Does it still give the same error?
a
acoustic-van-30942
03/06/2023, 7:24 AMI'm afraid it does. Oh and as you wrote your message, I just recreated the error once again:
v
victorious-lawyer-58417
03/06/2023, 8:00 AMsince it's a timeout error, are you sure that your (new) instance / RDS are large enough to handle the load?
victorious-lawyer-58417
03/06/2023, 8:01 AMwe have seen spurious issues when using tiny instances / DBs
a
acoustic-van-30942
03/06/2023, 8:06 AMI could try using larger instances. We are using the same instances as before where we didn't encounter these issues. Tomorrow I'll switch back to the original containers for the metadata service and the UI to better isolate the issue. We are only using images that deviate from the template because our security team wants us to use our own internal hardened images
👍 1
acoustic-van-30942
03/06/2023, 4:54 PMOk - can confirm it's the hardened container image. The container for static UI seems to be ok, but the one I created for the metadata service seems to be causing some problems.
I've narrowed it down to 2 root causes and addressing them has seemingly fixed the issue:
• Not sure if it's caused by the number of replicas for the ECS Services, but I downgraded from 2 to 1
• Downgraded the golang container from 1.19 to 1.18
v
victorious-lawyer-58417
03/06/2023, 5:28 PMinteresting, good to hear
a
acoustic-van-30942
03/06/2023, 5:29 PMOk nevermind - I found the actual issue. It is the password rotation (I suspected it might be). After I rotated the password, things start to lose calibration.
v
victorious-lawyer-58417
03/06/2023, 5:32 PMnice, that should be easy to fix 🙂
a
acoustic-van-30942
03/06/2023, 5:34 PMWell maybe not - I've already integrated the password rotation via lambda to login into RDS and change the master password and I'm using secrets manager injection into the ECS containers. The services are healthy too.
acoustic-van-30942
03/06/2023, 5:37 PMI think it's the secrets manager injection. If I redeploy the ECS containers, it works again. Not sure how to explain it, but I think when I do the password rotation, even though it propagates the new password to both RDS and ECS, it does something with the calibration of the systems integrating together.
👍 1
3 Views