https://outerbounds.com/ logo
Join Slack
Powered by
Hi, we have enabled aws security hub recently, and...
# ask-metaflow
s
Hi, we have enabled aws security hub recently, and one of the rule is 
ECS containers should be limited to read-only access to root filesystems
, so I tried to add 
ReadonlyRootFilesystem: true
to metaflow cf template's 
TaskDefinition
, and this renders Metaflow unusable (deployment also has issues). Wondering if it is possible to enable the read-only to root while still able to deploy and use Metaflow as designed. Someone also posted this question in github issues, link here: https://github.com/Netflix/metaflow/issues/1257
👀 1
✅ 1
s
@narrow-lion-2703 @little-apartment-49355 ideas?
a
Its definitely fixable but unfortunately we don't have a ready-to-consume recipe at this point for ECS. We actually had to fix this for K8S deployments in Outerbounds platform and Metaflow Sandboxes, but the database migrations are orchestrated through init containers there, so that migration service code is not executed at all. But this approach can't be directly ported to AWS "native" deployments
s
I see, would the fix be complex and time consuming, and require lots of customization?
a
Its not really all that complex, in fact it just removes some parts of the metadata service that require root fs access. However you'd need to some way to run metadata database migrations, maybe directly connecting to the db and running a script, something like that. Did you have Metaflow service running before, or is it 1st time setup?
s
Have had Metaflow running before
a
Then you should have current migrations done. We can try the non root image we use for k8s, to see if it works on ECS, I can share it in a bit
If you want to give it a try, use this container image for Metadata Service 
public.ecr.aws/outerbounds/metaflow_metadata_service:v2.2.4-20-ga13e300-obp
instead of the one provided with the template and add Command to task definition of the Metadata Service :
Copy code
Command:
            [
              "/opt/latest/bin/python3",
              "-m",
              "services.metadata_service.server",
            ]
you can look at UI server task definition to see where that custom command goes
s
Cool, I'll give it a go. Could I just commented out the UI part if its not been used?
a
in the cloudformation template? its disabled if you don't set 
EnableUI
s
It's set to 
false
for 
EnableUI
. There is task definition for UI, 
TaskDefinitionUIService
that's still created, which don't seem to be used anywhere due to 
EnableUI
condition, I assume its ok to just comment the whole section?
a
ah yes
👍 1
s
Perfect, looks like everything worked fine, tested a few flows and all passed. Thank you!
Just another question, if it's the first time deploying Metaflow, what would be the change. Deploy with write permission first using the default image, then apply changes afterwards?
a
yes that could be a workaround. Alternatively you could apply migrations "manually" using goose if you can somehow connect to the db directly
👍 1
s
Cool, thank you for the help!
👍 1
Should I comment this to the github issue mentioned above, or you guys are planning on responding later one?
a
I'll comment there, I think the issue is still open in a sense that we'd eventually provide a more "officially supported" path to do this
👍 1
🙂 1
5 Views
Open in Slack
PreviousNext
Powered by