Hi team, hope all is well. When configuring Metafl...
# ask-metaflow
c
Hi team, hope all is well. When configuring Metaflow using the output json from the Terraform stack like so: "metaflow configure import <point to the output file from Terraform>", why does another config file get created? and which config file is being used? Please see screenshot below, seems like the metaflow CLI duplicates the config
āœ… 1
Hi team, also when I trigger the step function and switch the name space to investigate the run in order to obtain the data artifacts, the client is throwing access denied error, not sure how this is because the config output from the terraform module holds the policy setup
v
it uses
config.json
by default but you can use
config_poc.json
by specifying
METAFLOW_PROFILE=poc
can you try e.g.
aws s3 ls <s3://metaflow-s3-poc/>
on this same instance to check that AWS permissions are working correctly?
c
Hi there, list permission denied error. I have the env var METAFLOW_PROFILE=poc. Is there something specific I need to setup on profile ~/.aws/credentials? On this I've setup aws credentials with a profile poc too, but is there a specific role, since I'm just using an admin role. It is here where I'm having the confusion between what metaflow config is required and what aws config is required and how metaflow authenticates to the provisioned AWS stack
Also this does not seem right, if I have the configuration set to run locally, why does it need s3 access?
what I'm figuring out via a trail and error process...is that just because it is set to run locally is not enough I also have to go and manually update the METAFLOW_DEFAULT_DATASTORE value from "s3" as provided by the Terraform stack output to "local"
Also the confusing bit is what aws profile should be used? Because in order to run: python flow.py --with retry step-functions create....it requires an AWS_PROFILE, but then what is the METAFLOW_PROFILE.....should the output roles from the stack be added into the aws credentials file or how does this work?
Here I show how I drill all the way down to the data artifact of my triggered state machine, but as soon as I call the "data" attribute on the data artifact object I get "GETOBJECT" access denied error
v
thanks for the context. Here's a simple way to think about it: • Metaflow doesn't do any authorization and authentication by itself. Hence
METAFLOW_PROFILE
only configures things pertaining to Metaflow, like where its data should be stored, where to find the metadata service etc • To make Metaflow work, you need to have IAM roles and credentials set up so that you can access the buckets and other resources specified in the Metaflow config (through
METAFLOW_PROFILE
). This is where
AWS_PROFILE
comes in.
if you are using an admin IAM role locally which can access all resources, everything should work
however since
aws s3 ls <s3://metaflow-s3-poc/>
fails too as you showed above, it seems your credentials don't allow you to access the
metaflow-s3-poc
bucket
to fix the issue, you'd need to log in to AWS console, find the role that corresponds to your
~/.aws/credentials
and attach a policy to the role that grants it access to
metaflow-s3-poc
. Or you can do
S3FullAccess
default policy, if that's ok in your environment
c
thank you
šŸ‘ 1