https://outerbounds.com/ logo
Join Slack
Powered by
May I ask a question please? I have Metaflow pipe...
# ask-metaflow
g
May I ask a question please? I have Metaflow pipeline which is deployed to Step Function which utilizes the 
foreach
. Since the 
foreach
will take place in the Step Function, I believe DynamoDB is required to manage the state of the 
foreach
. Suppose if the pipeline and the Step Function is running on AWS region 
us-west-2
, is it possible to have the DynamoDB running on other region, i.e. 
us-east-1
? 
METAFLOW_SFN_DYNAMO_DB_TABLE
seems to accept only Table Name instead of the full ARN. Is it possible for Metaflow to support Step Function and DynamoDB residing in two separate AWS regions? Thank you for your help!
✅ 1
f
hey @gentle-analyst-70055! yes you can, dynamo access isn't restricted by region unless there's an explicit 
Deny
IAM policy like the one below. IIRC you do not need to go through all of the hoops that you do in the cross-account case https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_aws_deny-requested-region.html
there is an edge case though if you have a vpc gateway endpoint set up for dynamodb since those are region-specific.. I think.
the only thing you'd have to be sure to do with the IAM policies would be to include the region in the ARN for the policy
ahh I misread that one - I wonder if setting up the IAM permissions with the full ARN inc region would be enough there - have you tried the cross-region deployment and its not working or have you not tried it yet
looking at the ddb code, it isn't using the sfn direct integration: https://github.com/Netflix/metaflow/blob/master/metaflow/plugins/aws/step_functions/dynamo_db_client.py#L17 which makes me think the cross-region access wouldn't be any trouble
g
Thank you for replying, I really appreciate it. We have a cross region deploy where the step function is in us-west-2 and the dynamo in us-east-1. It failed since the step is looking for dynamodb in us-west-2.
f
are you using the terraform modules from outerbounds?
g
Looking at the dynamo client code there I can't seem to find a way to set the dynamo region, only table name. I suppose this is why I reached out to ask the question.
No, we're building the dynamo using an internal tool so not using outerbounds tf module.
f
ah - so one heavyweight solution I think would be to use dynamo global tables, which are multi-region and should just work
would be a huge infra code smell though, attacking a fly with a sledgehammer kind of thing
but yeah I think all you need to do for this to work would be to not have an explicit DENY policy in place and have the IAM policy include the region, so no wildcard
if you are using the outerbounds tf modules, I think this line is problematic: https://github.com/outerbounds/terraform-aws-metaflow/blob/master/modules/step-functions/iam-step-functions.tf#L135
I've been living in cdk-land for the last 6 years so my terraform knowledge isn't all there but this doesn't look right to me. I think all you need to do would be to look in your deployment for the equivalent IAM policy and make sure to include the full ARN including region for dynamo
g
Thanks Bryan, I'll try setting 
METAFLOW_SFN_DYNAMO_DB_TABLE
to the full ARN of the dynamo and ping back here. We do have IAM policy which grants permission for the cross region access, so access is allowed in terms of IAM permission.
f
@gentle-analyst-70055 ahh so 
METAFLOW_SFN_DYNAMO_DB_TABLE
won't accept an ARN as input so that won't work. Looking at the code a bit more it is kind of inconsistent where only the ddb table and auth key don't accept arns. Looks like it wouldn't require that big of a change to support ARNs for ddb so I can create an issue to start that conversation and should have time tomorrow to work on the PR
that was a fun little trip into the weeds though
g
Hi Bryan, thank you for taking a look into the code - I really appreciate it! I have just completed setting 
METAFLOW_SFN_DYNAMO_DB_TABLE
to the full ARN, and unfortunately it failed with the following error (I scrubbed the account id in the message below)
Copy code
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the PutItem operation: 1 validation error detected: Value 'arn:aws:dynamodb:us-east-1:<aws account id>:table/ml-training-pipelines-metaflow-internal-tools' at 'tableName' failed to satisfy constraint: Valid ARN format is 'arn:<awsPartition>:<vendor>:<region>:<subscriber>:resourceType/resourceName', where Resource name must have length less than or equal to 255, Resource name must have length greater than or equal to 3, Resource name must satisfy regular expression pattern: [a-zA-Z0-9_.-]+
Looking at the code, I believe this is where the dynamo client is obtained: https://github.com/Netflix/metaflow/blob/2588f1d3030ebbd5ca3ab47a2fffb1eb9926855a/metaflow/plugins/aws/step_functions/dynamo_db_client.py#L13 If we can specify AWS region for the dynamo when obtaining client, I believe this would resolve the issue.
At the risk of jumping to solution mode too early - perhaps we can specify an optional environment variable called 
METAFLOW_SFN_DYNAMO_DB_AWS_REGION
. If 
METAFLOW_SFN_DYNAMO_DB_AWS_REGION
environment variable is specified, we can supply additional 
client_params={'region_name': <value from METAFLOW_SFN_DYNAMO_DB_AWS_REGION>}
https://github.com/Netflix/metaflow/blob/2588f1d3030ebbd5ca3ab47a2fffb1eb9926855a/metaflow/plugins/aws/step_functions/dynamo_db_client.py#L13 https://github.com/Netflix/metaflow/blob/2588f1d3030ebbd5ca3ab47a2fffb1eb9926855a/metaflow/plugins/aws/aws_client.py#L10
But of course if you do have a different solution in mind, please feel free to disregard my proposal above
Thank you again for taking a look at this, and taking the time to work on this. Please let me know if I can assist in any way - I'd be more than happy to do so.
f
ahhh that is a bummer, so you're kind of out of luck here: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-access-cross-acct-resources.html know these docs are for cross-account access but there's this note at the bottom 
Currently, cross-Region AWS SDK integration and cross-Region AWS resource access aren't available in Step Functions.
which means that even if 
METAFLOW_SFN_DYNAMO_DB_TABLE
accepted the ddb ARN it would be a no go, I thought for some reason that metaflow wouldn't use that internally because the sfn direct integration is fairly recent
so you're kind of left with dynamo global table solution for this
which honestly isn't that bad since you don't have to pay for provisioned capacity and the data size is minimal that it would prob be under the free tier still (but don't quote me on that, dynamo pricing can be whacky)
d
@square-wire-39606 @proud-eye-90172 any suggestions on this rather than having to make a global dynamodb table?
s
@damp-lizard-48279 as Bryan correctly pointed out, the support is lacking with Step Functions on AWS's end - hence the reason why Metaflow doesn't expose it either. The DynamoDB usage by Metaflow is very minimal - the only reason to have it is to work around some of the limitations of Step Functions (which hopefully can be removed some day). Global tables could be a possible approach here.
d
Makes sense, thanks @square-wire-39606 @flaky-plumber-70709
g
@flaky-plumber-70709 @square-wire-39606 - Thank you for looking into this - I really appreciate it
4 Views
Open in Slack
PreviousNext
Powered by