Hi Metaflow team, working through the last steps o...
# ask-metaflowc
clean-megabyte-14460
02/03/2023, 9:51 PMHi Metaflow team, working through the last steps of our POC for metaflow vs some other platforms, and I'm caught up on an IAM Permission for AWS. I don't know whether or not the cloud formation template sets up a role with sufficient permissions for both s3 and sagemaker (where we would be deploying endpoints). I am a bit out of my depth and can't find any information while googling
✅ 1
a
ancient-application-36103
02/03/2023, 9:52 PMHey Daniel - the cloud formation template is a starter template that deploys just enough resources to get started with Metaflow. If you can elaborate a bit about your setup, we can definitely help with more pointed recommendations.
c
clean-megabyte-14460
02/03/2023, 9:57 PMEverything is baseline from the cloud formation script, the metadata service is up and running, batch works, the only problem I'm having is the final deployment permissions on IAM I think.
I have deployed using the standard sagemaker role in the past, but when trying to deploy from metaflow itself I get:
Copy code
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Could not access model data at <s3://metaflow-s3>… Please ensure that the role "…" exists and that its trust relationship policy allows the action "sts:AssumeRole" for the service principal "<http://sagemaker.amazonaws.com|sagemaker.amazonaws.com>". Also ensure that the role has "s3:GetObject" permissions and that the object is located in us-west-2.clean-megabyte-14460
02/03/2023, 9:57 PMThis leads me to believe I need a metaflow role created by the devops team with the permissions for sagemaker and s3
clean-megabyte-14460
02/03/2023, 9:57 PMJust want to make sure I'm on the right track before I open a ticket as I don't have much privilege within our cloud (for good reason 😓 )
a
ancient-application-36103
02/03/2023, 10:47 PMThat’s correct. You would want to ensure that the role has permissions for sagemaker hosting
c
crooked-jordan-29960
02/03/2023, 10:53 PMHey Daniel! There is a short post on the basics of this topic in the Metaflow context here. I think you are on the right track to think that how to select a role depends on how your devops team wants to handle AWS IAM policies. Here is a reference on API permissions related to Sagemaker, and a link to AWS managed policies you can use to streamline the process if feasible.
❤️ 1
c
clean-megabyte-14460
02/03/2023, 11:05 PMThanks @crooked-jordan-29960! This is the page I've been working off of, I'll dive a bit deeper with my devops team to see whats up
👍 1
clean-megabyte-14460
02/04/2023, 1:41 AMWell I made it one step further.
Now I've got my tar.gz sitting in the bucket and I can read it, I downloaded and tested it and its functional, but I'm getting this error when creating the endpoint. It runs for about 7 minutes in the creating stage and then throws this error.:
Copy code
sagemaker.exceptions.UnexpectedStatusException: Error hosting endpoint trial-propensity-1675473598147-prod: Failed. Reason: Failed to download model data for container "container_1" from URL: "bueketinfo/112/model.tar.gz". Please ensure that there is an object located at the URL and that the role passed to CreateModel has permissions to download the object..clean-megabyte-14460
02/16/2023, 8:44 PMIf anybody stumbles across this, it was a KMS decrypt issue because the metaflow bucket had default encryption and the IAM Role lacked the key/decrypt policy.
4 Views