https://outerbounds.com/ logo
Join Slack
Powered by
Hey, guys! I try to launch simple toy code for su...
# ask-metaflow
b
Hey, guys! I try to launch simple toy code for subsequent deployment using 
argo workflows
, but I got the following error:
Copy code
(venv) denis@denis-TM1703:~/PycharmProjects/deploy$ python main.py --with retry argo-workflows create
Metaflow 2.9.14 executing ParameterFlow for user:denis
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
Deploying parameterflow to Argo Workflows...
    Metaflow service error:
    Metadata request (<https://5tm0zc5eg7.execute-api.us-east-1.amazonaws.com/api/ping>) failed (code 500): {"message": "Internal server error"}
Inside the link you can see: 
{"message":"Forbidden"}
Is it a problem connected with permission in aws? Thanks! P.S. I used this code: https://docs.metaflow.org/production/scheduling-metaflow-flows/scheduling-with-argo-workflows#pushing-a-flow-to-production
f
How did you create the underlying infrastructure?
b
In general, I totally followed this guide: https://outerbounds.com/engineering/deployment/aws-k8s/deployment/ My configuration 
config.json
looks like this:
Copy code
{
"METAFLOW_ARGO_EVENTS_EVENT":"metaflow-event",
"METAFLOW_ARGO_EVENTS_EVENT_BUS":"default",
"METAFLOW_ARGO_EVENTS_EVENT_SOURCE":"argo-events-webhook",
"METAFLOW_ARGO_EVENTS_SERVICE_ACCOUNT":"operate-workflow-sa",
"METAFLOW_ARGO_EVENTS_WEBHOOK_URL":"<http://argo-events-webhook-eventsource-svc.default:12000/metaflow-event>",
"METAFLOW_DATASTORE_SYSROOT_S3":"<s3://metaflows3cd4jylvb/metaflow>",
"METAFLOW_DATATOOLS_S3ROOT":"<s3://metaflows3cd4jylvb/data>",
"METAFLOW_DEFAULT_DATASTORE":"s3",
"METAFLOW_DEFAULT_METADATA":"service",
"METAFLOW_KUBERNETES_NAMESPACE":"default",
"METAFLOW_KUBERNETES_SERVICE_ACCOUNT":"argo-workflow",
"METAFLOW_SERVICE_AUTH_KEY":"xxx", # I'm not sure I can share with it :D
"METAFLOW_SERVICE_URL":"<https://5tm0zc5eg7.execute-api.us-east-1.amazonaws.com/api/>"
}
a
don't share METAFLOW_SERVICE_AUTH_KEY with me but can you verify it looks like ~40 random chars?
if its a much shorter string most likely its the API gateway key ID not the key itself. You'd need to go to awsconsole to get the actual key
b
don't share METAFLOW_SERVICE_AUTH_KEY with me but can you verify it looks like ~40 random chars?
yes, exactly @average-beach-28850
@average-beach-28850 do you have any ideas how to fix this problem? I totally followed the guide with small fixes of mine.
f
So we deployed a new stack and are facing same issue.
b
sad.. If we resolve it faster than you - I will let you know 🙂
a
ok let me try to reproduce, is it just argo that doesn't work? or just simple 
myflow.py run
doesnt either?
f
For me when I try to create a flow using 
argo-workflows create
I get the exact error with 500
b
@average-beach-28850 hm, I just tried to launch 
python3 main.py run
and get error 500 too Previously I didn't have this problem with simple 
main.py run
Copy code
(venv) denis@denis-TM1703:~/PycharmProjects/deployment$ python3 main.py run
Metaflow 2.9.14 executing ParameterFlow for user:denis
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
    Metaflow service error:
    Metadata request (/flows/ParameterFlow) failed (code 500): {"message": "Internal server error"}
a
I have tried it now and it seems to work for me; however i did upgrade some module versions to newer ones before running it. Let me push my code..
pushed to https://github.com/outerbounds/terraform-aws-metaflow/tree/master/examples/eks_argo . If you're doing this from scratch, I'd recommend starting from clean slate, i.e.: 1. run terraform destroy to nuke what you deployed so far (this will destroy S3 bucket that it created with artifacts! but if you haven't run anything important yet, thats fine) 2. pull the latest version of eks_argo example with my changes 3. deploy again following the instructions
(its possible to update your deployed infra with my changes without destroying it first, but its usually easier to debug starting from clean slate)
b
but thank you so much! I'll try it again with your recommendation. Thanks!!
Clause #4 It says that I should be ready to run metaflow using 
@kubernetes
. Am I need to decorate first step I have in my 
FlowSpec
? Or what does it mean here?
a
you can decorate certain steps using 
@kubernetes
so they run on kubernetes. You can also leave it out, but do 
python flow.py run --with kubernetes
and that has the same effect as adding @kubernetes decorator to all steps
I usually do it this way (
python flow.py run --with kubernetes
that is) so I can easily switch between running my flow locally and on kubernetes, without changing code
b
thank you so much! I'll try my best!
👍 1
a
let me know if it still doesn't work, we can debug further. Just want to make sure your setup and mine (that works) are exactly the same before digging deeper
b
okay, of course! thank you again!
17 Views
Open in Slack
PreviousNext
Powered by