https://outerbounds.com/ logo
Join Slack
Powered by
Hello, I would be surprised if my question hadn’t...
# ask-metaflow
i
Hello, I would be surprised if my question hadn’t been asked yet but I see no trace of something similar in this slack. Sorry for any inconvenience. I have successfully deployed my metaflow workflow as step functions. And I got a dedicated namespace for this deployment. It looked like: “production:my-flow-XXX” I then started the step-function and when it’s finished (with success), I tried to look at the artifacts: 
namespace("production:my-flow-XXX")
Metaflow().flows
but an empty list was returned. So, what did I do wrong?
✅ 1
v
can you try this
Copy code
namespace(None)
list(Metaflow())
the 
namespace(None)
bit shows everything across namespaces, ruling out any typos etc
i
Yes, in that case, I see all my workflows. However, if I try to access the runs(), I only got runs that I started locally on my machine (that is, not the step functions runs)
v
do you deploy to step functions on the same machine with the same Metaflow config? The client API uses the config to figure out where to look for runs. If the config used by step functions is not available, it might see only local runs
i
yes, it is on the same machine. I have just tried another experiment: renaming my flow, created S-F machine from it and run S-F. That is: this new flow has never been run on my machine, only from S-F. And, in that case, list(Metaflow()) does not returns it.
it seems list(Metaflow()) only returns flows that were run at least once on my machine.
v
you called 
namespace(None)
before calling it? Otherwise it’ll show only your local runs
i
yes I did call namespace(None)
It is really puzzling
v
can you do
Copy code
from metaflow import get_metadata
print(get_metadata())
it should output something like 
service@
...
i
It outputs: local@/path where ‘path’ is the path to my project.
v
cool - so that's the issue: somehow it uses a different config when deploying to Step Functions since SFN uses S3 and a stand-alone metadata service
i
So : Which config does it uses? I have only one config.json
v
are you executing the commands above in a notebook or in a Python script?
i
in both , same outputs
v
and you ran 
step-functions create
on the same machine on CLI manually?
i
yes indeed
v
if you run 
metaflow configure show | grep DEFAULT_DATASTORE
on the CLI, what does it say?
i
METAFLOW_DEFAULT_DATASTORE=s3
BTW: the metaflow info is in S3 (sfn-* folders)
After I run my SF machine
v
cool, and what do you get with 
metaflow configure show | grep DEFAULT_METADATA
?
i
Oh! I get nothing.
v
have you deployed a metadata service and an RDS e.g. using the default cloudformation template?
i
I don’t think so. In fact, I just added METAFLOW_SFN_DYNAMO_DB_TABLE to my config plus a bunch of required roles. I don’t think I did anything special with default cloudformation template
v
ok, you should configure the metadata service too. You can run 
metaflow configure
to do it or add 
METAFLOW_SERVICE_URL
and 
METAFLOW_DEFAULT_METADATA=service
in the config
Metaflow uses a central database to track execution across SFN and local executions. If you don't have it configured, you can only see local runs
i
Ok, I’ll try that and will let you know. Thank you very much for your help and your time.
👍 1
v
don't hesitate to follow up here if you don't get it working easily (and even if you do 🙂 )!
9 Views
Open in Slack
PreviousNext
Powered by