https://outerbounds.com/ logo
Join Slack
Powered by
Hello, I am having an issue when triggering a Flow...
# ask-metaflow
r
Hello, I am having an issue when triggering a Flow from Step Functions. The error states:
Copy code
Container Overrides length must be at most 8192
This is very similar to this issue (@square-wire-39606) I tried updating to the latest metaflow version, but still have the same problem
✅ 1
a
@red-fountain-16335 can you share the output of “python flow.py step-functions create —only-json”?
s
nm just saw the dm
👍 1
I will investigate and follow up
❤️ 1
r
One of the key things that changed (and cause the model no longer running via step functions) is that I am passing the 
--with=environment:vars='{"key":"value"}'
to ensure the remote execution uses the appropriate env variable (prod instead of dev etc.) I noticed when running the flow locally, it works fine if I set my local env variables (
export …
). But not sure how to enforce that into remote execution. Is there a special METAFLOW_ENVIRONMENT env variable I could use to keep track of that?
Hi, @square-wire-39606. Just following up: Indeed, without using the 
--with environment:vars
approach I am able to execute the Flow from step functions I am stuck now. What is the recommended way to define 
dev
vs 
prod
? I tried using 
Parameter
, but it does not work with 
step-functions create
step. Only when triggering it. But how can I define the parameter to be used on scheduled runs? I thought that was the whole point As a work-around, I was able to set the env variables via cli. But clearly that causes problems when executing from step-functions. I would like to schedule a training flow on a weekly basis, using 
prod
keyword (for example, in the Flow code, it will retrieve data from and write outputs to the appropriate environment) I would really appreciate your assistance on this
a
@red-fountain-16335 how many env vars are you passing using the env decorator currently?
r
just 2, 
SNOWFLAKE_ENV
and 
MF_NAMESPACE
, so I can control the snowflake and namespace to be used
Looks like I could manually edit the EventBridge Rule created and change the 
Parameters
there. But it seems silly/hacky Surely this is a common enough use case that would/should be supported natively by Metaflow What is the recommended way of doing this?
a
Ideally you should be able to rely on 
@environment
. I am chasing down an issue where the payload that is sent to AWS Step Functions has gotten quite big and is impinging on the length of parameters that can be passed.
👍 2
@red-fountain-16335 can you quickly try this PR out - https://github.com/Netflix/metaflow/pull/1215
once this works for you, I will merge the PR. This should give us plenty of head room for the time being without requiring anything drastic.
r
@square-wire-39606 legend!!! woohoo Just tested it and it works great. For future reference of others, here is my use case: • Parameters: for variables I might want to change, like adding a 
LIMIT 1000
to a sql query - but not applicable by default • Environment variables: define 
dev
vs 
prod
usage. When running/testing locally, 
os.getenv('ENV_NAME', 'dev')
means it will fallback to 
dev
as default value. But when running in production, as scheduled in step-functions, I can set the env variable `ENV_NAME=prod`: So my CI/CD deployment looks like this, running on merge:
Copy code
python <flow_name>.py --environment=conda --with=retry --with=card \
--with=environment:vars'{"ENV_NAME":"prod"}' \ 
step-functions create \
--tag prod
among us party 2
c
@brief-kite-90012
11 Views
Open in Slack
PreviousNext
Powered by