https://outerbounds.com/ logo
Join Slack
Powered by
Hey metaflow, is there a cli argument to specify t...
# ask-metaflow
c
Hey metaflow, is there a cli argument to specify the aws batch queue name, like what you can provide in the @batch decorator?
✅ 1
v
run --with batch:queue=...
you can specify any attributes with that syntax
or you can use environment variable 
METAFLOW_BATCH_JOB_QUEUE
if you want it to have an effect only on steps that are already decorated with 
@batch
. 
run --with batch
applies to all steps
c
Thank you @straight-shampoo-11124! 🙏
👍 1
Follow up question @straight-shampoo-11124, is the cli --with the highest level of precedence? Reason I'm asking is I want to throw a little wrapper around some of the cli commands for our quants and the main issue is them sending tasks to the wrong queue. So when they run step function create (or whatever the command is) can I just use this and it'll override the batch queue variable in config.json in the metaflow settings or do I need to reload the whole thing again with the new queue arn?
v
yes, config provides defaults that can be overridden on the cli or through env vars
🙌 1
c
Excellent
Hey @victorious-lawyer-58417, so this is running on the queue it initially ran on, not the one in the cli command, anything I am doing wrong? 
python src/my_flow.py --environment=conda resume --with batch:queue=new-queue --origin-run-id sfn-c12757fd-d8f9-4bb7-b9ad-01bf9a39ae93 --tag prod start
v
is there a queue name hardcoded in the code?
c
nope
it is just passed in at build time (CI ) to the step function when it is created.
v
do you have 
@batch
in the code?
c
yes
all of our steps are decorated with this
v
ok, would it be possible to use 
@resources
instead? Or you can try the environment variable instead unfortunately the command line 
--with batch
won't override the 
@batch
in the code
c
so @resources will still run it on batch ?
v
with 
@resources
it'll run on batch when you do 
run --with batch
. It should work exactly as before, unless you need to hardcore some batch-specific attributes in the code (besides the usual cpu/memory)
c
ok so in the CI we just do 
step functions create
so I guess we just need to ensure that has the 
--with batch
yeah?
v
step functions
will automatically execute everything on batch, so 
--with batch
is not needed in that case Just when you do 
run
manually
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by