https://outerbounds.com/ logo
Join Slack
Powered by
Hi team, Can older versions of Metaflow still be ...
# ask-metaflow
a
Hi team, Can older versions of Metaflow still be used? Giving the 
--use-distributed-map
mode from the PR (https://github.com/Netflix/metaflow/pull/1576) a try, but got the following error from the batch logs.
Copy code
Stdout: Failed to install Micromamba!
Stderr:
Bootstrap failed while executing: if ! command -v ./micromamba >/dev/null 2>&1; then
wget -qO- <https://micro.mamba.pm/api/micromamba/linux-64/latest> | tar -xvj bin/micromamba --strip-components=1 >/dev/null 2>&1;
export PATH=$PATH:$HOME/bin;
if ! command -v ./micromamba >/dev/null 2>&1; then
echo "Failed to install Micromamba!";
Bootstrapping virtual environment...
Task is starting.
Code package downloaded.
Downloading code package...
Setting up task environment.
Oh wait hmm it might be an older version of Metaflow requires a different assortment of OS packages. Looks like it's trying to install 
micromamba
, but can't
s
do you have an image that you have been using with recent versions of Metaflow? If your image includes 
micromamba
it should work
@square-wire-39606 may have other guidance too
a
Thanks Ville. I'll check with @sparse-florist-36640 tomorrow on his custom image he's using with the Batch workers. Normally works with latest versions of Metaflow though - can confirm that.
👍 1
Hey @victorious-lawyer-58417 and @ancient-application-36103, It almost worked, but looks like it throttles... Ran into this error below. Probably sending too many AWS Batch requests at once.
Will see if we can increase our API quota for this
Looks like the value is not adjustable, so will need some guidance please 🙏 Update: Looks like a 
retry
might be the way to go here? Reference: https://outerbounds-community.slack.com/archives/C020U025QJK/p1700150433974189?thread_ts=1695412569.760469&cid=C020U025QJK
a
did it work with retries?
a
Trying it now 🙂
I think Batch array job integration would make sense here
Nope - no luck! Didn't work with 
retry
😞
Copy code
python3 "history_extractor_flow.py" --with retry --environment=conda --production ${WITH_BATCH} step-functions create --max-workers $MAX_WORKERS --use-distributed-map --tag $2 --tag $METAFLOW_USER

python3 "history_extractor_flow.py" --environment=conda --production step-functions trigger
s
What was the error?
a
Same error unfortunately. Still hitting quota limits
s
what is the max-workers that you are setting?
a
512
Looks like Bennett was able to get it working w/ 1000 though so I'm perplexed
s
yeah it will be a function of what the quota limits are as well
a
I was under the impression everyone had 50 because it's not adjustable
s
when you set the retry do you see the batch api call being retried?
it might be adjustable if you ping your AWS TAM
there are plenty of undocumented limits that a TAM might be able to take care of
a
Yeah, I pinged our TAM and should hopefully get a response tomorrow 🙏
But everyone else from AWS has directed me to use Batch array jobs
when you set the retry do you see the batch api call being retried?
From cloudtrail event logs?
nod 1
a
Cool - yeah batch array would be the next option to explore
or the step-functions UI
you should see more events
a
Nope I don't see them being retried in Step Functions UI
a
if the retries are happening as expected, one option is to add long jitters in between retries. it's something step functions supports now and you should be able to manually edit the step function state machine json to try it out
a
Ah yes they recommended that as well. But that seems a bit removed from Metaflow when we start modifying via the console
s
if that works out for you, we can introduce support for it in Metaflow
right now, my goal is to see which approach works well that can be in turn memorialized in the code base
a
Gotcha makes sense. Let me give it a try. I need to first check if it's actually retrying though because it doesn't appear as though it does
🤞 1
Thanks Savin!
❤️ 1
I'm stumped. We added a retry block manually to the state machine and now it seems to be working. It seems that Metaflow didn't transmit the retry configuration to the state machine. Although, I can see it in the state machine definition json.
a
It could be a bug with the PR
a
Ok I uncovered the issue now! The 
retry
does appear in the definition of the state machine, BUT only for the first state and no others. So somehow 
python3 "history_extractor_flow.py" --with retry --environment=conda --production ${WITH_BATCH} step-functions create --max-workers $MAX_WORKERS --use-distributed-map --tag $2 --tag $METAFLOW_USER
didn't propagate the 
retry
to all the states in the state machine?
Other than that, it worked once I manually configured the retry.
a
Cool. Could be a bug in the PR. Can you verify this PR works at your scale?
a
That will be the next test. I'll probably let @sparse-florist-36640 handle that. Will keep you posted.
👍 1
Hey @ancient-application-36103, I think it is a bug? I think 
--with retry
creates the 
RetryStrategy
block in the resource definition, but it's actually the 
Retry
block that seems to facilitate the retries, and that's the one that is missing (and that I had to enter manually). Am I on the right track?
a
seems correct - i will have to brush up on amazon state languages a bit though 🙂
do you see the same behavior with and without the PR?
a
Yeah I wanna test that out next 🙂
Hey Savin, Even without the PR (i.e. using the latest version of Metaflow), the 
Retry
block doesn't show up in the state machine definition. Is there anything wrong with this step functions invocation command?
Copy code
python3 "history_extractor_flow.py" --environment=conda --production --with batch:cpu=1,memory=8000,queue=gaiatrn-p-ue1-cpu-amp --with retry:times=3,minutes_between_retries=2 step-functions create --max-workers $MAX_WORKERS --tag $2 --tag $METAFLOW_USER

python3 "history_extractor_flow.py" --environment=conda --production step-functions trigger
Other than that, We can confirm that step functions distributed mode works quite well for the kind of scale that we need. Could we sync soon to discuss next steps? • Merging the PR • How to integrate Batch array job (we don't need nesting for this particular use case) • Why the 
--with retry
isn't working for the state machine?
a
Sure - I can spend some time this week with the PR and follow up early next week to discuss next steps
Do you want to set up some time for Tuesday? I am not sure if @salmon-exabyte-11054 is following this thread, but would be good to loop him in as well
a
Thanks Savin. @salmon-exabyte-11054 let me know if you want an invite to the meeting.
P.S. Talked to our TAM and 50 batch submit jobs is the hard limit and cannot be modified. So for our next sync, I want to talk about integration with batch array jobs. Even a plugin (similar to 
metaflow-ray
) will work for our use case. This flow created by @sparse-florist-36640 has no nesting involved, so batch array not supporting nesting on Metaflow should not be a problem.
P.S.S Just in case you're curious - we got a ~12x speed boost with step functions distributed map mode 🙂
noice 1
🚀 1
s
👋 hey folks….I have some additional changes I am using locally that I can merge back into the PR. I’ll try and get that in place before tuesday so folks can take a look
🙌 2
thankyou 1
FYI for the retries I specifically scoped it to a batch error not a code one — it’ll be more clear in the PR
@acoustic-van-30942 I’d love an invite bhiles@perennial.earth 😄
a
Cool Bennett! Will send an invite now.
s
thankyou 1
a
Hi folks, Just wanted to follow-up regarding ETA on the merge for this PR that Bennett authored. Looks like it's been approved by one reviewer so far.
a
Can you try this PR out at your scale?
s
@square-wire-39606 wow — your change is so much smaller than what I had, that’s awesome. I haven’t had a chance to try it out myself, but maybe next week 🤞
a
Thanks guys, Unfortunately, neither Thomas or I have cycles to test this PR. We hope to resume this in a couple of weeks
23 Views
Open in Slack
PreviousNext
Powered by