https://outerbounds.com/ logo
Join Slack
Powered by
The only thing available to fail at this point, th...
# ask-metaflow
c
The only thing available to fail at this point, that I see, is 
self.next(self.end)
, the function you see defined immediately below. What could be causing this to fail? UPDATE: Here's what's wilder. When I rename 
validate_model
to 
end
and I make that print statement the last thing that ever happens in the flow, we get to the print statement, and it prints! And then the flow...fails.
Check it out:
a
@cold-carpenter-83628 can you help me with the entire flow definition?
actually, even just the contents of 
validate_model
will work
c
@square-wire-39606 perfect! one moment please 🙂
Here's that step
LTVModel
is a custom model on our side, and 
validation_results.csv
contains all the appropriate numbers indicating that 
.compare_actuals
was successful.
a
if you comment out 
<http://results_test.to|results_test.to>_csv...
and reexecute - do you still run into an error? also, it might be quicker to iterate using 
python flow.py resume
- it will execute this failed task immediately
👀 1
c
I'll try that; I believe it will fail because I replaced a print statement with that line while this was failing, but I want to confirm that before I lie to you by accident 😅
among us party 1
@square-wire-39606 sadly, no dice 😢
a
if you only instantiate the client in this step and comment out all the other lines related to ltvmodel, does the task still succeed? I am trying to figure out if there is anything funny going on with either the 
bigquery.Client
or the 
LTVModel
class that is not allowing the step to terminate properly.
c
Hi again! (in multiple places now haha)
Well...that did succeed
Screenshot 2024-01-31 at 11.52.54 AM.png
Thing is a'course we do need the, uh, model to be verified in the verify model step 🙂
I'll start putting lines back in and see what happens
Okay, the lines where I instantiate and hydrate the model don't break it. I've now moved onto checking if it breaks when I add 
.compare_actuals
back in.
UPDATE: yes, it does. Consistently.
But that function runs successfully and the print statement after it runs, too. Before, of course, the step fails.
Very curious @square-wire-39606 if this is known/expected behavior 😅
I am currently examining workarounds, but the ones I am finding in the documentation are 
retry
and 
timeout
, and those are both for cases where a line of code _fails_; not a line of code succeeds and then the workflow fails. About to dig into the metaflow code itself to see if I can figure out what might be going on.
Update: dropped into a console and seeing....no data for the task in question. Is this weird?
a
Was 
compare_actuals
the culprit?
there isn't any data for the task because the task failed while it was trying to materialize this information
c
well, commenting it out makes the flow succeed and putting it back in makes the flow fail, but the flow does all the steps and gets to the print statement after 
compare_actuals
regardless.
a
can you share the implementation of 
compare_actuals
?
c
One would presume, if that line were failing, the lines after it should not run. They're not in a catch...finally block. They're just inline after the fact.
@square-wire-39606 I'll check it for anything secret first, but probably. One sec
It is worth noting, however, that there is no 
try catch finally
block in it that would cause it to complete even if there were a problem.
Huh. No revelatory data in any of these other attributes either.
But you can see the odd thing here, which is that the line after 
compare_actuals
that prints "Results saved!" does, indeed, run:
@square-wire-39606 I'll DM you the implementation of 
compare_actuals
and we can follow up here!
a
sure!
9 Views
Open in Slack
PreviousNext
Powered by