https://outerbounds.com/ logo
Join Slack
Powered by
Hello, I have a flow with `foreach` that result in...
# ask-metaflow
m
Hello, I have a flow with 
foreach
that result in 300 child-tasks, some of them failed, how can I access their 
input
data to know which data they worked on so I could re-run it?
tried 
task.data.input
and got
Copy code
return self._artifacts[name].data
KeyError: 'input'
my 
task
type is 
<class 'metaflow.client.core.Task'>
Any other ideas?
s
If you resume the flow, only the tasks that failed will be resumed
m
I ended up reading the data from the parent step (that completed) and cross reference it with data from the logs. I couldn't just resume the flow since I had steps that failed silently (I catch the error). If I resume and I have steps that failed due to 
pod deleted
will they re-try?
s
yep
👍 1
m
@square-wire-39606 I don’t that this is an old one, but any indiction on why 
task.data.input
is failing to access the task input data?
a
If the task never started, there is a possibility that task.data.input might not be accessible
m
The task started and completed (it had an error but we catch it)
a
And it’s part of the foreach?
task.data would list all the attributes that are accessible
m
yes, part of the 
foreach
are still runing
a
What is the output of task.data?
One more question - the immediately preceding step is where you do self.next(self.foo, foreach=blah)?
m
Copy code
<MetaflowData: bucketed_data, configs, sessions, flow_link, manifest, name, ns, since, until>
bucketed_data
is the variable we used for the 
foreach
with 
self.next(self.generate_data, foreach="bucketed_data")
I’m using 
metaflow==2.9.11
a
Interesting - and you are able to access task.data.input for older runs of the same flow and same step?
m
checking now….
a
Also if you can create a very simple foreach flow and verify if you are running into the same issue, that will help us reproduce the problem
m
in another process that already completed I still can’t access 
input
and get the error 
KeyError: 'input'
for the step after that is defined with
Copy code
def write_to_storage(self, inputs):
for 
task.data
I get
Copy code
<MetaflowData: name, flow_link, until, since, ns, manifest>
a
Yes that’s the join step - it doesn’t have the property task.data.input since it is getting many inputs - if you want to store any of those inputs you can inspect the inputs variable (container of all the artifacts from all the parents) in your code and assign the right values to artifacts
The property task.data.input works on the step that immediately succeeds the one that has self.next(self.foo, foreach=bar) - the join step is the one after that
m
got it, are you able to reproduce it on your end?
a
Yes this is expected behaviour. In this example (https://docs.metaflow.org/metaflow/basics#foreach), task a will have task.data.input available but not the task join.
The join step doesn’t have one input but many inputs each of them coming from the multitude of immediate parents that feed into it
m
I understand I can’t access it in 
join
step, I just mentioned it to have more examples. What about accessing 
task.data.input
in one of the 
foreach
tasks?
a
Yes you should be able to access that without any issues
m
I just managed to reproduce it using a simple flow. I used this flow, after it completed I ran
Copy code
step = Step('ForeachFlow/3484/a')
for t in step.tasks():
    print(t.data.input)
and got 
KeyError: 'input'
@square-wire-39606 hey updates on this one?
s
@many-processor-72561 sorry for the delay. it seems that this is an oversight and we are discussing internally to address it
👍 1
6 Views
Open in Slack
PreviousNext
Powered by