https://outerbounds.com/ logo
Join Slack
Powered by
hi what is this `_foreach_stack` thing? i'm gettin...
# ask-metaflow
w
hi what is this 
_foreach_stack
thing? i'm getting an error about 
*** AttributeError: Flow StandardModelBuilder has no attribute '_foreach_stack'
and am confused how that is even possible?
u
this is an internal attribute to keep track of the current level/info of nested foreach/split (if there is any). could you share the full stack trace?
w
Copy code
Project: model_builder, Branch: user.SFN
    Internal error
Traceback (most recent call last):
  File "/home/hadoop/metaflow/metaflow/cli.py", line 1176, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/home/hadoop/metaflow/metaflow/tracing/__init__.py", line 27, in wrapper_func
    return func(args, kwargs)
  File "/home/hadoop/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
  File "/home/hadoop/metaflow/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/hadoop/metaflow/metaflow/_vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hadoop/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
  File "/home/hadoop/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
  File "/home/hadoop/metaflow/metaflow/_vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, args, kwargs)
  File "/home/hadoop/metaflow/metaflow/cli.py", line 258, in dump
    datastore_set = TaskDataStoreSet(
  File "/home/hadoop/metaflow/metaflow/datastore/datastore_set.py", line 47, in __init__
    self.pathspec_index_cache[ds.pathspec_index] = ds
  File "/home/hadoop/metaflow/metaflow/datastore/task_datastore.py", line 205, in pathspec_index
    idxstr = ",".join(map(str, (f.index for f in self["_foreach_stack"])))
  File "/home/hadoop/metaflow/metaflow/datastore/task_datastore.py", line 45, in method
    return f(self, args, kwargs)
  File "/home/hadoop/metaflow/metaflow/datastore/task_datastore.py", line 836, in __getitem__
    _, obj = next(self.load_artifacts([name]))
  File "/home/hadoop/metaflow/metaflow/datastore/task_datastore.py", line 359, in load_artifacts
    to_load[self._objects[name]].append(name)
KeyError: '_foreach_stack'
d
what version of metaflow is this and was it working before?
2.11.1 made a small change around there but this 
_foreach_stack
variable should still be around.
w
It was working before, but we're trying to use a different authentication mechanism for the Metadata service (not sure if this is the root cause). Does the version of the Metadata service need to be the same as the version of Metaflow? What does the Metadata service do anyway?
d
it doesn’t need to be the same version necessarily. The metadata service records all the runs/steps/tasks/articacts and other info there. This error though is not in the metadata service but in the S3 saved metadata
could you confirm which version of metaflow it worked with and which version it doesn’t work with?
w
looks like metaflow 
2.11.0
(for both working and non-working). The non-working stuff is happening after making the metadata authentication changes (which doesnt seem to be related according to your comment)
d
I mean it shouldn’t but if that’s the only thing that’s changed, it seems possible. if it was a metaflow version change, I would have started there.
w
is it possible that the flow was created on one version of the metadata service, and now we're trying to run it hitting a different version of metadata service so the stuff in s3 is messed up?
idk what version(s) the metadata service is running.. they were installed a while ago but likely are running different versions. Is there a way to check?
4 Views
Open in Slack
PreviousNext
Powered by