Hey all. I’m running a flow, let’s say A=>B[]=>C, but when I join the foreach (from B) in C, it’s telling me that I have to explicitly set the artifacts generated in A prior to merging. That’s a bug, right?

Not really. Here is more context - https://docs.metaflow.org/metaflow/basics#data-flow-through-the-graph

But these artifacts are common

Merge_artifacts gives you the “pass common artifacts” down behavior you want but by default it doesn’t try to “be smart”. Determining common artifacts is not an exact science (pickle is unstable, etc) so metaflow tries to be conservative. Note also that Python gives very little in the way of creating a “const” (non modifiable) object so accessing artifacts (but not modifying) looks the same to metaflow.

But these artifacts aren’t being set here

if you have the following: • step A: self.foo = 42 • later step B: x = self.foo then, in step A: •

42

will be serialized and stored with

foo

as the key in step B: •

42

will again be serialized and stored with

foo

as the key. The difference though here is that if

42

serializes to the same thing as in A (it should in this case but it is not guaranteed) then it won’t be actually stored (uploaded) but we still have to go through the process of serializing. The reason for that is that we have absolutely no way of knowing that you didn’t modify

foo

in some way. Python does not allow us to make this type of assumption easily.

foo

could be a complex object that contains “nested” structures and you could be modifying something way deep in it. Metaflow takes a “safe” if not optimal approach and anything that you access/set is re-serialized. We do optimize, as I mentioned, and only store objects once though (it’s a content addressed store in effect). The

merge_artifacts

function will not access the artifacts that are merged so it is typically faster that way. It makes the assumption (which is again not fully accurate but errors out in a “safe” way) that two objects are identical if they have the same hash (it typically errors out when two objects are the same but have DIFFERENT hashes).

Is there a way to stop that? What if I access self.foo from B and then delete self.foo right after. Would that stop it from getting serialized and stored again?

if you deleted it, yes, it would not get saved again.

So in our example, if I delete self.foo from within B, it would no longer be accessible in later steps, right?

hum, that’s a good question actually. I suspect that it may still be but let me check.

I appreciate the help!

as I suspected:

from metaflow import FlowSpec, step

class TestFlow(FlowSpec):
    @step
    def start(self):
        self.x = 42
        self.next(self.a)

    @step
    def a(self):
        print("Reading x to be %d" % self.x)
        del self.x
        self.next(self.end)

    @step
    def end(self):
        print("Reading x to be %d" % self.x)

if __name__ == "__main__":
    TestFlow()

works

Whoah. That breaks my mind. Why did that work?

so in your case, if you have B as a foreach step and C was the join step, you would have access to it in C (through the

inputs

list) but it would not make it past C unless you merged it or assigned it.

I thought that would only work in a foreach maybe

it works because we are lazy 🙂. More seriously, at the start of each step, the step has what we call a datastore which effectively is a mapping between names and hashes. It’s just a mapping. When you want to access a variable, we do two things: • look up in the

self

object if the variable is there — if it is return it and we are done, if not • look up in the mapping — if it is there, we load from the backing storage layer (S3, etc), deserialize it and add it as an attribute to

self

and return it Then, at the end of the step, we basically create the state of the datastore at the end of the step and we form it by: • taking the datastore at the beginning (so the mapping I talked about) • updating it with anything that is in

self

(so we replace anything in the mapping with stuff in

self

) Crucially, when you do

del self.x

, you just remove stuff from

self

but not from the mapping so it still goes through.

Ohhhh ok that’s good to know!

glad to help. Let us know if you have more questions. I will admit it is a bit quirky but yes, if you access a very large object and don’t want to reserialize it because you are absolutely sure it has not changed, you can use the trick of deleting it and it will still be “accessible” but won’t be re-serialized.