https://outerbounds.com/ logo
Join Slack
Powered by
Question about artifact serialization -- suppose I...
# ask-metaflow
w
Question about artifact serialization -- suppose I have a flow that looks like the following:
Copy code
class LinearFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.a)

    @step
    def a(self):
        self.my_gigantic_var = 'xxx'
        self.next(self.b)

    @step
    def b(self):
        do_stuff(self.my_gigantic_var)
        self.next(self.end)

    @step
    def end(self):
        pass
Assuming step 
b
does not modify 
my_gigantic_var
, will 
my_gigantic_var
be serialized as an artifact again when step 
b
ends? As additional context, I noticed that my flow is getting OOM errors after 
do_stuff
has completed, which I'm guessing is due to serializing the artifacts
✅ 1
s
it needs to serialize 
my_gigantic_var
in b to see if it has changed. It has some overhead so you need to adjust 
@resources
accordingly when dealing with gigantic artifacts
w
do i have to specify 
@resources
again even if i specified it via 
@batch
?
s
no, just make sure the 
memory
is high enough
w
hmm ok.. i think i'll just delete 
my_gigantic_var
before calling 
end()
then
aside: what do people usually do in 
end()
? i never have anything in there
s
you don’t have to delete it. Artifacts are loaded lazily so if you don’t use it, it doesn’t make a difference
w
Artifacts are loaded lazily so if you don’t use it, it doesn’t make a difference
well i'm using it in step 
b
but not 
end.
Are you saying it's not going to be serialized between ``b`` and ``end`?`
s
you can push results to outside systems in 
end
or not. It’s useful that all flows have an unambiguous beginning and end, which is why those two steps exist
it’s serialized in b to see if it has changed. Python doesn’t have constants so it’s hard to know otherwise. If it’s not changed, it is not uploaded redundantly. However just the serialization about doubles the memory footprint of the artifact. For small artifacts it doesn’t make a significant difference but for gigantic ones it does, like in your case
w
so if i delete it before calling 
self.next(self.end)
that will avoid the check (and thus re-serialization) to see if it has changed right?
v
the check happens at the end of 
b
so it’s unavoidable if you use the artifact in 
b
w
to be specific, i am proposing:
Copy code
@step
    def b(self):
        do_stuff(self.my_gigantic_var)
      
        delattr(self, "my_gigantic_var")

        self.next(self.end)
how come this doesn't avoid the check? it shouldn't know to check 
my_gigantic_var
since it's gone from 
self
v
ah right, that might/should work
indeed it checks active instance variables (artifacts) so if it can’t find it, it can’t do anything with it
👍 1
4 Views
Open in Slack
PreviousNext
Powered by