Hi, I have a question around artifacts, and huge amounts of time being wasted between steps. Issues: 1. See first_step. The print step shows me that the step took 8 minutes to run, however, the time it takes between that print statement (the last line of code before the self.next) and for the "task completed" message is >5minutes! I assume this time is spent in passing the artifacts to the next step (?) 2. In the second_step, it takes me 10 minutes to retrieve the artifact (df). So in total, it takes me an additional 15minutes of runtime, just because I've split up the steps. I'd like to have the steps, for a number of reasons (the same reasons most people use steps in metaflow). So before I jump into the obvious solution of combining into a single large step, I'd like to see if there's anything obvious I'm missing, or can do to avoid this major time wastage. Here's some skeleton code to supplement my text , above

import time

    @batch(cpu=8, memory=125000, use_tmpfs=True)
    @step
    def first_step(self):
        start_time = time.time()
        # df = blah##
        # self.df = df
        print("time for this step: {} seconds".format(time.time() - start_time))
        #### takes 5 minutes between this print statement and to finally say "task is completed"
        self.next(self.second_step)

    @batch(cpu=8, memory=125000, use_tmpfs=True)
    @step
    def second_step(self):
        start_time = time.time()
        retrieved_df = self.df # this takes 10 minutes!!!
        print("time for retrieving artifacts: {} seconds".format(time.time() - start_time))
        self.next(self.second_step)

I see you have

self.df = df

commented out in

first_step

above. Do you know how long it takes without the dataframe? It'd be good to understand how much is baseline latency of launching the instances vs. processing the dataframe

I just had it commented in this example I wrote. without the dataframe it takes approx 1 minute (find the EC2, load everything up). I have isolated every line of code (timed every line), and retrieving the artifacts is the major cost in my step. I looked into it a bit more, and noticed that a bunch of my newly-introduced columns have float64s, so this may be increasing the memory of my DF, causing it to explode. So I am currently looking into that, and will see if it fixes. may not, but hopefully

by default Metaflow uses Python pickle to serialize artifacts which can be inefficient with large dataframes. Try this approach to serialize the df as parquet instead: include these utilities and write in

first_step

self.df = to_parquetbytes(df)

and then

second_step

df = to_dataframe(self.df)

awesome! thanks, i'll try that out

while serializing the dataframe, it'll need to keep a few copies of it in memory, so you may need to increase

@resources(memory=)

to make it work

ah ok, i am using aws, with @batch, and already specifying memory there...so @resources won't do anything extra, yeh?

oh right, increase memory in the

@batch

decorator

ok thanks - will give it a go now

If the method Ville outlined doesn’t work, when dealing with big DFs 100+GB I just store them in S3 in parquet format then store a pointer to the S3 path as a Data Artifact.

yeh the above method didn't work. I may store to s3 if needed...or just combine into one big step. Thanks both for the tips