Hello! I am having an issue where it seems to have memory leakage when running the Flow remotely on AWS Batch. More especifically when storing data as artifact, such as

self.df = df

, it seems to spike the memory usage! I have already increased the

@resource

to use 30Gb memory and it is still failing with

OutOfMemory

error. Funny enough, I can run the flow locally and it almost bricks my computer, but it does complete fine. The laptop only has 16Gb ram and had other programs running, such as Chrome, Teams, Outlook etc. Note: the dataframe.info() shows usage of about 3Gb, for ~7 million rows with 100+ columns. It feels like storing artifact is causing duplication of memory instead of just pointing to the variable reference. Am I missing something? Is there a best practice/pattern to follow? My flow is defined as the following steps: 1. Start: pandas.read_sql with a simple

select * from …

Snowflake 2. Transform: applies additional feature engineering, such as one hot encoding (get_dummies) -> it runs fine, running out of memory on the last part where I store it as class attribute to be used on subsequent steps 3. Train Models in parallel 4. Select best model and store it as artifact

hey! there's some overhead at the end when the artifact is stored, which is what's causing the issue likely. Pandas is not particularly memory-efficient, so even a relatively small amount of data can consume a surprising amount of memory depending on the column types

Thanks, @victorious-lawyer-58417. I actually read your book and tried implementing PyArrow/Parquet instead of dataframe - but there were two issues: 1. As you pointed out, there were additional data manipulation that required using pandas 2. For some reason, the method

df.read_sql

querying Snowflake performed muuuuch better than the alternative: a. Snowflake to S3, using a COPY INTO data_type=parquet b. Using the metaflow.S3.get_recursive / or just using pyarrow’s

<http://dataset.to|dataset.to>_table(usethreads=True)

c. Finally, converting the parquet table into a dataframe

df = <http://table.to|table.to>_pandas(split_blocks=True, self_destruct=True)

I am trying to shove most data transformation to the data-warehouse side, so the Metaflow Flow becomes simpler and can leverage the benefits of pyarrow. But it requires a whole lot of effort, whereas the idea was to move quick from prototype (Notebook) to production (running on a schedule with Step Functions) I like the

to_parquetbytes

/`to_pandas` idea. I will give it a try. I am trying to optimise the developer experience, making it super easy to onboard new users - which means minimising extra details to be learned. But this sounds like a feasible compromise 👍 🤔 The only thing that still bugs me is that the actual Flow completed successfully in my local Mac (with only 16Gb ram while running other programs), but looking at the resource usage in AWS I can see mega spikes going from 10 to 20+ Gb)

nice! Good to hear that it helps. Eventually we'd like to handle this automatically but a small gotcha is that not all Pandas dataframes can be converted to Parquet losslessly

@red-fountain-16335 what tool are you using to view the memory usage of a run?

@ambitious-bird-15073 that was just AWS container insight. I am using Fargate as the Batch backend, so required to enable “container insight” to surface resource usage at the container task level

@red-fountain-16335 thank you!

@red-fountain-16335 is it possible to view this chart per batch job via ContainerInsights? Correct me if I am wrong but the plot you show works across the entire Fargate ECS Cluster right?

You are correct @curved-island-17262, that is a limitation. The chart shows details for the whole container, not per individual tasks. There might be a way to further tag it, so it can be filtered, but I haven’t explored that yet. Additionally, this is using Fargate as the Batch backend. The operational monitoring Might be simpler if using EC2