Hi all We are running into a somewhat bizarre problem when r Outerbounds #ask-metaflow

Hi all, We are running into a somewhat bizarre pr...

shy-refrigerator-15055

03/23/2023, 6:44 PM

Hi all, We are running into a somewhat bizarre problem when running flows in AWS Batch. The largest instance in our compute environment is a

c7g.16xlarge

(i.e. 128GB RAM). When any step ends up running on this instance, querying total memory with

psutil

returns ~128GB. However, the actually available memory is what is specified in the @batch decorator. This leads to problems when any function tries to police its own memory use (e.g. output errors when available memory is too low, or restrict its memory use to a fraction of system memory). Is there anything fundamental we are missing here on how this could be avoided or fixed? Minimal working example in 🧵, thanks!

✅ 1

shy-refrigerator-15055

03/23/2023, 6:46 PM

This is our flow, which allocates memory for a large numpy array. If we allocate less than specified in the decorator, the script runs fine. If we allocate more than the instance memory, numpy gives an error and exits. If we specify anything in between, the script crashes.

Copy code

from metaflow import FlowSpec, batch, step
import psutil
import os
import numpy as np
import math


class MemTest(FlowSpec):
    @batch(memory=15000)
    @step
    def start(self):
        import psutil

        print(f"psutil thinks we have {psutil.virtual_memory().total / 1024**3} GB RAM")
        arrsize = int(math.sqrt(6000000 * 1024**2 / 8))
        arr = np.ones((arrsize, arrsize))
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    MemTest()

Output when allocating space for a 6TB array:

Copy code

[862/start/8589 (pid 1695)] [0992e97b-08bd-41fd-9b51-7be919048053] numpy.core._exceptions._ArrayMemoryError: Unable to allocate 5.72 TiB for an array with shape (886810, 886810) and data type float64

Output when allocating 60GB:

Copy code

[860/start/8584 (pid 1414)] OutOfMemoryError: Container killed due to memory usage This could be a transient error. Use @retry to retry.

fresh-laptop-72652

03/23/2023, 9:57 PM

psutil

uses

/proc/meminfo

which shows the system's, not the container's, capabilities see https://github.com/giampaolo/psutil/issues/1011

shy-refrigerator-15055

03/24/2023, 10:27 AM

That makes a lot of sense. It has some unfortunate practical implications when libraries like GDAL make incorrect assumptions about available memory, but it is a fixable issue. Thanks!

8 Views

Open in Slack

Previous Next