Hi! I'm trying to scale my flow up to 256 tasks in...
# ask-metaflowi
important-address-85681
10/27/2024, 11:23 AMHi! I'm trying to scale my flow up to 256 tasks in parallel on GKE (with 128 workers everything works fine). I'm facing issue that my connection breaks (don't know if it's networking or ssh server issue) and processes getting killed. Had someone similar problems? If so how did you overcome them?
✅ 1
important-address-85681
10/27/2024, 3:16 PMOk, it seems the problem was in OOM. Btw why do we have such peak in RAM in flow init?
a
ancient-application-36103
10/27/2024, 6:06 PMWhere was the OOM happening?
i
important-address-85681
10/27/2024, 6:33 PMOn the host during tasks creation, at peak it was 25 Gb for 256 workers
a
ancient-application-36103
10/27/2024, 6:34 PMWhat decorators are you using for your tasks?
i
important-address-85681
10/27/2024, 6:35 PM Copy code
@kubernetes(
memory=4096,
cpu=8,
image="xxxx",
namespace="default",
service_account="my-ksa",
node_selector={
"<http://cloud.google.com/compute-class|cloud.google.com/compute-class>": "Performance",
"<http://cloud.google.com/machine-family|cloud.google.com/machine-family>": "n2d",
},
)
@retry
@stepa
ancient-application-36103
10/27/2024, 6:37 PMAre there any flow level decorators? The only thing that’s worth of note here is that the code package is downloaded when the task in being initialized - depending on what you have in your code package, there might be a memory spike but not anything that will consume 4G
ancient-application-36103
10/27/2024, 6:38 PMDo you see the oom before the - task is starting - logline?
i
important-address-85681
10/27/2024, 6:42 PMNo, I don't have flow decorators. Btw in my project I have following structure:
• root
◦ flows
▪︎ A.py
▪︎ B.py
◦ other libs
And I run
python flows/A.py run ....a
ancient-application-36103
10/27/2024, 6:43 PMYour current working directory is packaged. You can check what is packaged using the ‘package list’ command
i
important-address-85681
10/27/2024, 6:45 PMBut it packs only *.py files?
important-address-85681
10/27/2024, 6:45 PMHow to use 'package list'?
important-address-85681
10/27/2024, 6:52 PMI see, it packed all my .venv🤦