Does anyone know if the newer versions of the datastore are more efficient than the older one? We’re currently on version 0.80 of the datastore terraform module on AWS and occasionally we get a “datastore error” thrown when running very large foreach flows. My hope is that the newer version has some additional efficiencies

the TF deployment template only sets up the right s3 buckets for datastore - so there aren't any efficiencies to gain just yet in any of the recent updates

Looks like I got my terminology mixed up I’m referring to the

metadata-service

which manages the metadata database. I can’t entirely recall how that metadata-service version is set or where it’s deployed, but I seem to recall someone saying there have been additional efficiencies added to the service in recent version @dry-beach-38304 it’s possible it was you who mentioned this? I was unsuccessful in confirming my recollection in slack message history

there were some changes yes in handling of task status: https://github.com/Netflix/metaflow-service/pull/338

@dry-beach-38304 That’s helpful! Could you share some of the high level lessons? How much of a lift was it to modify the metadata service to handle the large runs? Our data scientists do most of the data engineering in our org as well, so some of these runs can get very large. We’ve had issues with this such as the status page never loading for those particular runs, likely because there’s too much data on the database being queried Also wondering how much the PR

Fix/avoid expensive ops

helps with these larger loads