kudos @calm-motorcycle-38345 for a very thoughtful analysis - https://github.com/Netflix/metaflow/issues/592

I think base serialise is the simplest way forward —- no new dependencies. The

qs

package is also an option; it’s supposedly much faster, and there’s an existing PR for it we can finish off. In either case I think we can preserve backwards compatibility with existing objects in S3. I probably won't start on this for a while but let me know what you think.

With the

qs

package we will have to ensure that it is present in the image used for executing on top of AWS Batch. We have purposefully tried to limit the number of dependencies Metaflow has - unless there is a significant value prop that

qs

is providing, I would recommend relying on the base serde methods in R.