Is there any way to force Metaflow to store artifacts with higher Pickle protocols? At a glance, it looks like Metaflow attempts to pickle everything with

protocol=2

and then attempts

protocol=4

if that fails. While the method linked has an optional

force_v4

kwarg, I think none of the calls to the method pass a value, so that the default of

False

gets used. It seems like a waste of compute to be attempting to pickle large artifacts at

protocol=2

and then failing and retrying with

protocol=4

given that we have no use for the backwards compatibility (protocol 4 has been supported since Python 3.4, which was released more than 10 years ago).

let me check - @dry-beach-38304 it should be safe to move to 4 by default?

safe from our point of view — we just have to make sure we are backward compatible and all that but yes, we should move.

Thanks for the replies. Glad to hear you're on board with this change! Would it be helpful for someone to put together a PR?

that will be wonderful

Okay, great. I will start a thread in #C020U025QJK to clarify some implementation details before I begin.