Hi team I am trying to use metaflow to train T5 with tensorflow and get a weird underlying bug

tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot convert a Tensor of dtype variant to a NumPy array.

I have tested the training codes work without metaflow. This happens in the

train

step, but after all the operation in

train

step done. So I guess this error may be related to metaflow. Does anyone have some ideas about this? Thanks! The training codes with metaflow are here, the notebook for the training without metaflow is here.

it's related to serialization of artifacts. Based on a quick look at your code, you create two new artifacts in the

train

step

self.tf_train_ds = to_tf_dataset(train_ds)
self.tf_valid_ds = to_tf_dataset(valid_ds)

Try dropping the

self.

here to see if it helps

cc: @acoustic-van-30942

there are ways to make serialization work with those objects too but it's good to confirm that you get it running first

Hi ville could you help explain the "serialization of artifacts" you mentioned? I am interested in this

Metaflow keeps track and persists everything you assign to

self.

automatically when a task finishes which is why the error happens at the end of the task, as you observed. It does this so you can run tasks in different environments seamlessly (e.g.

train

in the cloud and the next step locally etc), and also to help keeping track of everything. It uses Python default serialization

pickle

to convert objects to bytes, which works in most cases but some libraries are not very pickle-friendly, Tensorflow being one of them. Things related to Tensorflow tend to require a bit special treatment (e.g. see this example that serializes Keras models). PyTorch tends to be more pickle-friendly. Similar to that Keras example, if you wanted to persist these objects as artifacts, you'd need to wrap them in a few extra lines that use Tensorflow's internal serialization methods. I'm happy to help you with that if needed.