Is there a limit on what classes of objects can be logged in metaflow? It looks like there’s an issue when I try to use the default card:

Card of type default is unable to be rendered with arguments None.
Stack trace :  Traceback (most recent call last):
  File ".../.venv/lib/python3.10/site-packages/metaflow/plugins/cards/card_cli.py", line 503, in create
    rendered_info = render_card(mf_card, task, timeout_value=timeout)
  File ".../.venv/lib/python3.10/site-packages/metaflow/plugins/cards/card_cli.py", line 384, in render_card
    rendered_info = mf_card.render(task)
  File ".../.venv/lib/python3.10/site-packages/metaflow/plugins/cards/card_modules/basic.py", line 605, in render
    json.dumps(final_component_dict).encode("utf-8")
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Timestamp is not JSON serializable

The error appears by default in the UI because Metaflow can run on schedulers which are outside a user's laptop (where they generally wont observe the std out). If you wanted the error on stdout you can set `@card(save_errors=False, type='default')`; Now for the error you are facing. We had added support for the datetime64[ns] datatypes in pandas dataframe but not all types such as Timestamps. We can add support for this too very soon. But to unblock you , if you convert to your Timestamp column to the datetime64[ns] and Metaflow will automatically parse that.

I think the condition in Metaflow is very tightly constrained. I was using

<http://pd.to|pd.to>_datetime()

with utc=True, resulting in a

datetime64[ns, UTC]

object. 🙂

yes. When we built this we didn't dive very deep into supporting a lot more types. Its been over a year now since launching this feature and we have a lot more insight on problems users are facing. So it seems like a good time for a bit of spring cleaning to fix some of these issues ! 😄

@hallowed-glass-14538 Oops, I didn’t realize it was you who wrote this issue: https://github.com/Netflix/metaflow/issues/1319

Just saw your comment on the using

to_json

; Which seems reasonable but I think we made a decision to not do it since to_json doesn't preserve the index and the dataframe appearing on the UI can have completely different view from what the user intended. The main goal was to keep the "appearance" (layout) of the dataframe very similar to what the user would see in their notebook. I think thats why we did all the complex logic in that

_parse_pandas_dataframe

function. There maybe an avenue where to_json works with some additional tweeks but didn't get the chance to explore that. To solve this problem we can test if

df.astype('object')

can do a native conversion. If it can then it would make it much easier to parse the dataframe.

Awesome, I thought it was coincidentally just another user, so I figured I could try my hand at it. That said, I didn’t start anything, so I’m happier to take the back seat 🙂

This one has been haunting us for a bit now, is there a PR to follow or any fix coming here?

2.8.3 was the first version. Latest will have it too