hi guys - starting to explore metaflow for a commercial engagement - wanted to get a quick sanity check on the following hypothesis: • The anyscale team just published a benchmark comparing ray with spark and sagemaker batch predictions on a resnet benchmark. • Do you think it would be feasibly to run recsys workflows using metaflow + ray and expect to see similar improvements in scalability for batch inference jobs?

batch inference cases like this are bread and butter of Metaflow - it'd be really fun to compare a native Metaflow solution to this benchmark. Highly likely it'll beat Spark and Sagemaker in this setup. I'm not sure how much extra Ray does over what you could do with Metaflow easily in this scenario. If you want to call Ray from Metaflow, you can do that using their client API.

@victorious-lawyer-58417 thanks - a bit confused on the similarities between metaflow/ray. I came in thinking metaflow gives me orchestration, while ray/spark/batch transform give me distributed compute. I was thinking of metaflow as a substitute for airflow. Sounds like I may have misinterpreted the value proposition?

in contrast to Airflow, Metaflow handles compute too. In embarrassingly parallel use cases like the one described in the benchmark, Metaflow's foreach construct gives you access to any number of cloud (GPU) instances, similar to Spark and Ray.

thanks and does it work equally seamlessly on GCP?

yep, using GKE

right so there is an equivalent to

metaflow.S3

for GCS then yeah?

no, but you can use their default Python client for similar purpose. We haven't done as thorough performance benchmarking with GCS as we have done with S3, so I can't say if their client would be much slower (I hope not)

@important-answer-56430 a deep learning sota implementation with metaflow + aws can be found here (including data processing etc) https://github.com/jacopotagliabue/recs-at-resonable-scale

Thanks @worried-mechanic-36312 that repo has been our starting point and inspiration already!