Are there any best practise repos on how to use Optuna in conjunctions with MetaFlow (especially AWS batch). I am thinking to use simply sqllite to persist in S3 as file as there will be only one job per study (to be parallelised via AWS batch …). Thanks!

yep! that can work!

@ancient-application-36103 do you know if anyone (not saying Outerbounds.. but not NOT saying Outerbounds 🙂) is thinking about using, say, the new

mpi

capabilities to make this and other sorts of functionality even more straightforward in the future?

We already do have support for running mpi workloads through the @mpi decorator. Curious - what new capabilities are you looking into?

Yep! that's what I was referring to. I was curious if that decorator could help with these sorts of distributed hyperparameter tuning libraries since they need to respond to feedback based on the output of any given run. If so, maybe it could be packaged as some sort of extension?

We are thinking of implementing a proper HPO decorator supporting optuna etc. The underlying constructs already exist.

That would be very cool. Thanks for the info!