https://outerbounds.com/ logo
Join Slack
Powered by
This message was deleted.
# dev-metaflow
m
This message was deleted.
a
Hi @cuddly-rocket-69327! We recently introduced a functionality - 
unbounded foreach
that allows us to offload the Metaflow for-each functionality to an external subsystem. In future, we can offer integrations to various HPO tools (Optuna, Ray Tune etc.) using this mechanism, for example. There isn't any integration with firecracker yet but it will be fun to prototype one - 
@firecracker
to guarantee resource isolation on laptops. Can you expand on what use case you have in mind?
c
unbounded foreach
is very cool! We'd like one where we could we could do a foreach on millions of items without launching a container for each item. Do you have an example Flow using 
unbounded foreach
?
Copy code
class HelloFlow(FlowSpec):
    @step
    def start(self):
       self.items = metaflow.input(path) # SparkDF / DaskDF / Ray / streaming DF 
       self.next(self.mapper, self.items)

    @mapper # may be uses firecracker
    @step
    def start(self):
       do_map(self.input)
Where Spark or Dask handle the partitions/splits (or mini-batch in streaming) that launch N parallel containers. And each item is light weight (
firecracker
?) instead of a new container for each item. Where 
path
could be a Hive table, s3 path, or stream name that the data provider (Spark, Dask, Ray) supports.
3 Views
Open in Slack
PreviousNext
Powered by