Hi everyone! I am running Metaflow on one machine. There is always an initial delay or idle time for 20-30 mins. There is no difference it if metadata is stored locally or via metadata-service. Artifacts are always stored in the cloud. Is it possible to reduce this delay somehow? Thanks

That shouldn’t be the case. Can you help me with your console logs?

sry for the late reply. Here are the logs

Metaflow 2.9.11 executing DevDataPipeline for user:ubuntu
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2024-01-30 16:07:09.311 Workflow starting (run-id 5243):
2024-01-30 16:07:09.354 [5243/start/144396 (pid 1548)] Task is starting.
2024-01-30 16:07:11.062 [5243/start/144396 (pid 1548)] Task finished successfully.
2024-01-30 16:07:11.163 [5243/volue_funcs_parameters/144397 (pid 1585)] Task is starting.
2024-01-30 16:07:12.752 [5243/volue_funcs_parameters/144397 (pid 1585)] Foreach yields 8 child steps.
2024-01-30 16:07:12.753 [5243/volue_funcs_parameters/144397 (pid 1585)] Task finished successfully.
2024-01-30 16:07:12.883 [5243/calculate_volue_funcs/144398 (pid 1623)] Task is starting.
2024-01-30 16:07:12.931 [5243/calculate_volue_funcs/144399 (pid 1633)] Task is starting.
2024-01-30 16:07:12.989 [5243/calculate_volue_funcs/144400 (pid 1636)] Task is starting.
2024-01-30 16:07:13.035 [5243/calculate_volue_funcs/144401 (pid 1653)] Task is starting.
2024-01-30 16:07:13.098 [5243/calculate_volue_funcs/144402 (pid 1657)] Task is starting.
2024-01-30 16:07:13.150 [5243/calculate_volue_funcs/144403 (pid 1674)] Task is starting.
2024-01-30 16:07:13.226 [5243/calculate_volue_funcs/144404 (pid 1684)] Task is starting.
2024-01-30 16:07:13.326 [5243/calculate_volue_funcs/144405 (pid 1696)] Task is starting.
2024-01-30 16:07:20.590 [5243/calculate_volue_funcs/144398 (pid 1623)] Task finished successfully.
2024-01-30 16:07:20.865 [5243/calculate_volue_funcs/144405 (pid 1696)] Task finished successfully.
2024-01-30 16:07:23.116 [5243/calculate_volue_funcs/144399 (pid 1633)] Task finished successfully.
2024-01-30 16:07:38.457 [5243/calculate_volue_funcs/144400 (pid 1636)] Task finished successfully.
2024-01-30 16:08:06.341 [5243/calculate_volue_funcs/144403 (pid 1674)] Task finished successfully.
2024-01-30 16:08:06.622 [5243/calculate_volue_funcs/144402 (pid 1657)] Task finished successfully.
2024-01-30 16:08:07.459 [5243/calculate_volue_funcs/144404 (pid 1684)] Task finished successfully.
2024-01-30 16:08:10.017 [5243/calculate_volue_funcs/144401 (pid 1653)] Task finished successfully.
2024-01-30 16:08:10.150 [5243/join_volue_funcs_foreach/144406 (pid 2001)] Task is starting.
2024-01-30 16:08:14.985 [5243/join_volue_funcs_foreach/144406 (pid 2001)] Task finished successfully.
2024-01-30 16:08:15.105 [5243/get_contract_ids/144407 (pid 2138)] Task is starting.
2024-01-30 16:08:16.757 [5243/get_contract_ids/144407 (pid 2138)] Task finished successfully.
2024-01-30 16:08:16.874 [5243/define_paramaters/144408 (pid 2176)] Task is starting.
2024-01-30 16:08:19.076 [5243/define_paramaters/144408 (pid 2176)] Foreach yields 2 child steps.
2024-01-30 16:08:19.076 [5243/define_paramaters/144408 (pid 2176)] Task finished successfully.
2024-01-30 16:08:19.205 [5243/get_trades/144409 (pid 2217)] Task is starting.
2024-01-30 16:08:19.253 [5243/get_trades/144410 (pid 2227)] Task is starting.
2024-01-30 16:08:21.439 [5243/get_trades/144409 (pid 2217)] Task finished successfully.
2024-01-30 16:08:21.545 [5243/get_contract_history/144411 (pid 2284)] Task is starting.                                                                                                                                                            2
024-01-30 16:08:22.287 [5243/get_trades/144410 (pid 2227)] Task finished successfully.
2024-01-30 16:08:22.389 [5243/get_contract_history/144412 (pid 2318)] Task is starting.                                                                                                                                                      2024-01
-30 16:08:24.900 [5243/get_contract_history/144412 (pid 2318)] Task finished successfully.
2024-01-30 16:08:25.004 [5243/get_orderbooks/144413 (pid 2375)] Task is starting.
2024-01-30 16:08:25.161 [5243/get_contract_history/144411 (pid 2284)] Task finished successfully.
2024-01-30 16:08:25.266 [5243/get_orderbooks/144414 (pid 2392)] Task is starting.
2024-01-30 16:08:27.193 [5243/get_orderbooks/144413 (pid 2375)] Task finished successfully.
2024-01-30 16:08:27.310 [5243/calculate_labels/144415 (pid 2457)] Task is starting.
2024-01-30 16:08:27.471 [5243/get_orderbooks/144414 (pid 2392)] Task finished successfully.
2024-01-30 16:08:27.577 [5243/calculate_labels/144416 (pid 2474)] Task is starting.
2024-01-30 16:08:29.694 [5243/calculate_labels/144415 (pid 2457)] Task finished successfully.
2024-01-30 16:08:29.907 [5243/calculate_labels/144416 (pid 2474)] Task finished successfully.
2024-01-30 16:08:30.067 [5243/join_foreach/144417 (pid 2557)] Task is starting.
2024-01-30 16:08:32.405 [5243/join_foreach/144417 (pid 2557)] Task finished successfully.
2024-01-30 16:08:32.511 [5243/add_delivery_start_end_to_trades_full/144418 (pid 2602)] Task is starting.
2024-01-30 16:08:34.680 [5243/add_delivery_start_end_to_trades_full/144418 (pid 2602)] Task finished successfully.
2024-01-30 16:08:34.807 [5243/calculate_candle_stick/144419 (pid 2646)] Task is starting.
2024-01-30 16:08:42.357 [5243/calculate_candle_stick/144419 (pid 2646)] Task finished successfully.
2024-01-30 16:08:42.467 [5243/drop_na_candle_stick/144420 (pid 2687)] Task is starting.
2024-01-30 16:08:44.611 [5243/drop_na_candle_stick/144420 (pid 2687)] Task finished successfully.
2024-01-30 16:08:44.710 [5243/calculate_features/144421 (pid 2728)] Task is starting.
2024-01-30 16:08:47.683 [5243/calculate_features/144421 (pid 2728)] Task finished successfully.
2024-01-30 16:08:47.808 [5243/calculate_inference/144422 (pid 2781)] Task is starting.
2024-01-30 16:08:50.770 [5243/calculate_inference/144422 (pid 2781)] Task finished successfully.
2024-01-30 16:08:50.876 [5243/trade_signal/144423 (pid 2817)] Task is starting.
024-01-30 16:08:53.194 [5243/trade_signal/144423 (pid 2817)] Task finished successfully.
2024-01-30 16:08:53.302 [5243/end/144424 (pid 2864)] Task is starting.
2024-01-30 16:08:55.689 [5243/end/144424 (pid 2864)] Task finished successfully.
2024-01-30 16:08:55.765 Done!

if you execute the workload once with

python flow.py --metadata=local run

and once with

python flow.py --datastore=local run

- what are the latencies that you observe in terms of the first task starting up?

I've tested both

python flow.py --metadata=local run

and

python flow.py --datastore=local run

and there is no difference - it takes about 10 mins to start the first task

do you have any top level imports in your flow file that take a lot of time to execute? does executing this flow take similar amount of time or less?

i don't have any imports that take a lot time to execute. It seems that it also takes similar time to execute the flow from the documentation

can you create an empty directory, copy the flow inside that folder and execute the flow from within that folder?

i have run the flow in an empty folder. The flow has no delay

Ah, it could be that the directory that you are working from has a lot of files which is slowing metaflow down

thanks for your help 🙂