Hi Everyone!! I'm currently working on a migratio...
# ask-metaflow
r
Hi Everyone!! I'm currently working on a migration project and therefore created a utility that converts the list of python functions into metaflow steps that fills up placeholders in a Metaflow template. After running the utility its creating a very big flow containing approximately 950 steps. Some steps will be executing in parallel while others will run in a sequential manner. I've redacted the step names and trimmed it to contain around 300 nodes. Also each step just points to the next step. Please try to execute this flow in your machine. In my project, after running multiple experiments, I've found out that , Metaflow starts lagging as soon as the number of steps increases beyond 125 steps and refuses to start beyond 300 steps. I'd appreciate if you could provide a way to execute these flows. I tried executing the flow in a decent machine with 16 cores/64GB RAM, and during the startup its not utilizing more than 20% of its resources
👀 1
✅ 1
🤯 1
s
@rough-microphone-11807 - looking at the structure of your flow, is it possible to use
foreaches
instead of
branches
?
r
@square-wire-39606 I thought about that, but it's not possible since some steps at times are very different that I've trimmed. from this example. I'm wondering if it's because of the fact I'm using branches. I tried executing with --max-workers or --max-num-splits to ensure if it's not the resource limitation, but it just hangs like this:
s
I am happy to find time to chat about this use case - it is likely that the issue is with parsing such a big graph. Metaflow supports 100s of steps and 10k-100k of tasks within a flow today - let's find you a workaround.
also, this was the issue I ran into while executing your flow -
Copy code
Step vvvvvvvvvv_join seems like a join step (it takes an extra input argument) but an incorrect number of steps (aaaaaaaaaaa, aaaaaaaaaaaa, aaaaaaaaaaaaa, aaaaaaaaaaaaaa, ab, ac, ad, ae, af, ag, ah, ai, aj, ak, al, am, an, ao, ap, aq, ar, ass, at, au, av, aw, ax, ay, az, ba, bb, bbbbbbbbbbb, bbbbbbbbbbbb, bbbbbbbbbbbbb, bbbbbbbbbbbbbb, bc, bd, be, bf, bg, bh, bi, bj, bk, bl, bm, bn, bo, bp, bq, br, bs, bt, bu, bv, bw, bx, by, bz, ca, cb, cc, ccccccccccc, cccccccccccc, ccccccccccccc, cccccccccccccc, cd, ce, cf, cg, ch, ci, cj, ck, cl, cm, cn, co, cp, cq, cr, cs, ct, cu, cv, cw, cx, cy, cz, da, db, dc, dd, ddddddddddd, dddddddddddd, ddddddddddddd, dddddddddddddd, de, df, dg, dh, di, dj, dk, dl, dm, dn, do, dp, dq, dr, ds, dt, du, dv, dw, dx, dy, dz, ea, eb, ec, ed, ee, eeeeeeeeeee, eeeeeeeeeeee, eeeeeeeeeeeee, ef, eg, eh, ei, ej, ek, fffffffffff, ffffffffffff, fffffffffffff, ggggggggggg, gggggggggggg, ggggggggggggg, hhhhhhhhhhh, hhhhhhhhhhhh, hhhhhhhhhhhhh, iiiiiiiiiii, iiiiiiiiiiii, iiiiiiiiiiiii, jjjjjjjjjjj, jjjjjjjjjjjj, jjjjjjjjjjjjj, kkkkkkkkkkk, kkkkkkkkkkkk, kkkkkkkkkkkkk, lllllllllll, llllllllllll, lllllllllllll, ma, mb, mc, me, mf, mg, mk, ml, mm, mmmmmmmmmmm, mmmmmmmmmmmm, mmmmmmmmmmmmm, mn, mo, mp, mq, mr, ms, mt, mu, mv, mw, mx, my, mz, na, nb, ne, nf, ng, nh, ni, nj, nk, nl, nnnnnnnnnnn, nnnnnnnnnnnnn, no, np, nq, nr, ns, nt, nu, nv, nw, nx, ny, nz, oa, ob, oc, od, oe, of, og, oh, oi, oj, ok, ooooooooooo, oooooooooooo, ooooooooooooo, ppppppppppp, pppppppppppp, ppppppppppppp, qqqqqqqqqqq, qqqqqqqqqqqq, qqqqqqqqqqqqq, rrrrrrrrrrr, rrrrrrrrrrrr, rrrrrrrrrrrrr, sssssssssss, ssssssssssss, sssssssssssss, train_model_xgboost_mdbcd_mobility_final_hand_grip, train_model_xgboost_mdbcd_mobility_lower_extremidsfasdf_qmt, train_model_xgboost_mdbcd_mobility_no_step, train_model_xgboost_mdbcd_mobility_upper_extremidsfasdf_qmt, train_model_xgboost_mdbcd_swallowing_final_10_meter_walk_run, train_model_xgboost_mdbcd_swallowing_final_fvc_sitting, train_model_xgboost_mdbcd_swallowing_final_fvc_supine, train_model_xgboost_mdbcd_swallowing_final_hand_grip, train_model_xgboost_mdbcd_swallowing_lower_extremidsfasdf_qmt, train_model_xgboost_mdbcd_swallowing_no_step, train_model_xgboost_mdbcd_swallowing_upper_extremidsfasdf_qmt, train_model_xgboost_mdbcd_upper_extremidsfasdf_final_10_meter_walk_run, train_model_xgboost_mdbcd_upper_extremidsfasdf_final_fvc_sitting, train_model_xgboost_mdbcd_upper_extremidsfasdf_final_fvc_supine, train_model_xgboost_mdbcd_upper_extremidsfasdf_final_hand_grip, train_model_xgboost_mdbcd_upper_extremidsfasdf_lower_extremidsfasdf_qmt, train_model_xgboost_mdbcd_upper_extremidsfasdf_no_step, train_model_xgboost_mdbcd_upper_extremidsfasdf_upper_extremidsfasdf_qmt, train_model_xgboost_total_for_dm1_activ_final_hand_grip, train_model_xgboost_wpai_total_score_final_10_meter_walk_run, train_model_xgboost_wpai_total_score_final_fvc_sitting, train_model_xgboost_wpai_total_score_final_fvc_supine, train_model_xgboost_wpai_total_score_final_hand_grip, train_model_xgboost_wpai_total_score_lower_extremidsfasdf_qmt, train_model_xgboost_wpai_total_score_no_step, train_model_xgboost_wpai_total_score_upper_extremidsfasdf_qmt, ttttttttttt, tttttttttttt, ttttttttttttt, uuuuuuuuuuu, uuuuuuuuuuuu, uuuuuuuuuuuuu, vvvvvvvvvv, vvvvvvvvvvv, vvvvvvvvvvvv, vvvvvvvvvvvvv, wwwwwwwwwww, wwwwwwwwwwww, wwwwwwwwwwwww, xxxxxxxxxx, xxxxxxxxxxx, xxxxxxxxxxxx, xxxxxxxxxxxxx, yyyyyyyyyy, yyyyyyyyyyy, yyyyyyyyyyyy, yyyyyyyyyyyyy, zzzzzzzzzz, zzzzzzzzzzz, zzzzzzzzzzzz, zzzzzzzzzzzzz) lead to it. This join was expecting 56 incoming paths, starting from split step(s) bb, cc, dd, ee, ff, gg, hh, ii, jj, kk, ll, mm, nn, oo, pp, qq, rr, ss, tt, uu, vv, ww, xx, yy, zz, aaa, bbb, ccc, ddd, eee, fff, ggg, hhh, iii, jjj, kkk, lll, mmm, nnn, ooo, ppp, qqq, rrr, sss, ttt, www, xxx, yyy, zzz, aaaa, bbbb, cccc, dddd, eeee, ffff, gggg.
r
Apologies, I might have messed up while redacting the method/step names, but changing the steps from branch to foreach in my metaflow template solved the issue. Metaflow executed right off the bat. 🙂. Really appreciate your suggestion and the team is really happy with its performance as compared to their previous pipeline.
But it seems that the branch modes definitely has some performance issues, Let me create a new file for the team to test/debug this scenario:)
a
sounds good!