Hi <@U03GM752TU3> Sir, <@U07K6BVS6HX> Sir, <@WM13W...
# dev-metaflow
w
Hi @rhythmic-controller-77489 Sir, @hundreds-rainbow-67050 Sir, @dry-beach-38304 Sir, @bulky-afternoon-92433 Sir, @freezing-telephone-6958 Sir, @square-wire-39606 Sir & @little-apartment-49355 Sir, My name is Ganesh Patil, and I'm a 2nd computer science student at Pimpri Chinchwad University Pune India. I'm writing to express my interest in the Metaflow VS Code Extension project for GSoC 2026 and to ask a few questions before I finalize my proposal. A bit about me: I've been actively contributing to open source for the past 6 Months. I have 60+ pull requests merged across organizations including the Apache Software Foundation and the University of Alaska's ControlCore (ConCore) project. My Apache contributions involved bug fixes, documentation, feature work on a specific project like Apache Airflow/Shiro/Camel/etc., and my work on ConCore focused on the control systems framework, Python tooling, etc. These experiences taught me how to work with large codebases, follow contribution guidelines, communicate async with maintainers, and ship code through review cycles. Merged PR links: https://docs.google.com/document/d/1VHHT7otLvMU2pJt_32_ibZprbopVPU5I-WyBLSGw6zE/edit?tab=t.0 I've cloned the Metaflow repo, run the tutorials locally, and started reading through graph.py, the Client API, and the existing metaflow-dev-vscode extension. I have a working understanding of how FlowGraph builds the DAG from AST parsing and how the Flow → Run → Step → Task → DataArtifact hierarchy maps to the local datastore. I had a few questions that would help me write a stronger proposal: 1. On the DAG viewer - import vs. AST-only parsing: FlowGraph._create_nodes() currently requires importing the FlowSpec class (it uses inspect.getsource). This could fail if the user's flow has heavy dependencies not installed locally (e.g., PyTorch, cloud SDKs). Would you prefer a fallback that does lightweight AST-only parsing (just extracting self.next() calls and @step decorators without importing), or is it acceptable to require that the flow file be importable? 2. . On the artifact browser - metadata provider scope: Should the artifact browser only support the local filesystem datastore (.metaflow directory), or do you also want it to work with the remote metadata service (S3/Azure/GCS backends) from day one? Supporting local-only would simplify the initial implementation, and the Client API abstracts the backend anyway - but I want to know what you'd prioritize for the midterm deliverable. 3. On one-click debugging - step isolation: When debugging a single step (not the full flow), Metaflow expects --run-id and --task-id arguments. My current plan is to let the user pick from recent runs via a QuickPick dropdown. Is there a simpler way to create a "synthetic" debug run for a step in isolation, or is replaying from an existing run the intended approach? 4. On the run configuration UI - cloud backends: For the backend selector (local, @batch, @kubernetes), should the extension actually attempt to run flows on cloud backends, or should it just construct the CLI command and let the user handle credentials/config? I'm leaning toward the latter to avoid scope creep, but wanted your input. 5.On the existing extension: The current metaflow-dev-vscode extension - is the plan to build on top of it or start fresh? From what I've seen, it's fairly minimal (two keybindings), so starting from a clean scaffold might be cleaner. But if there's architectural work or user settings you'd want preserved, I'd like to know. 5. General scope question: Given the 350-hour budget, which features would you consider "must-have" for a successful project vs. "nice-to-have"? My current priority order is: DAG viewer → Artifact browser → One-click debugging → Run config UI → Card preview (stretch). Does that match your expectations? I'm also happy to do a small preliminary contribution (a bug fix or improvement to the existing extension or Metaflow repo) if that would be helpful for my application. Thank you for your time. I'm looking forward to your thoughts. Best regards, Ganesh Patil Pimpri Chinchwad University, Pune India GitHub: https://github.com/GaneshPatil7517 Email: ganesh.patil24@pcu.edu.in
d
Hello. Be sure to check out here: https://github.com/outerbounds/metaflow-dev-vscode/issues 1. Metaflow is typically launched using
myflow.py <cmd>
which requires the entire flow to be loaded (typically) because the
__main__
function at the end is what triggers the whole processing so having the flow file be importable is usually needed although I will say that it has caused issues in the past. I think though, that users typically try to run things locally too so it should be ok. 2. Ideally in the fullness of time, all artifacts. Using the Client API is probably a good way to not have to deal with it. 3. did you check out spin as well. Some ideas there. 4. That’s a good question. Ideally it depends on what he user is trying to do: a. local debugging so the “remote” things should be ignored b. actually trying to run it — it would be nice to be able to run it directly from VSCode. c. if you are really motivated, a HUGE bonus would be hooking it up to VSCode’s debugging integration and actually running the debugger on the remote node. THAT is something that has been asked for a while. 5. Pretty open. The existing extension was just a small thing done with spin. You can start fresh. Your plan is good I think. The debugging aspect can include a lot of stuff though so…
w
Hey @dry-beach-38304 Sir, thank you so much for taking the time to answer all my questions in such detail - this is incredibly helpful..... A few quick takeaways from your replies: 1. Importability - Got it, I'll make sure the extension validates that the flow is importable before doing any graph extraction or parameter parsing. That makes total sense since Metaflow itself needs to execute the module to resolve decorators and parameters. 2. Client API for artifacts - Great to know it abstracts the backend. I'll use the Client API (Flow → Run → Step → Task → DataArtifact) for the artifact browser so it works seamlessly whether artifacts are stored locally or on S3/Azure/GCS. 3. spin command - This was a huge insight. I dug into the codebase (run_cmds.py, step_cmd.py) and spin is exactly what I need — it handles artifact loading from a prior run, supports --skip-decorators, and even allows artifact overrides via --artifacts-module. Much cleaner than the raw step approach I was originally considering. I also noticed the existing metaflow-dev-vscode extension already uses spin, so I'll build on that pattern. 4. Remote debugging - Really excited that attaching a debugger to a remote node is something the team has wanted for a while. I've made it a high-priority stretch goal in my proposal — the idea is to inject
debugpy
into a step via a decorator or environment variable, then have VS Code attach via `debugpy`'s remote attach protocol. Even a basic version of this could be really valuable. 5. Starting fresh - I'll use the existing extension as a reference but build from scratch with a proper architecture (TreeDataProvider for the sidebar, D3.js for the DAG, CodeLens for inline actions, etc.). I've updating my proposal to incorporate all of this feedback. Would it be okay if I shared the updated draft with you for any further thoughts? Thanks again for your time and guidance!