Hello everyone So, I’ve been spending some time reading through the Metaflow client code to understand how logs are retrieved via the Client API. From tracing

Task.stdout

→

_load_log()

→

loglines()

→

FileCache.get_logs_stream()

, it looks like the current behavior fully materializes the log content into memory when accessd through the client basically joining all lines into a single string. I’m trying to understand whether there has been prior discussion around bounded log access (for example, retrieving only the last n lines or partial log reads), particularly for notebook or monitoring-style workflows. Is this an area where scoped improvements would be useful to explore, or is there existing design context I should be aware of before going deeper? Would love to dig fuurther, just wanted to sanity check my understanding first.

Hi I dont know if its been discussed before but I do think there's areas for improvement. Quick question though: is this related to gsoc? If yes, we have a couple proposals under which this idea can fold under. If not, can you please describe the painpoint youre having with the logs and we can decide what the best solution is

Thanks for the clarification. Yes, this exploration is mainly part of preparing for a gsoc proposal. I’ve been setting up the dev environment locally and tracing how logs and metadata are fetched through the client and metadata service to understand the architecture better. I’ll move further discussion to the gsoc channel once I have a clearer proposal direction.