Hello again all, I'm fine-tuning the new open sou...
# ask-metaflow
a
Hello again all, I'm fine-tuning the new open source LLM called Mistral-7B using Metaflow and for some reason the logs are not being propagated to both the terminal in which I'm running
python mistral_fine_tune.py --environment=conda run
AND the Metaflow UI. The logs are being propagated to CloudWatch though and I can see the progress bar from there. See attached image of the Metaflow UI. The code for running the fine-tuning job can be reproduced here: https://github.com/rileyhun/llm_finetuning_metaflow/tree/main/fine-tuning-mistral
1
c
Hey Riley, have an idea about how to run subprocess that might help this, but want to try it first before sending you on a goose chase. will hopefully have enough time before our chat today.
🙏 1
Issue is something the PyTorch Lightning ProgressBar is doing with TQDM logs that has weird interaction with Metaflow i don't understand yet. Making this line an empty list and I start to see action in the terminal. The thing I wanted to try was the
| tr '\r' '\n'
trick and didn't have desired effect. Stopping for now, can show you that at least something happens in our chat today
thankyou 1