https://outerbounds.com/ logo
Join Slack
Powered by
Hello there ! I am trying to setup metaflow for on...
# ask-metaflow
b
Hello there ! I am trying to setup metaflow for on prem usage, using : • 
docker-compose -f docker-compose.development.yml up
from the metaflow-service package • 
docker run -p 3000:3000 -e METAFLOW_SERVICE=<http://localhost:8083/> metaflow-ui:latest
from the metaflow ui package and then running 
METAFLOW_SERVICE_URL=<http://localhost:8080> METAFLOW_DEFAULT_METADATA="service" python3 sample_metaflow.py run
works, but on the UI i get the message 
Since this run uses local datastore instead of cloud services, some information may be incomplete.
(indeed i cant see the stdout, stderr, artifacts... and the metaflow/_parameters/data.json says "datastore: local") and i havent found any doc on how to change it to get the view on the full data in the UI. Can i keep this simple config or do i need to switch to k8s ? (and if so, do you have a doc related ?)
a
do you have "METAFLOW_DEFAULT_DATASTORE": "s3" in your metaflow config?
logs are stored in the datastore, like artifacts. When using local datastore, MFGUI won't be able to get them
b
so the url in the markdown that is said to explain how to configure metaflow is this one : https://admin-docs.metaflow.org/overview/configuring-metaflow and it redirects to nothing. Perhaps it has been moved and the new url isnt in the doc ? edit : it is https://outerbounds.com/engineering/operations/configure-metaflow/
Hi @average-beach-28850 i have added the METAFLOW_DEFAULT_DATASTORE in the .config.json and applied it but still cant figure out where to put the key id and secret of the minio bucket and under what name, cant find any info. Can you help me with that ? edit : i ran this command and it worked : 
AWS_ACCESS_KEY_ID=XXX AWS_SECRET_ACCESS_KEY=XXX python sample_metaflow.py run
the problem is now to load the data from the minio bucket to the metaflow-ui, do you have a clue ? best Noe
a
typically you'd have those variables for minio in a k8s secret, and mount that secret as env vars into mfgui backend pod. It uses boto3 for aws auth so it should automatically pick them up
see here https://github.com/outerbounds/metaflow-with-airflow-minio steps 8 and 9 ( you don't have to use ngrok, since mfgui runs in the same cluster as minio it could connect directly)
it just needs those credentials + s3 endpoint url set to point to minio
b
Hi @average-beach-28850 Now i have the same problem as this issue : https://github.com/outerbounds/metaflow-tools/issues/48 i have the credentials saved :
Copy code
root@root:~# kubectl get secret
NAME                             TYPE                 DATA   AGE
minio-secret                     Opaque               2      37m
and the metaflow-ui pod keeps crashing because it doesnt find the password (i port-forwarded all services just to make sure but it still doesnt work)
12 Views
Open in Slack
PreviousNext
Powered by