https://outerbounds.com/ logo
Join Slack
Powered by
Hey, you'll this is Sid. Happy Friday. I am evalua...
# ask-metaflow
m
Hey, you'll this is Sid. Happy Friday. I am evaluating Metaflow on rancher k8's. I have no issues running a flow, however I am not able to see the logs that are produced as a result of the flow. My default_metadata_store is local and I am not inclined to use S3 in any way. Is there any possibility to see the logs associated with a run. I am attaching the screenshot. Thanks!
✅ 1
a
Hey 👋 - the logs need to be stored at a location that is accessible by both kubernetes and the UI - that location is currently a blob store - it can be minio too.
m
Hey, thanks for your reply. I will check it out and give it a try.
👍 1
@square-wire-39606 I did give a try using minio. I created a bucket and made sure the bucket is accessible from both cluster and the metaflow-service. I am getting the following error
Copy code
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint is happy!
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 01:58:36.139 Workflow starting (run-id 1683010713899068):
2023-05-02 01:58:36.621 [1683010713899068/start/1 (pid 65510)] Task is starting.
2023-05-02 01:58:39.030 [1683010713899068/start/1 (pid 65510)] INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 01:58:39.167 [1683010713899068/start/1 (pid 65510)] INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 01:58:40.307 [1683010713899068/start/1 (pid 65510)] [pod t-cbx9p-lwvnv] Task is starting (Pod is running, Container is running)...
2023-05-02 02:00:18.145 [1683010713899068/start/1 (pid 65510)] INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
2023-05-02 02:00:19.522 [1683010713899068/start/1 (pid 65510)] Data store error:
2023-05-02 02:00:19.523 [1683010713899068/start/1 (pid 65510)] No completed attempts of the task was found for task 'HelloFlow/1683010713899068/start/1'
2023-05-02 02:00:19.738 [1683010713899068/start/1 (pid 65510)]
2023-05-02 02:00:20.251 [1683010713899068/start/1 (pid 65510)] Task failed.
I do see the logs and Metaflow directory been created in the bucket.
running the tutorail workflow using this
Copy code
METAFLOW_S3_ENDPOINT_URL=<http://10.35.34.136:30937/> METAFLOW_DATASTORE_SYSROOT_S3=<s3://metaflow/> python3 helloworld.py --datastore=s3 run --with kubernetes
s
@little-apartment-49355 can you help here?
l
looks like you are running with 
local
metadata. Is there any chance you can deploy the metadata service ? 
minio
doesn't provide as strong consistency guarantees as S3 so using a local metadata won't be a viable option. You can install the metaflow-service from this helm chart. When deploying the helm chart ensure that the 
metaflow-ui
also gets the same metaflow configuration variables / minio credentials as your flow :
Copy code
helm upgrade --install metaflow <path/to/mf/helm/chart> \
	--timeout 15m0s \
    --set metaflow-ui.METAFLOW_DATASTORE_SYSROOT_S3=<s3://metaflow> \
    --set metaflow-ui.METAFLOW_S3_ENDPOINT_URL="<http://10.35.34.136:30937/>" \
    --set "metaflow-ui.env[0].name=AWS_ACCESS_KEY_ID" \
    --set "metaflow-ui.env[0].value=<MINIO_ACCESS_KEY_OR_USERNAME>" \
    --set "metaflow-ui.env[0].name=AWS_SECRET_ACCESS_KEY" \
    --set "metaflow-ui.env[0].value=<MINIO_SECRET_KEY_OR_PASSWORD>" \
2 Views
Open in Slack
PreviousNext
Powered by