https://outerbounds.com/ logo
Join Slack
Powered by
aah, super useful :ok_hand: i made some minimal tw...
# ask-metaflow
n
aah, super useful 👌 i made some minimal tweaks and added a dev container (i'm on a windows machine) to run flows against the metadata service and see them appear in the UI, with artifacts appearing in the minio that's bind-mounted on local:
Copy code
# <https://docs.docker.com/reference/compose-file/>
---
name: metaflow-development
services:
  metaflow-metadata:
    image: docker.io/netflixoss/metaflow_metadata_service:v2.5.0
    container_name: metaflow-metadata
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      MF_METADATA_DB_HOST: postgres
      MF_METADATA_DB_PORT: 5432
      MF_METADATA_DB_USER: metaflow_user
      MF_METADATA_DB_PSWD: metaflow_passw0rd
      MF_METADATA_DB_NAME: metaflow_db
      MF_METADATA_PORT: 8080
      MF_METADATA_HOST: 0.0.0.0
    ports:
      - '127.0.0.1:8080:8080'
    healthcheck:
      test: ["CMD", "curl", "-f", "<http://localhost:8080/ping>"]
      interval: 10s
      timeout: 5s
      retries: 5

  metaflow-ui:
    image: docker.io/netflixoss/metaflow_metadata_service:v2.5.0
    container_name: metaflow-ui
    depends_on:
      metaflow-metadata:
        condition: service_healthy
      minio:
        condition: service_healthy
      postgres:
        condition: service_healthy
    environment:
      MF_METADATA_DB_HOST: postgres
      MF_METADATA_DB_PORT: 5432
      MF_METADATA_DB_USER: metaflow_user
      MF_METADATA_DB_PSWD: metaflow_passw0rd
      MF_METADATA_DB_NAME: metaflow_db
      MF_UI_METADATA_PORT: 8083
      MF_UI_METADATA_HOST: 0.0.0.0
      UI_ENABLED: 1
      AWS_ACCESS_KEY_ID: minio_user
      AWS_SECRET_ACCESS_KEY: minio_passw0rd
      FEATURE_ARTIFACT_SEARCH: 1
      FEATURE_ARTIFACT_TABLE: 1
      METAFLOW_DEFAULT_DATASTORE: s3
      METAFLOW_DATASTORE_SYSROOT_S3: <s3://metaflow/>
      METAFLOW_S3_ENDPOINT_URL: <http://minio:9000/>
    ports:
      - '127.0.0.1:8083:8083'
    command: ["/opt/latest/bin/python3", "-m", "services.ui_backend_service.ui_server"]

  minio:
    image: docker.io/minio/minio:RELEASE.2025-09-07T16-13-09Z
    container_name: metaflow-minio
    volumes:
      - type: bind
        source: minio-data
        target: /data
        read_only: false
    environment:
      MINIO_ROOT_USER: minio_user
      MINIO_ROOT_PASSWORD: minio_passw0rd
    ports:
      - '127.0.0.1:9000:9000'
      - '127.0.0.1:9001:9001'
    command: server /data --console-address ":9001"
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 10s
      timeout: 5s
      retries: 5

  minio-initialize:
    image: docker.io/minio/mc:RELEASE.2025-08-13T08-35-41Z
    container_name: metaflow-minio-initialize
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      mc alias set myminio <http://minio:9000> minio_user minio_passw0rd;
      mc mb myminio/metaflow || true;
      mc anonymous set download myminio/metaflow;
      exit 0;
      "
  postgres:
    image: docker.io/library/postgres:18.1-trixie
    container_name: metaflow-postgres
    volumes:
      - type: bind
        source: postgres-data
        target: /var/lib/postgresql
        read_only: false
    environment:
      POSTGRES_USER: metaflow_user
      POSTGRES_PASSWORD: metaflow_passw0rd
      POSTGRES_DB: metaflow_db
    ports:
      - '127.0.0.1:5432:5432'
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U metaflow_user -d metaflow_db"]
      interval: 10s
      timeout: 5s
      retries: 5

  runner:
    image: python:3.14.5-slim
    container_name: metaflow-runner
    working_dir: /flow
    volumes:
      - ./flows:/flow
    environment:
      METAFLOW_DEFAULT_METADATA: service
      METAFLOW_SERVICE: <http://metaflow-metadata:8080>
      METAFLOW_SERVICE_URL: <http://metaflow-metadata:8080>
      METAFLOW_DEFAULT_DATASTORE: s3
      METAFLOW_DATASTORE_SYSROOT_S3: <s3://metaflow/>
      METAFLOW_S3_ENDPOINT_URL: <http://minio:9000>
      AWS_ACCESS_KEY_ID: minio_user
      AWS_SECRET_ACCESS_KEY: minio_passw0rd
    command: tail -f /dev/null
i've put the example 
my_flow.py
in the ./flows dir, so i can invoke it from inside the 
runner
service container:
Copy code
(quickstart) C:\Users\scher\Repositories\quickstart>docker ps
CONTAINER ID   IMAGE                                         COMMAND                  CREATED          STATUS                    PORTS                                NAMES
d04af86107a1   netflixoss/metaflow_metadata_service:v2.5.0   "/opt/latest/bin/pyt…"   55 seconds ago   Up 32 seconds             127.0.0.1:8083->8083/tcp             metaflow-ui
a25571c665df   netflixoss/metaflow_metadata_service:v2.5.0   "/bin/sh -c 'python3…"   55 seconds ago   Up 43 seconds (healthy)   127.0.0.1:8080->8080/tcp             metaflow-metadata
b7e8a5c52219   python:3.14.5-slim                            "tail -f /dev/null"      55 seconds ago   Up 54 seconds                                                  metaflow-runner
8b2789cd15f8   minio/minio:RELEASE.2025-09-07T16-13-09Z      "/usr/bin/docker-ent…"   55 seconds ago   Up 54 seconds (healthy)   127.0.0.1:9000-9001->9000-9001/tcp   metaflow-minio
1bb10594a2f5   postgres:18.1-trixie                          "docker-entrypoint.s…"   55 seconds ago   Up 54 seconds (healthy)   127.0.0.1:5432->5432/tcp             metaflow-postgres

(quickstart) C:\Users\scher\Repositories\quickstart>docker exec -it b7e8a5c52219 bash
root@b7e8a5c52219:/flow# pip install metaflow
Collecting metaflow
  Downloading metaflow-2.19.32-py2.py3-none-any.whl.metadata (7.1 kB)
Collecting requests (from metaflow)
  Downloading requests-2.34.2-py3-none-any.whl.metadata (4.8 kB)
Collecting boto3 (from metaflow)
  Downloading boto3-1.43.25-py3-none-any.whl.metadata (6.6 kB)
Collecting botocore<1.44.0,>=1.43.25 (from boto3->metaflow)
  Downloading botocore-1.43.25-py3-none-any.whl.metadata (5.6 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->metaflow)
  Downloading jmespath-1.1.0-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.19.0,>=0.18.0 (from boto3->metaflow)
  Downloading s3transfer-0.18.0-py3-none-any.whl.metadata (1.7 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.44.0,>=1.43.25->boto3->metaflow)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore<1.44.0,>=1.43.25->boto3->metaflow)
  Downloading urllib3-2.7.0-py3-none-any.whl.metadata (6.9 kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.44.0,>=1.43.25->boto3->metaflow)
  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Collecting charset_normalizer<4,>=2 (from requests->metaflow)
  Downloading charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
Collecting idna<4,>=2.5 (from requests->metaflow)
  Downloading idna-3.18-py3-none-any.whl.metadata (6.1 kB)
Collecting certifi>=2023.5.7 (from requests->metaflow)
  Downloading certifi-2026.5.20-py3-none-any.whl.metadata (2.5 kB)
Downloading metaflow-2.19.32-py2.py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 25.8 MB/s  0:00:00
Downloading boto3-1.43.25-py3-none-any.whl (140 kB)
Downloading botocore-1.43.25-py3-none-any.whl (15.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.2/15.2 MB 46.7 MB/s  0:00:00
Downloading jmespath-1.1.0-py3-none-any.whl (20 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading s3transfer-0.18.0-py3-none-any.whl (88 kB)
Downloading urllib3-2.7.0-py3-none-any.whl (131 kB)
Downloading six-1.17.0-py2.py3-none-any.whl (11 kB)
Downloading requests-2.34.2-py3-none-any.whl (73 kB)
Downloading charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (215 kB)
Downloading idna-3.18-py3-none-any.whl (65 kB)
Downloading certifi-2026.5.20-py3-none-any.whl (134 kB)
Installing collected packages: urllib3, six, jmespath, idna, charset_normalizer, certifi, requests, python-dateutil, botocore, s3transfer, boto3, metaflow
Successfully installed boto3-1.43.25 botocore-1.43.25 certifi-2026.5.20 charset_normalizer-3.4.7 idna-3.18 jmespath-1.1.0 metaflow-2.19.32 python-dateutil-2.9.0.post0 requests-2.34.2 s3transfer-0.18.0 six-1.17.0 urllib3-2.7.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: <https://pip.pypa.io/warnings/venv>. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 26.1.1 -> 26.1.2
[notice] To update, run: pip install --upgrade pip
root@b7e8a5c52219:/flow# export USERNAME=test
root@b7e8a5c52219:/flow# python my_flow.py run
Metaflow 2.19.32 executing TestFlow for user:test
Validating your flow...
    The graph looks good!
Running pylint...
    Pylint not found, so extra checks are disabled.
2026-06-08 23:29:12.659 Workflow starting (run-id 1):
2026-06-08 23:29:13.106 [1/start/2 (pid 42)] Task is starting.
2026-06-08 23:29:13.631 [1/start/2 (pid 42)] hello from metaflow
2026-06-08 23:29:14.922 [1/start/2 (pid 42)] Task finished successfully.
2026-06-08 23:29:15.131 [1/end/3 (pid 79)] Task is starting.
2026-06-08 23:29:15.643 [1/end/3 (pid 79)] done
2026-06-08 23:29:16.877 [1/end/3 (pid 79)] Task finished successfully.
2026-06-08 23:29:16.981 Done!
see screenshot 👇 for the UI showing the flow/run. you can also check the artifact download from minio is working:
Copy code
root@b7e8a5c52219:/flow# python
Python 3.14.5 (main, May 19 2026, 23:46:18) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from metaflow import Task
>>> Task('TestFlow/1/start/2', attempt=0)['message'].data
'hello artifact world'
>>>
the 
myflow.py
in question:
Copy code
from metaflow import FlowSpec, step

class TestFlow(FlowSpec):

    @step
    def start(self):
        print("hello from metaflow")

        # 👇 This becomes an artifact
        self.message = "hello artifact world"

        self.next(self.end)

    @step
    def end(self):
        print("done")

if __name__ == "__main__":
    TestFlow()
hope this ☝️ helps others - took me quite a while to find a working setup on windows, and still not sure how traffic flows between metadata service and ui, and why both are implemented in the same image 😅
9 Views
Open in Slack
PreviousNext
Powered by