https://outerbounds.com/ logo
Join Slack
Powered by
I'm running into a strange issue and am a bit stum...
# ask-metaflow
f
I'm running into a strange issue and am a bit stumped. doesn't seem like 
host_volumes
is working properly. i've got an AWS setup, using fargate. the recent change is that i'm using a custom container. the batch job is able to launch the ec2 instance, and the task starts, but for some reason the mount isn't passed through to the container properly. more details and things in thread.
✅ 1
here's a flow i'm using to test:
Copy code
"""
dead simple flow to make sure metaflow things are working like you expect
"""

from metaflow import batch, environment, FlowSpec, conda, pypi, resources, S3, step

import sys


class Flow(FlowSpec):
    @batch(
        gpu=1,
        host_volumes="/data:/data",
        image="ECR_REGISTRY:TAG",
    )
    @environment(vars={"NVIDIA_DRIVER_CAPABILITIES": "compute,utility"})
    @step
    def start(self):
        print(f"step executable: {sys.executable}")

        from pathlib import Path

        import os

        print("conda list:")
        print(os.system("conda list"))

        print("nvidia-smi:")
        print(os.system("nvidia-smi"))

        print("output of `df -h`")
        print(os.system("df -h"))

        import torch

        print("output of `ls -alh /data")
        print(os.system("ls -alh /data"))

        # import boto3

        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    Flow()
i've got user data set up for the EC2 instance that mounts the SSD on a 
g4dn
instance to 
/data
container definition looks like this:
Copy code
FROM --platform=linux/amd64 pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime

COPY env.yaml ./
RUN conda env update -n base --file env.yaml

RUN mkdir /logs && chown 1000 /logs
RUN mkdir /metaflow && chown 1000 /metaflow
RUN mkdir /data && chown 1000 /data
ENV HOME=/metaflow
WORKDIR /metaflow
USER 1000
output from the 
ls
command looks like this:
Copy code
is /data available
total 0
drwxr-xr-x 2 root root  6 Oct 31 00:23 .
drwxr-xr-x 1 root root 77 Oct 31 00:26 ..
0
looking at my batch job, the source path seems wrong.
separately, even though 
nvidia-smi
will report a GPU, and it's the pytorch cuda container, pytorch in the runtime will report it does not have access to CUDA.
f
how did you set up user_data ^?
f
that's a bit complicated, but we throw it into some CFN that's slightly modified from the metaflow template
relevant section from the user data:
Copy code
# mount the NVME drive
mkdir -p /data
mkfs -t xfs /dev/nvme1n1
mount /dev/nvme1n1 /data
and here's the output (i didn't copy the last line before i lost access to the console, but the mount command completes successfully)
Copy code
[   44.277328] cloud-init[4184]: + mkdir -p /data
[   44.278523] cloud-init[4184]: + mkfs -t xfs /dev/nvme1n1
[   44.296694] cloud-init[4184]: meta-data=/dev/nvme1n1           isize=512    agcount=4, agsize=7629395 blks
[   44.296880] cloud-init[4184]: =                       sectsz=512   attr=2, projid32bit=1
[   44.297009] cloud-init[4184]: =                       crc=1        finobt=1, sparse=1, rmapbt=0
[   44.297129] cloud-init[4184]: =                       reflink=1    bigtime=0 inobtcount=0
[   44.297256] cloud-init[4184]: data     =                       bsize=4096   blocks=30517578, imaxpct=25
[   44.297374] cloud-init[4184]: =                       sunit=0      swidth=0 blks
[   44.297494] cloud-init[4184]: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
[   44.297614] cloud-init[4184]: log      =internal log           bsize=4096   blocks=14901, version=2
[   44.297733] cloud-init[4184]: =                       sectsz=512   sunit=0 blks, lazy-count=1
[   44.297858] cloud-init[4184]: realtime =none                   extsz=4096   blocks=0, rtextents=0
[   44.397529] cloud-init[4184]: Discarding blocks...Done.
[   44.397771] cloud-init[4184]: + mount /dev/nvme1n1 /data
(as for pytorch + cuda, the CPU version is being reported in 
conda list
, so nevermind on that one)
f
you should look into using a launch template if you’re not already, my gut tells me your issue is here and not w the host volume mapping
when you say console, is this user data script part of an ami build or ?
f
we're using a launch template to inject the user data:
Copy code
{{ .LaunchTemplateName }}:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateData:
        UserData: '{{ .MountNVMELaunchTemplateUserData }}'
and by console i mean the actual console output you get from racing to connect to the ec2 instance as soon as it's launched
the other thing is, this launch template worked fine when i wasn't bringing my own container.
it is a permissions issue!
looks like a fix was just adding the last line in this updated user data:
Copy code
# mount the NVME drive
mkdir -p /data
mkfs -t xfs /dev/nvme1n1
mount /dev/nvme1n1 /data
chown 1000 /data
but i think that means there's some special sauce happening somewhere on the metaflow side of things that i couldn't find because this worked just fine using their default container. took a quick look through the codebase and didn't find anything, but am pretty curious what they're doing.
f
think its in the batch client code, I was poking around in there when your issue first popped up
I was interested bc my first reaction was “host volumes aren’t supported”, then went to the step decorator docs for batch and it wasn’t there, but neither was EFS and I know that’s supported so began my batch client journey
having the API docs be so out of date is a bummer, not only for humans directly referencing them but robots are getting inaccurate info
11 Views
Open in Slack
PreviousNext
Powered by