https://outerbounds.com/ logo
Join Slack
Powered by
hey all, running into a couple of issues that have...
# ask-metaflow
f
hey all, running into a couple of issues that have to do with conda and batch mode (with AWS). first one: trying to use the pytorch-blessed conda install-with-cuda method. am specifying 
"nvidia::pytorch-cuda": "11.7"
in the 
@conda
decorator, but am getting this error when metaflow tries to build the conda env:
Copy code
Bootstrapping conda environment...(this could take a few minutes)
    Conda ran into an error while setting up environment.:
    Step: start, Error: PackagesNotFoundError: The following packages are not available from current channels:

      - nvidia::pytorch-cuda==11.7

    Current channels:

      - <https://conda.anaconda.org/pytorch/linux-64>
      - <https://conda.anaconda.org/pytorch/noarch>
      - <https://conda.anaconda.org/conda-forge/linux-64>
      - <https://conda.anaconda.org/conda-forge/noarch>
      - <https://repo.anaconda.com/pkgs/main/linux-64>
      - <https://repo.anaconda.com/pkgs/main/noarch>
      - <https://repo.anaconda.com/pkgs/r/linux-64>
      - <https://repo.anaconda.com/pkgs/r/noarch>
if I try to use conda directly to do the same thing, conda is able to find the package:
Copy code
CONDA_SUBDIR=linux-64 conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
finds
Copy code
pytorch-cuda       pytorch/linux-64::pytorch-cuda-11.7-h778d358_5
i feel like i'm missing something. any clues?
here's the flow I'm using spelled out:
Copy code
"""
dead simple flow to make sure metaflow things are working like you expect
"""

from metaflow import batch, environment, FlowSpec, conda, resources, step

import sys


class Flow(FlowSpec):
    @batch(gpu=1, host_volumes="/data")
    # @batch(host_volumes="/data")
    @conda(
        libraries={
            "pytorch::pytorch": "1.13.1",
            "pytorch::torchvision": "0.14.1",
            "nvidia::pytorch-cuda": "11.7",
        },
        python="3.10.6",
    )
    @environment(vars={"NVIDIA_DRIVER_CAPABILITIES": "compute,utility"})
    @step
    def start(self):
        print(f"step executable: {sys.executable}")
        from pathlib import Path

        import torch
        import os

        print(f"cuda is available: {torch.cuda.is_available()}")

        test_file = Path("/data/test_file.txt")
        test_text = "hello, you are a winner"
        test_file.write_text(test_text)
        read_text = test_file.read_text()
        print(read_text)

        print("output of `df -h`")
        print(os.system("df -h"))

        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    Flow()
prepending the command to run the flow with 
CONDA_CHANNELS=pytorch,nvidia
doesn't seem to have an effect
👀 1
I didn't realize 
_create
was passing the 
::
-style 
channel::package
requirements directly to the conda CLI. can replicate the bug with
Copy code
CONDA_SUBDIR=linux-64 conda install pytorch::pytorch==1.13.1 pytorch::torchvision==0.14.1 nvidia::pytorch-cuda=11.7
but mamba doesn't have a problem with it. this command ran successfully:
Copy code
CONDA_SUBDIR=linux-64 mamba install pytorch::pytorch==1.13.1 pytorch::torchvision==0.14.1 nvidia::pytorch-cuda=11.7
i changed my metaflow config to use mamba as well and that seemed to fix the issue. i guess this is a bug with conda? i guess i should file an issue on their github? kinda wild
and then i updated conda/mamba and now mamba can't solve it 😞
conda/mamba don't have a problem with the 
-c CHANNEL_NAME
syntax—i could submit a PR to extract the channels and reformat the command
s
could you try with a fresh install of conda/mamba?
f
yes, have done this already, actually. conda 23.5.0 stil exhibits this issue.
Copy code
conda --version
CONDA_SUBDIR=linux-64 conda install pytorch::pytorch==1.13.1 pytorch::torchvision==0.14.1 nvidia::pytorch-cuda==11.7
conda 23.5.0
Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - nvidia::pytorch-cuda==11.7

Current channels:

  - <https://conda.anaconda.org/pytorch/linux-64>
  - <https://conda.anaconda.org/pytorch/noarch>
  - <https://conda.anaconda.org/conda-forge/linux-64>
  - <https://conda.anaconda.org/conda-forge/noarch>
  - <https://repo.anaconda.com/pkgs/main/linux-64>
  - <https://repo.anaconda.com/pkgs/main/noarch>
  - <https://repo.anaconda.com/pkgs/r/linux-64>
  - <https://repo.anaconda.com/pkgs/r/noarch>

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    <https://anaconda.org>

and use the search bar at the top of the page.
removing 
nvidia::
and adding 
-c nvidia
hangs:
Copy code
conda --version
CONDA_SUBDIR=linux-64 conda install pytorch::pytorch==1.13.1 pytorch::torchvision==0.14.1 pytorch-cuda==11.7 -c nvidia
conda 23.5.0
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: | - / \ | / %
c
@fast-honey-9693 did you manage to solve this?
10 Views
Open in Slack
PreviousNext
Powered by