https://outerbounds.com/ logo
Join Slack
Powered by
hello all, I got Bootstrapping conda environment e...
# ask-metaflow
a
hello all, I got Bootstrapping conda environment error again and need some help! I used to see the error when conda cache is activated (please check out two github action yaml files in thread), the way how I resolved the error was increasing cache number or disable conda cache, but the error raised again yesterday. Within same github action, one run may succeed and the next run may fail. I have two yaml files, they kind of does almost the same thing, but one may succeed and the other may fail, and the errors could happen at any step of any metaflow pipelines, and it's hard to debug and replicate. Does anyone know what may cause the errors? check out detailed error and github action file in thread, appreciate any help!
✅ 1
sample error
Copy code
Bootstrapping conda environment...(this could take a few minutes)
    Conda ran into an error while setting up environment.:
    Step: run_visualization_and_upload_files, Error: command '['/usr/share/miniconda/condabin/conda', 'create', '--yes', '--no-default-packages', '--name', 'metaflow_CRNNTTCaption_linux-64_7317c62484b0e34129b5cdcb286dac6b3c879ef9', '--quiet', b'python==3.8.13', b'requests==2.26.0', b'boto3==1.19.12', b'awscli==1.21.12', b'cloudpickle==2.0.0', b'dask==2021.3.0', b'dataclasses-json==0.5.3', b'fastavro==1.4.7', b'fastparquet==0.7.2', b'joblib==1.1.0', b'numpy==1.20.3', b'pandas==1.3.4', b'pip==21.3.1', b'pyparsing==2.4.7', b'pyarrow==6.0.1', b'python-snappy==0.6.0', b'pytz==2021.3', b's3fs==0.4.2', b'scikit-learn==0.23.2', b'scipy==1.7.3', b'snappy==1.1.8', b'smart_open==5.2.1', b'sqlalchemy==1.3.23', b'testpath==0.5.0', b'tornado==6.1', b'tzdata==2021e', b'pycryptodome==3.11.0', b'wandb==0.12.1', b'fastai::fastcore==1.3.26', b'snowflake-connector-python[pandas]==2.7.0', b'snowflake-sqlalchemy==1.2.4', b'great-expectations==0.13.31', b'anaconda::widgetsnbextension==3.5.1', b'jinja2==3.0.3', b'nbconvert==5.6.1', b'nbformat==5.1.3', b'jupyter_client==6.1.7', b'jupyterlab==3.2.4', b'seaborn==0.11.2', b'ipython==7.30.1']' returned error (-9): b'', stderr=b''

Error: Process completed with exit code 1.
github action yaml files
Copy code
name: Build Conda Dependencies for Flows

on:
  pull_request:
    types: [ opened, synchronize, reopened ]

jobs:
  # This is the first job that build basic for this entire workflow
  build_flow_dependency:
    timeout-minutes: 120
    runs-on: ubuntu-latest
    name: Build flow dependency
    defaults:
      run:
        shell: bash -l {0}

    strategy:
      matrix:
        flow_script: [
          train_cr_yt.py,
          train_cr_tt.py,
          train_cr_tt_snorkel.py,
          train_cr_tt_caption_only.py,
          train_cr_tt_snorkel_caption_only.py,
        ]

    steps:
      - uses: actions/checkout@v2
      - name: Declare env-var variables
        id: vars
        shell: bash
        run: |
          echo "::set-output name=branch::$(echo $GITHUB_REF | cut -d'/' -f 3)"
          echo "::set-output name=commit_hash::$(git rev-parse "$GITHUB_SHA")"
      # - name: Cache conda
      #   uses: actions/cache@v2
      #   env:
      #     # Increase this value to reset cache if ../environment.yml has not changed
      #     CACHE_NUMBER: 4
      #   with:
      #     path: ~/conda_pkgs_dir
      #     key:
      #       ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
      #       hashFiles('environment.yaml') }}
      - name: Setup conda env
        uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: metaflow
          channels: conda-forge,defaults
          auto-activate-base: true #false
          environment-file: environment.yaml
          # IMPORTANT: This needs to be set for caching to work properly! 
          # true uses previous conda cache, false does not 
          use-only-tar-bz2: false #true 
          python-version: 3.8
      # - name: Cache pip
      #   uses: actions/cache@v2
      #   env:
      #     # Increase this value to reset cache if ../environment.yml has not changed
      #     CACHE_NUMBER: 2
      #   with:
      #     # This path is specific to Ubuntu
      #     path: ~/.cache/pip
      #     # Look to see if there is a cache hit for the corresponding requirements file
      #     key: ${{ runner.os }}-pip-${{ env.CACHE_NUMBER }}-${{ hashFiles('requirements.txt', 'test_requirements.txt') }}
      #     restore-keys: |
      #       ${{ runner.os }}-pip-${{ hashFiles('requirements.txt', 'test_requirements.txt') }}
      #       ${{ runner.os }}-pip-
      #       ${{ runner.os }}-
      - name: Handle pip install
        env:
          PIP_INDEX_URL: https://${{secrets.ARTIFACTORY_USERNAME}}:${{secrets.ARTIFACTORY_PASSWORD}}@zefr.jfrog.io/zefr/api/pypi/pypi/simple
        run: pip install -r requirements.txt -r test_requirements.txt
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.DSPROD_AWS_ACCESS_KEY_ID_METAFLOW }}
          aws-secret-access-key: ${{ secrets.DSPROD_AWS_SECRET_ACCESS_KEY_METAFLOW }}
          aws-region: us-east-1
      - name: Conda info
        run: conda info
      - name: Conda list
        run: conda list
      - name: Configure MetaFlow
        run: metaflow configure import ds-prod.json
      - name: Build metaflow flow dependency
        env:
          FLOW_SCRIPT: ${{matrix.flow_script}}
          ENV: ds-prod
        # package list is a metaflow cli command that only lists all dependencies and see if they build
        run: >
          python $FLOW_SCRIPT --environment=conda package list
Copy code
name: DS-PROD Flow Deployment

# on:
#   workflow_dispatch:

#   push:
#     branches:
#       - master

# TODO: delete and change back 
on:
  pull_request:
    types: [ opened, synchronize, reopened ]

jobs:
  build:
    timeout-minutes: 120
    runs-on: ubuntu-latest
    name: Build flow dependency
    defaults:
      run:
        shell: bash -l {0}

    strategy:
      matrix:
        flow_script: [
          train_cr_yt.py,
          train_cr_tt.py,
          train_cr_tt_snorkel.py,
          train_cr_tt_caption_only.py,
          train_cr_tt_snorkel_caption_only.py,
        ]
        include:
          # includes an AWS Step Functions deployment token for each flow
          - flow_script: train_cr_yt.py
            flow_name: ContentRelevancyNN
            deploy_token: CONTENT_RELEVANCY_NN_FLOW_TOKEN
          - flow_script: train_cr_tt.py
            flow_name: ContentRelevancyNNTikTok
            deploy_token: CONTENT_RELEVANCY_NN_TIKTOK_FLOW_TOKEN
          - flow_script: train_cr_tt_snorkel.py
            flow_name: ContentRelevancyNNTikTokSnorkel
            deploy_token: CONTENT_RELEVANCY_NN_TIKTOK_SNORKEL_FLOW_TOKEN
          - flow_script: train_cr_tt_caption_only.py
            flow_name: CRNNTTCaption
            deploy_token: CR_NN_TT_CAPTION_FLOW_TOKEN
          - flow_script: train_cr_tt_snorkel_caption_only.py
            flow_name: CRNNTTSnorkelCaption
            deploy_token: CR_NN_TT_SNORKEL_CAPTION_FLOW_TOKEN

    steps:
      - uses: actions/checkout@v2
      - name: Declare env-var variables
        id: vars
        shell: bash
        run: |
          echo "::set-output name=branch::$(echo $GITHUB_REF | cut -d'/' -f 3)"
          echo "::set-output name=commit_hash::$(git rev-parse "$GITHUB_SHA")"
      - name: Setup conda env
        uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: metaflow
          channels: conda-forge,defaults
          auto-activate-base: true #false
          environment-file: environment.yaml
          # IMPORTANT: This needs to be set for caching to work properly!
          # true uses previous conda cache, false does not 
          use-only-tar-bz2: false #true 
          python-version: 3.8
      - name: Handle pip install
        env:
          PIP_INDEX_URL: https://${{secrets.ARTIFACTORY_USERNAME}}:${{secrets.ARTIFACTORY_PASSWORD}}@zefr.jfrog.io/zefr/api/pypi/pypi/simple
        run: pip install -r requirements.txt -r test_requirements.txt
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.DSPROD_AWS_ACCESS_KEY_ID_METAFLOW }}
          aws-secret-access-key: ${{ secrets.DSPROD_AWS_SECRET_ACCESS_KEY_METAFLOW }}
          aws-region: us-east-1
      - name: Conda info
        run: conda info
      - name: Conda list
        run: conda list
      - name: Configure MetaFlow
        run: metaflow configure import ds-prod.json
      - name: Deploy flow to AWS Step Functions
        env:
          BRANCH: ${{steps.vars.outputs.branch}}
          COMMIT_HASH: ${{steps.vars.outputs.commit_hash}}
          ENV: ds-prod
        run: >
          python ${{matrix.flow_script}}
          --no-pylint
          --package-suffixes=.ipynb,.json
          --environment=conda
          --with retry
          --with batch
          --production
          step-functions create
          --authorize ${{secrets[matrix.deploy_token]}}
          --max-workers 250
s
where does the error occur? for the first action or the second?
a
the error can happen at any step of any metaflow pipelines of any github action flow
s
I believe your GH runner is running out of memory while setting up the environment.
exit code 9 is usually reflective of process termination due to OutOfMemory
a
it is exit code 1, the weird thing is that some pipeline worked fine in previous run but failed now.
we also have multiple pipelines, and their dependency is almost the same, I don't fully understand if it's OOM, why some pipeline build succeeded and some failed.
s
the conda process 
returned error (-9):
I am not sure how the GH runners are configured - but it's likely that GH may allow your runners to burst memory consumption if there is availability and the failures that you see happen when extra memory is not available - making these failures seemingly random.
a
gotcha, let me try increasing GH action memory and see how it goes
17 Views
Open in Slack
PreviousNext
Powered by