https://outerbounds.com/ logo
Join Slack
Powered by
Hi Team, I am working on seting up auto-deployment...
# ask-metaflow
b
Hi Team, I am working on seting up auto-deployment to step-functions using github actions. Locally I have everything working as expected but on my github actions workflow, I keep running into the below error message, even though I have boto3 installed. Anyone run into this issue before? 
python churn_prediction_flow.py --environment=conda --with batch --with retry step-functions create
Copy code
Uploading S3 files failed.
    First key: <http://conda.anaconda.org/conda-forge/linux-64/libssh2-1.10.0-hf14f497_3.tar.bz2/d85acad4b47dff4e3def14a769a97906/libssh2-1.10.0-hf14f497_3.tar.bz2|conda.anaconda.org/conda-forge/linux-64/libssh2-1.10.0-hf14f497_3.tar.bz2/d85acad4b47dff4e3def14a769a97906/libssh2-1.10.0-hf14f497_3.tar.bz2>
    Error: Traceback (most recent call last):
      File "/home/runner/.local/lib/python3.10/site-packages/metaflow/plugins/datatools/***/***op.py", line 18, in <module>
        from boto3.***.transfer import TransferConfig
    ModuleNotFoundError: No module named 'boto3'
✅ 1
Copy code
name: Metaflow Deploy

on:
  push:
    branches-ignore:
      - main

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_DEFAULT_REGION: us-west-2
  AWS_DEFAULT_OUTPUT: json

jobs:
  deploy:
    runs-on: ubuntu-latest
    defaults:
      run:
        shell: bash -l {0}
    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Set up Miniconda
      uses: conda-incubator/setup-miniconda@v2
      with:
        auto-activate-base: true

    - name: Install Metaflow and dependencies 
      run: |
        pip install metaflow
        pip install pandas
        pip install boto3
        conda install mamba -n base -c conda-forge
        conda install -c conda-forge -n base metaflow pandas boto3

    - name: Configure Metaflow
      env:
        METAFLOW_BATCH_JOB_QUEUE: ${{ secrets.METAFLOW_BATCH_JOB_QUEUE }}
        METAFLOW_DATASTORE_SYSROOT_S3: ${{ secrets.METAFLOW_DATASTORE_SYSROOT_S3 }}
        METAFLOW_DATATOOLS_SYSROOT_S3: ${{ secrets.METAFLOW_DATATOOLS_SYSROOT_S3 }}
        METAFLOW_DEFAULT_DATASTORE: ${{ secrets.METAFLOW_DEFAULT_DATASTORE }}
        METAFLOW_DEFAULT_METADATA: ${{ secrets.METAFLOW_DEFAULT_METADATA }}
        METAFLOW_ECS_FARGATE_EXECUTION_ROLE: ${{ secrets.METAFLOW_ECS_FARGATE_EXECUTION_ROLE }}
        METAFLOW_ECS_S3_ACCESS_IAM_ROLE: ${{ secrets.METAFLOW_ECS_S3_ACCESS_IAM_ROLE }}
        METAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE: ${{ secrets.METAFLOW_EVENTS_SFN_ACCESS_IAM_ROLE }}
        METAFLOW_SERVICE_AUTH_KEY: ${{ secrets.METAFLOW_SERVICE_AUTH_KEY }}
        METAFLOW_SERVICE_INTERNAL_URL: ${{ secrets.METAFLOW_SERVICE_INTERNAL_URL }}
        METAFLOW_SERVICE_URL: ${{ secrets.METAFLOW_SERVICE_URL }}
        METAFLOW_SFN_DYNAMO_DB_TABLE: ${{ secrets.METAFLOW_SFN_DYNAMO_DB_TABLE }}
        METAFLOW_SFN_IAM_ROLE: ${{ secrets.METAFLOW_SFN_IAM_ROLE }}
        METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE: ${{ secrets.METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE }}
        METAFLOW_AWS_SECRETS_MANAGER_DEFAULT_REGION: ${{ secrets.METAFLOW_AWS_SECRETS_MANAGER_DEFAULT_REGION }}
        METAFLOW_CONDA_DEPENDENCY_RESOLVER: ${{ secrets.METAFLOW_CONDA_DEPENDENCY_RESOLVER }}

    - name: Deploy Metaflow project to Step Functions
      run: |
        python churn_prediction_flow.py --environment=conda --with batch --with retry step-functions create
a
can you try with 
python -m pip install boto3
to make sure there is no issue with pip installer
b
Sure, let me give that a try
Nope, same error
@ancient-application-36103 any thoughts around this?
c
Was able to reproduce and fix the issue by adding 
'boto3': '1.26.136'
or whatever version you'd like into the 
@conda
or 
@conda_base
decorator.
b
@crooked-jordan-29960 I am still running into the same error, I followed you instructions and tried deploying with different boto3 versions with no luck
Copy code
@schedule(cron="0 1 ? * 6 *")  # every Friday at 1am
@conda_base(
    libraries={
        "numpy": "1.23.5",
        "pandas": "1.5.3",
        "scikit-learn": "1.2.1",
        "xgboost": "1.7.4",
        "shap": "0.41.0",
        "snowflake-connector-python": "3.0.3",
        "seaborn": "0.12.2",
        "matplotlib": "3.7.1",
        "boto3": "1.26.135",
    },
    python="3.10.6",
)

class TemplateFlow(FlowSpec):
c
does a hello world flow work with your git actions set up? also, have you tried activating an environment in the action before you install metaflow, and then keeping it active for the 
python churn_prediction_flow.py --environment=conda --with batch --with retry step-functions create
part? I think there may be a virtual env misalignment.
b
I finally managed to get it working, followed your advice to activate the conda environment first and then install the dependencies and configure metaflow. Thank you so much.
🙌 1
8 Views
Open in Slack
PreviousNext
Powered by