I’ve tried rolling back to old versions of metaflow but am seeing the same error

i've seen similar errors when some of the prerequisites were missing like wget/curl/tar

We’ve been using this for a while without issue

its typically something with not being able to download task code package. The (hard) way I debug this if nothing else helps is: • get pod YAML via kubectl get, edit the bootstrap shell script so it doesn't redirect errors to

/dev/null

• apply modified YAML to cluster • look at the logs again

I tried editing this block and removing the /dev/null reference after the job.tar:

- ${METAFLOW_INIT_SCRIPT:+eval "${METAFLOW_INIT_SCRIPT}"} && true && mkdir -p
      $PWD/.logs && export PYTHONUNBUFFERED=x MF_PATHSPEC=HelloConda/6167/start/43059
      MF_DATASTORE=s3 MF_ATTEMPT=0 MFLOG_STDOUT=$PWD/.logs/mflog_stdout MFLOG_STDERR=$PWD/.logs/mflog_stderr
      && mflog(){ T=$(date -u -Ins|tr , .); echo "[MFLOG|0|${T:0:26}Z|task|$T]$1"
      >> $MFLOG_STDOUT; echo $1;  } && mflog 'Setting up task environment.' && python
      -m pip install requests -qqq && python -m pip install awscli boto3 -qqq && mkdir
      metaflow && cd metaflow && mkdir .metaflow && i=0; while [ $i -le 5 ]; do mflog
      'Downloading code package...'; python -m awscli ${METAFLOW_S3_ENDPOINT_URL:+--endpoint-url="${METAFLOW_S3_ENDPOINT_URL}"}
      s3 cp <s3://dsci-metaflow-s3-prd/metaflow/HelloConda/data/3b/3bcc18bb34ce8b096f3b4c4c1452138b2b2ac4b0>
      job.tar >/dev/null && mflog 'Code package downloaded.' && break; sleep 10; i=$((i+1));
      done && if [ $i -gt 5 ]; then mflog 'Failed to download code package from <s3://-->-/metaflow/HelloConda/data/3b/3bcc18bb34ce8b096f3b4c4c1452138b2b2ac4b0
      after 6 tries. Exiting...' && exit 1; fi && TAR_OPTIONS='--warning=no-timestamp'
      tar xf job.tar && mflog 'Task is starting.' && (python -u aaa_basic_flow.py
      --quiet --metadata service --environment local --datastore s3 --event-logger
      nullSidecarLogger --monitor nullSidecarMonitor --datastore-root <s3://dsci-metaflow-s3-prd/metaflow>
      --with kubernetes:cpu=1,memory=4096,disk=10240,image=public.ecr.aws/docker/library/python:3.11,service_account=argo-workflows,namespace=development,gpu_vendor=nvidia,tmpfs_tempdir=True,tmpfs_path=/metaflow_temp
      --package-suffixes .py,.R,.RDS --pylint step start --run-id 6167 --task-id 43059
      --input-paths ${METAFLOW_INPUT_PATHS_0} --retry-count 0 --max-user-code-retries
      0 --namespace user:tyler.potts) 1>> >(python -m metaflow.mflog.tee task $MFLOG_STDOUT)
      2>> >(python -m metaflow.mflog.tee task $MFLOG_STDERR >&2); c=$?; python -m
      metaflow.mflog.save_logs; exit $c

I didn’t see any additional logs

I created another pod with only a sleep timer so that I could shell into it in kubernetes It’s failing on the command

python -m pip install awscli

(not sure if it has anything to do with it, it just stood out to me)

I’ll have to dig into the core reasons later if needed. For now I gotta go fork metaflow to add this to the bash script. All our prod jobs are failing currently because of it 😕

Thanks for the debugging direction! It’s been a big help

Ultimately, I think the fix is for

metaflow

to switch to the v2 CLI. Until then, the S3 datastore won't work. Happy to collaborate on a fix, since we're heavy users!

@ancient-application-36103 would it be possible to roll out an update to include

python -m pip install pyyaml==5.3.1

to the bash script installation or do we all have to fork Metaflow and manually add it ourselves?