Hello does anyone know how to get reliable conda env creatio Outerbounds #ask-metaflow

Hello, does anyone know how to get reliable conda ...

rich-agent-87730

04/10/2023, 9:27 PM

Hello, does anyone know how to get reliable conda env creation in metaflow? we have a pretty big foreach step (lets say 500 branched steps) and some of them fail intermittently with

Copy code

requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)"

when trying to create the conda env.

✅ 1

ancient-application-36103

04/10/2023, 9:29 PM

@rich-agent-87730 is this locally on your laptop before any step executes or in the cloud?

rich-agent-87730

04/10/2023, 9:29 PM

this is on an EC2 instance in the cloud

rich-agent-87730

04/10/2023, 9:29 PM

we are also running the flow through AWS Batch

rich-agent-87730

04/11/2023, 6:56 PM

Hi @square-wire-39606, sorry for the ping but is there anything I can do about this?

ancient-application-36103

04/11/2023, 6:57 PM

Hi! Where exactly is the error happening? Can you share the user facing logs?

rich-agent-87730

04/11/2023, 6:59 PM

The log is really large but its happens after Environment bootstrapped:

Copy code

2023-04-09 05:37:27.300 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650] Environment bootstrapped.\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650] # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     Traceback (most recent call last):\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 761, in _update_chunk_length\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         self.chunk_left = int(line, 16)\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     ValueError: invalid literal for int() with base 16: b''\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     During handling of the above exception, another exception occurred:\
2023-04-09 05:37:26.950 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     Traceback (most recent call last):\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 444, in _error_catcher\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         yield\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 828, in read_chunked\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         self._update_chunk_length()\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 765, in _update_chunk_length\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         raise InvalidChunkLength(self, line)\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     During handling of the above exception, another exception occurred:\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     Traceback (most recent call last):\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/requests/models.py", line 816, in generate\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         yield from self.raw.stream(chunk_size, decode_content=True)\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 624, in stream\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         for line in self.read_chunked(amt, decode_content=decode_content):\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 816, in read_chunked\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         with self._error_catcher():\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/contextlib.py", line 153, in __exit__\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         self.gen.throw(typ, value, traceback)\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]       File "/metaflow/miniconda3/lib/python3.10/site-packages/urllib3/response.py", line 461, in _error_catcher\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]         raise ProtocolError("Connection broken: %r" % e, e)\
2023-04-09 05:37:26.951 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650]     urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))\
...
2023-04-09 05:37:26.952 [24314/start/450182 (pid 17375)] [f95be356-aab1-4dbb-acc1-474873686650] `$ /metaflow/miniconda3/bin/conda-env create -f /tmp/tmp9ycxq0gb/conda.yaml -p /metaflow/metaflow_PullDefaultDashboardData_linux-64_91c6dae53afa414af0b00cf6ca5cd062c91b31e3 -q`\

ancient-application-36103

04/12/2023, 12:59 AM

can you share what packages are you specifying for

@conda

ancient-application-36103

04/12/2023, 1:00 AM

Also, I am not sure where this logline

$ /metaflow/miniconda3/bin/conda-env create -f /tmp/tmp9ycxq0gb/conda.yaml -p /metaflow/metaflow_PullDefaultDashboardData_linux-64_91c6dae53afa414af0b00cf6ca5cd062c91b31e3 -q

is coming from? Are you using metaflow's

@conda

implementation or something else (metaflow extension's

@conda

or a custom implementation)?

ancient-application-36103

04/12/2023, 1:02 AM

it seems to me that your

@conda

implementation is trying to read from a custom

conda.yaml

file on AWS Batch and DDOSing the upstream conda channel resulting in this issue. To prevent this issue, Metaflow's

@conda

caches the packages in S3 from your workstation and builds the environment on AWS Batch using these cached packages bypassing any DDOS concerns

rich-agent-87730

04/12/2023, 1:04 AM

oh icic. that makes sense. we are using a custom

@conda

implementation

👍🏼 1

rich-agent-87730

04/12/2023, 1:08 AM

Thank you for digging into this

ancient-application-36103

04/12/2023, 1:09 AM

no worries. glad we could figure out the issue.

brief-kite-90012

04/12/2023, 7:35 AM

@curved-island-17262

13 Views

Open in Slack

Previous Next