https://outerbounds.com/ logo
Join Slack
Powered by
Hello colleagues! Today I faceced with the issue ...
# ask-metaflow
b
Hello colleagues! Today I faceced with the issue (on screenshot). And I don't understand where to look because in the morning everyhting was working. I already deleted workflow, cleaned s3 and now I'm completely stuck. Environment pypi. Run on Step functions.
a
@bitter-school-29575 can you share what you specify using 
@pypi
decorator?
b
Thank you for reply! I tried:
Copy code
@pypi_base(python='3.9', packages={
    'httpx': '>=0.20.0,<0.26.0',
    'attrs': '>=21.3.0',
    'python-dateutil': '2.8.2',
    'numpy': '>=1.26.2',
    'imageio': '2.33.1',
    'imageio-ffmpeg': '0.4.9',
    # 'ffmpeg-python': '>=0.2.0',
    'matplotlib': '>=3.8.2',
    'opencv-python': '>=4.2'
})
I also tried without python. I have 
3.11.6
installed locally.
What changed: initially I created flow with 
conda
. After this, I switched to 
pypi
and everything started working. After this problems with 
conda
repo everything stopped working
Ok. Seems I found an issue. If I remove open-cv it will work... But I really need it... Yes, 99% this is related to OpenCV. Can it be related with micromamba issue ? Because it was working in the morning
b
there is an open issue on this, or at least the same error: https://github.com/Netflix/metaflow/issues/1498 Been looking into the possible causes for this but so far the only luck I've had reproducing it are related to the cached dependencies disappearing from the blob storage either by manually deleting them, or by a retention policy that has been set on the storage. the PyPI environment holds a mapping locally in a file named 
conda.manifest
, this can be found in the 
.metaflow/FlowName/
folder. The environment resolving process uses the manifest to determine whether a package has already been pushed to the datastore or not (this depends on the package having a known 
path:
or not). As a first step could you try verifying if the path for the opencv package in your conda.manifest still exists in the blob storage?
Copy code
{
  "url": "<https://files.pythonhosted.org/packages/b7/8a/b2f7e1a434d56bf1d7570fc5941ace0847404e1032d7f1f0b8fed896568d/opencv_python-4.8.1.78-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>",
  "path": "<http://files.pythonhosted.org/packages/b7/8a/b2f7e1a434d56bf1d7570fc5941ace0847404e1032d7f1f0b8fed896568d/opencv_python-4.8.1.78-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl|files.pythonhosted.org/packages/b7/8a/b2f7e1a434d56bf1d7570fc5941ace0847404e1032d7f1f0b8fed896568d/opencv_python-4.8.1.78-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
}
as an example this is an excerpt from my conda.manifest. The path is relative to the configured datastore, and should be found under the 
conda/
prefix in there.
b
I don't have in Blob (in S3). Will try to delete 
.metaflow
I've deleted. Issue still exit. And now It's reproduces without open-cv. Is there any workaround? I really stuck with this...
b
I've seen something like this where a corrupted package was uploaded to the S3 cache. There should be packages cached in your S3 bucket in 
metaflow/conda
If you delete all recently uploaded packages there, Metaflow should check which ones are cached and re-upload if required. Maybe there is also one corrupted on your local machine, back then I checked all conda/tar archives in my local conda packages (e.g. 
~/miniforge3/pkgs
) and found one that wasn't a valid tar file. This one was cached in S3 and caused flows that need it to crash in the bootstrap phase
I think I also saw it before when a colleague resolved an environment with a different Metaflow profile and the packages were cached in a different S3 bucket and then tried to run with the original Metaflow profile. The 
conda.manifest
/ 
conda.dependencies
files only contain the relative paths so if the datastore root changes, the packages will also be missing
❤️ 1
b
@billowy-exabyte-36648, thank you! Honestly speaking I'm not familiar with micromamba. And I didn't know that it created a lot of internal caches. So I opened my 
~/
directory and cleaned everything related to conda / anaconda / mamba. After this seems everything is working.
🙌 1
8 Views
Open in Slack
PreviousNext
Powered by