https://outerbounds.com/ logo
Join Slack
Powered by
is there any documentation on how to create a Meta...
# ask-metaflow
w
is there any documentation on how to create a Metaflow extension? I found https://github.com/Netflix/metaflow-extensions-template but it's kind of confusing. Is there a HelloWorld example somewhere?
d
hey — I can probably improve the documentation but the “HelloWorld” is basically that repo. If you let me know what you want to do, I can do a starter repo for you with just what you need. Note that although this is extensively used at Netflix and is solid, it may break over time (when we change internal things). It will always be fixed over time but just giving the standard warning.
w
well, I dunno if I even need an extension. I'm just trying to create a 
@pip
decorator to install from a private CodeArtifact repo (I heard from Savin that this is already in the works, but I don't really wanna wait)
d
you probably don’t need an extension for that (if you are just trying to run a few commands before your step). I do have a full 
@pip
decorator. It doesn’t use CodeArtifact though but may be extensible. It’s here: https://github.com/Netflix/metaflow-nflx-extensions
if it isn’t too hard to support CodeArtifact, I can see about doing it (or feel free to submit a PR)
w
d
if you could pass it to pip then yes.
it will proxy it to pip
if it doesn’t work, let me know. You can try it out without a flow using something like 
metaflow environment resolve -r requirement.txt
and put 
--extra-index-url <your url>
in that file for example.
w
should work then (at least chatgpt says it would)
do i need to use this extensions repo to use 
pip_sources
or its in Metaflow mainline too?
d
you need the extension.
I need to push out an update. if you give me to tonight I can push out some fixes. It should work as is but i haven’t updated it to the latest version we use internally.
the extension is on pypi as well so all you need to do is install it and it should just work.
👍 1
still haven’t pushed an update. hoping to do that tomorrow.
✅ 1
alrighty, v0.1.1 is out. You can check it out and let me know if you have any feedback. It should hopefully do what you need/want and more 🙂
w
sorry for the delay.. just got around to trying this -- it doesnt seem to be using my 
pip_sources
😕 do i need to change 
--environment=conda
? how do i make it use the requirements.txt file?
d
what is in your requirements.txt file ? You mention pip_sources as well which is not in requirements.txt
I have a few updates to that too. I am making another release internally so will do it externally shortly as well.
w
i dont have a requirements.txt file.. i just put the packages under 
libraries
and put my internal codeartifact url under 
pip_sources
but it's saying it cant find my internal package. I tried installing the package with pip and it works (it did add a bunch of stuff to my 
~/.config/pip/pip.conf
but i figure i could pass the same thing to 
METAFLOW_CONDA_DEFAULT_PIP_SOURCE
d
you can try 
METAFLOW_CONDA_DEFAULT_PIP_SOURCE
. but pip_sources should also work. Could you run with METAFLOW_DEBUG_CONDA=1 ?
w
i dont think i can use 
pip_sources
because the generated index url has the username/password after i do "aws codeartifact login"
d
ah.
w
so this is what i have in the decorator now:
Copy code
@project(name="nissan_test")
@conda_base(
    libraries={
        "saitama": "0.1.0",
        "pytorch::pytorch": "1.12.1",
    },
    python="3.10",
)
and i have set 
METAFLOW_CONDA_DEFAULT_PIP_SOURCE
Copy code
s/hp/bf0s0t_x36xb56y4b4qg_xwc2sdr34/T/tmpubiqi7iw/prefix', '--dry-run', '--no-channel-priority', 'python==3.10', 'requests==>=2.21.0', 'boto3==>=1.14.0', 'saitama==0.1.0', 'pytorch::pytorch==1.12.1']
Pretty-printed STDOUT:
Could not solve for environment specs
The following package could not be installed
└─ saitama 0.1.0  does not exist (perhaps a typo or a missing channel).
{
    "success": false
}
d
is saitama a conda package?
w
no, its in my internal codeartifact pypi repo
d
ah, it should go in the 
pip_packages
section then
i fixed a few things around the mixed mode so if that doesn’t work (yet), you can also try:
Copy code
@pip(packages={"pytorch": "1.21.1", "saitama": "0.1.0"}, sources=[...])
w
k making progress.. now its complaining i dont have a resolver
so whats the diff between that and this:
Copy code
@project(name="nissan_test")
@conda_base(
    libraries={
        "pytorch::pytorch": "1.12.1",
    },
    pip_packages={
        "saitama": "0.1.0",
    },
    python="3.10",
)
d
in the mixed mode you need conda-lock.
you need version 2.0.0
For something like though, I would recommend the pure pip mode — it is more stable.
I just opened another PR against conda-lock for eg 🙂
w
ok so this is what i have:
Copy code
from metaflow import FlowSpec, step, project, conda_base, pip, pip_base

@project(name="nissan_test")
@pip_base(packages={"pytorch": "1.21.1", "saitama": "0.1.0"})
class LinearFlow(FlowSpec):

    @step
    def start(self):
        self.my_var = 'hello world'
        self.next(self.a)

    @step
    def a(self):
        print('the data artifact is: %s' % self.my_var)
        self.next(self.end)

    @step
    def end(self):
        print('the data artifact is still: %s' % self.my_var)

if __name__ == '__main__':
    LinearFlow()
and i ran:
Copy code
METAFLOW_CONDA_DEPENDENCY_RESOLVER=conda METAFLOW_DEBUG_CONDA=1 python flows/test.py --environment=conda run
there's an error about:
Copy code
subprocess.CalledProcessError: Command '['/Users/pownissa/anaconda3/bin/conda', 'info', '--json']' returned non-zero exit status 1.
which is weird because i tried running "conda info --json" and it works
Copy code
File "/Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 3276, in _call_conda
    err = [output["err"]]  # type: List[str]
KeyError: 'err'
shouldnt it be looking for "error"?
Copy code
3275                 if isinstance(output, dict) and "error" in output:
3276                     err = [output["err"]]  # type: List[str]
3277                 else:
3278                     err = [output]  # type: List[str]
d
And it’s the same conda in both cases?
The error processing thing is only if it returns a non zero status iirc so we have to solve the other problem first.
w
ok its not the same conda.. the one its picking up is busted
d
It uses which to get it.
I think anyways
w
its weird because it's a shell fn:
Copy code
conda () {
	\local cmd="${1-__missing__}"
	case "$cmd" in
		(activate | deactivate) __conda_activate "$@" ;;
		(install | update | upgrade | remove | uninstall) __conda_exe "$@" || \return
			__conda_reactivate ;;
		(*) __conda_exe "$@" ;;
	esac
}
the one that works is the shell fn
the busted one is in my path
anyway i removed the busted one and am trying again 🤞
Copy code
conda.CondaMultiError: EnforceUnusedAdapter called with url <https://conda.anaconda.org/conda-forge/noarch/setuptools-67.7.2-pyhd8ed1ab_0.conda>
This command is using a remote connection in offline mode.

EnforceUnusedAdapter called with url <https://conda.anaconda.org/conda-forge/noarch/pip-23.1.2-pyhd8ed1ab_0.conda>
This command is using a remote connection in offline mode.

EnforceUnusedAdapter called with url <https://conda.anaconda.org/conda-forge/osx-64/libsqlite-3.42.0-h58db7d2_0.conda>
This command is using a remote connection in offline mode.

EnforceUnusedAdapter called with url <https://conda.anaconda.org/conda-forge/osx-64/openssl-3.1.1-h8a1eda9_1.conda>
This command is using a remote connection in offline mode.
idk if it matters but im using mamba
is it supposed to be using offline mode?
Copy code
debug[conda /Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:3266]: Conda call: ['/Users/pownissa/mambaforge/bin/conda', 'create', '--yes', '--quiet', '--offline', '--no-deps', '--name', 'metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_39c31592d3551aed1483bd9729e59a3009fa3b20', '--file', '/var/folders/hp/bf0s0t_x36xb56y4b4qg_xwc2sdr34/T/tmp8xuogon4']
No STDOUT
alright i made it further by installing micromamba.. but now it's failing on:
Copy code
File "/Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py", line 1299, in _transmute
    pkg_spec.add_local_file(dst_format, dst_file, transmuted=True)
  File "/Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 378, in add_local_file
    added_hash = pkg_hash or self._hash_pkg(local_path)
  File "/Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/env_descr.py", line 502, in _hash_pkg
    with open(path, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/pownissa/mambaforge/pkgs/ca-certificates-2023.5.7-h8857fd0_0.tar.bz2'
idk what this file is for
d
so conda/mamba are borked in installing (see here: https://github.com/conda/conda/issues/11775).
for your other error, could you re-run with METAFLOW_DEBUG_CONDA=1 and send me the output? This is when it is trying to convert a .tar.bz2 file into .conda and somehow thinks that is where the .tar.bz2 is but somehow it isn’t.
w
Untitled
Untitled
i dont see that file:
Copy code
ls /Users/pownissa/micromamba/pkgs/ | grep ca-cer
ca-certificates-2023.5.7-h8857fd0.tar.bz2
ca-certificates-2023.5.7-h8857fd0_0
ca-certificates-2023.5.7-h8857fd0_0.conda
d
what do you know. It’s another bug in micromamba 🙂
you seem to be requesting .tar.bz2 files so metaflow will convert the file from .conda to .tar.bz2 but, for some obscure reason, micromamba doesn’t name the file right.
you can see that it named it 
ca-certificates-2023.5.7-h8857fd0.tar.bz2
instead of 
ca-certificates-2023.5.7-h8857fd0_0.tar.bz2
cph (conda-package-handling) does it right.
micromamba is generally faster so I have been using that but clearly there is some issue with it.
even more surprising, if it goes from .tar.bz2 to .conda, it names the files correctly which is why I didn’t catch it (I prefer .conda packages)
w
i dont know where im requesting tar.bz2?
d
I’ll update it to use cph in this case (and not micromamba) and report the bug to micromamba.
there is a flag for that. let me see.
in the config, are you setting 
METAFLOW_CONDA_PREFERRED_FORMAT
?
there may be a bug too. I personally set it to 
.conda
so that’s how I have mostly been testing it
w
i dont see METAFLOW_CONDA_PREFERRED_FORMAT set in my metaflow configs
should i set METAFLOW_CONDA_PREFERRED_FORMAT=.conda?
d
let’s try that. I’ll see if there is a bug. Am going to make a release with a bunch of other things so will incorporate the micromamba fix issue and see if there is an issue with this too.
w
it worked 😮
d
ok, and I found the bug 🙂
I’ll be including it in the next fix.
well yes, it is supposed to work 🙂 . Doesn’t always but it’s usually not too far 🙂.
w
Is this pip_base thing supposed use the settings in pip.conf? I don't really wanna have to set 
METAFLOW_CONDA_DEFAULT_PIP_SOURCE
d
it uses pip under the hood but it’s not your system pip (it’s a pip inside a conda environment). I’ll have to check on that. I don’t know if that pip reads the one from pip conf. If it doesn’t, I should be able to make it work still and get it to read it. If you run in debug mode, you should see which pip it is using under the hood.
w
i'm going crazy. i just tried running the flow again today and it's complaining about:
Copy code
Step(s): ['start', 'a', 'end'], Error: URL '<https://amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/johnny5/simple/bunch/1.0.1/bunch-1.0.1.zip#sha256=3c554816ba260634b55c1edf8de6467d8122dbd18ec20f323e181869bc48ac93>' is not a supported format (['.whl', '.tar.gz'])
d
Let me push our the latest update. I fixed that internally. I’ll do this tonight and ping you.
👍 1
sorry for the delay — v0.2.0 is out now and should hopefully fix all your issues. If you run into something else, let me know.
👀 1
w
seems like authentication isnt working anymore.. did you change something related to that?
Copy code
cat protobuf3-to-dict-0.1.5.tar.gz
Unauthenticated: request did not include an Authorization header. Please provide your credentials.
i have METAFLOW_CONDA_DEFAULT_PIP_SOURCE set with the credentials
d
I don’t think so. You can run wiht 
METAFLOW_DEBUG_CONDA=1
to see what it is doing exactly but I don’t think I changed anything there.
but clearly something changed 🙂
w
i'm just gonna delete everything under 
~/micromamba/pkgs/pip
maybe i ran it once without authentication and the packages are screwed up?
d
that’s where I downllad the local packages yes but I don’t think it’ll matter in terms of where packages are getting fetched from.
but worth a try.
although one thing I am thinking of which I don’t think I handle is downloading the actual packages
the auth is probably used to get to the pip listing and figure out what you need but then it may not be used to actually download the packages.
w
should stop creating the package with that text in it if the authorization header is missing
d
hum. That part I am not sure about.
where do you usually specify this auth?
the easiest may be for me to just read it from there.
w
i usually set it with 
POETRY_HTTP_BASIC_ARTIFACT_USERNAME
and 
POETRY_HTTP_BASIC_ARTIFACT_PASSWORD
(because i'm using poetry). but i also have it set in 
~/.config/pip/pip.conf
and 
METAFLOW_CONDA_DEFAULT_PIP_SOURCE
because i'm not sure which one you look at
so my script sets it in 3 places now 🙃
d
lol. ok. And ar you usinga pure pip environment or a combo one?
under the hood, for pure environments, it uses pip and otherwise it uses conda-lock which internally uses poetry (but their own packaged version of it).
w
im just using 
pip_base
(so pure pip?)
d
yes — pure pip
w
tried it again and still failing (after clearing out the download dir:
Copy code
/pip/antlr4-python3-runtime-4.9.3.tar.gz']
debug[conda /Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:3875]: Binary call: ['/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/bin/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--progress-bar', 'off', '-w', '/var/folders/hp/bf0s0t_x36xb56y4b4qg_xwc2sdr34/T/tmpqqm_02i_/build_5', '/Users/pownissa/micromamba/pkgs/pip/protobuf3-to-dict-0.1.5.tar.gz']
Pretty-printed STDOUT:
Processing /Users/pownissa/micromamba/pkgs/pip/bunch-1.0.1.zip

Pretty-printed STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/commands/wheel.py", line 147, in run
    requirement_set = resolver.resolve(reqs, check_supported_wheels=True)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 453, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
    super().__init__(
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 587, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 176, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 240, in unpack_file
    unzip_file(filename, location, flatten=not filename.endswith(".whl"))
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 115, in unzip_file
    zip = zipfile.ZipFile(zipfp, allowZip64=True)
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/zipfile.py", line 1267, in __init__
    self._RealGetContents()
  File "/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/lib/python3.10/zipfile.py", line 1334, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
d
I am somewhat surprised pip isn’t reading the config.
ah well that’s further
it’s trying to build a wheel from the source only package
w
idk why it's picking the .tar.gz package.. there's a .whl available
d
it should pick the wheel if it is available — at least if pip does the right thing.
what is the env you are trying to resolve?
you can also do 
METAFLOW_DEBUG_CONDA
, it will give you the full output of the pip resolution.
w
nvm i checked, this one doesnt have a .whl
d
ah 🙂
ok, so that makes sense.
what doesn’t make sense is why it can’t build it
w
its because the file is a text file
it literally says that "missing header" thing
d
oh 🙂
w
so somehow the credentials are messed up and its writing the error into the file
d
lol — so it “downloaded” the file and then it is complaining about it. Well that’s stupid. It should have failed on the download but ok at least that explains it.
so just to confirm 
/Users/pownissa/micromamba/pkgs/pip/protobuf3-to-dict-0.1.5.tar.gz
is effectively a text file with the error in it right?
w
yup
Copy code
file /Users/pownissa/micromamba/pkgs/pip/protobuf3-to-dict-0.1.5.tar.gz
/Users/pownissa/micromamba/pkgs/pip/protobuf3-to-dict-0.1.5.tar.gz: ASCII text, with no line terminators
d
ok, well ok, so I see at least one issue with my code:
Copy code
with open(local_path, "wb") as f:
                    with session.get(pkg_spec.url, stream=True) as r:
                        for chunk in r.iter_content(chunk_size=None):
                            base_hash.update(chunk)
                            f.write(chunk)
which doesn’t check the return code of 
r
(I think I assumed it would fail loudly if it didn’t get a response. I’ll fix that but that won’t solve your immediate config problem so let’s see if we can figure that one out too. So you see that it says something like this:
Copy code
/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/bin/python', '-m', 'pip'
this is the pip it is using. Could you see if it is reading the config file you have?
ah, wiat, I think I get why it’s not working with the auth. I’m probably downloading the package based on the url.
w
Copy code
Step(s): ['start', 'a', 'end'], Error: Binary command for Conda '['/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/bin/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--progress-bar', 'off', '-w', '/var/folders/hp/bf0s0t_x36xb56y4b4qg_xwc2sdr34/T/tmpqqm_02i_/build_1', '/Users/pownissa/micromamba/pkgs/pip/bunch-1.0.1.zip']' returned error (2); see pretty-printed error above
d
because it’s a general download (for both conda and pip packages)
thinking.
w
it was working yesterday though? (and last week)
d
ok, let’s confirm this first. Could you: • clean up everything • resolve but with debug mode • check for a line that says something like download XXX to XXX for that package that it is trying to build
I did change something around how environments are built
w
can you clarify "clean up everything". which folders?
d
it used to be that i would use pip directly but pip doesn’t actually do cross version properly (even if you tell it “solve for python 3.8, if you are running on python 3.10, it won’t do that).
I mean just what you did: • remove the micromamba/pkgs/pip/ thing • remove any condav2.cnd file
basically to get it to download the tar ball again.
MY current suspicion is that since I use that code I pointed to earlier to download the package, that code isn’t using any authentication.
but I am not 100% sure.
w
condav2.cnd file
where do i find this file?
d
you should have a 
.metaflow
directory in your current directory (or along its parent path). The file should be in there.
w
ok.. i just deleted the entire 
.metaflow
dir. i see it using the credentials now
ah damn it still failed
Untitled
Untitled
i cant tell if its using the credentials to download though.. i only see it printed once (search for REDACTED in the snippet)
shouldnt this url also have the credentials in it:
Copy code
debug[conda /Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2064]: Need to install {'download_info': {'url': '<https://amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/johnny5/simple/ether/0.1.1/ether-0.1.1-py3-none-any.whl>', 'archive_info': {'hash': 'sha256=911982a95f56856d9c212cc26e055f778312c889a069bad713660dfda9aecfca', 'hashes': {'sha256': '911982a95f56856d9c212cc26e055f778312c889a069bad713660dfda9aecfca'}}}}
d
ok, but this is what I suspected.
let me confirm by walking through the log
ya, so this is what is happening: • it is using the credentials to query PIP to figure out what to install • pip returns stuff like this:
Copy code
debug[conda /Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:2064]: Need to install {'download_info': {'url': '<https://amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/johnny5/simple/bunch/1.0.1/bunch-1.0.1.zip>', 'archive_info': {'hash': 'sha256=3c554816ba260634b55c1edf8de6467d8122dbd18ec20f323e181869bc48ac93', 'hashes': {'sha256': '3c554816ba260634b55c1edf8de6467d8122dbd18ec20f323e181869bc48ac93'}}}}
which, as you noted, does not include the credentials (it’s probably safer since otherwise they would leak all over the place) • we then download the package ourself because we want to build a wheel out of it:
Copy code
debug[conda /Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1530]: bunch-1.0.1 -> preferred .zip @ web
followed by:
Copy code
Downloading 7(web) + 0(cache) packages for arch osx-64 ...debug[conda /Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/lib/python3.10/site-packages/metaflow_extensions/netflix_ext/plugins/conda/conda.py:1338]: bunch-1.0.1 -> download <https://amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/johnny5/simple/bunch/1.0.1/bunch-1.0.1.zip> to /Users/pownissa/micromamba/pkgs/pip/bunch-1.0.1.zip
This then fails because it’s not a zip file beacuse it didn’t have authorization.
w
i think it would be easiest to read the credentials from the 
pip.conf
so we dont need to pollute the urls
d
what I somehow need to do is for that download pass the proper header (since the URL already lost the information)
yes
that would work too.
I wonder if conda also has this.
to connect to this, you just need username/password right?
w
yea.. its of the form https//{user}{password}@{url}
d
do you foresee a case where you would need different logins for different sources?
w
yeah.. but that's handled already in pip.conf.. you can login to multiple repos
d
and for conda repos? Do you have that use case (sorry — trying to make sure I come up with a general enough solution — we don’t have this case internally thus my rather idiotic questions)
w
we arent using conda repos so idk about that
techincally we also arent using multiple pip repos because we are using the codeartifact chaining system. but i could see someone else using it
d
could you provide a redacted version of your pip.conf to see how you structure your multiple repos’s auth? Also, could you confirm if the pip I pointed to actually reads it. So something like:
Copy code
/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/bin/python -m pip config list
to see if it actually sees your config?
what I am thinking is that I would add an adapter to the session (the requests session) that would properly add auth tokens read from the config file so it should be able to handle different sources and ptoentially conda stuff if we need to read a config from there too.
w
so my pip.conf looks like this (auto populated using "aws codeartifact login"). i'm only using one repo:
Copy code
[global]
index-url = <https://REDACTED@amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/saitama/simple/>
but i believe you can also specify others using "extra-index-url" like so:
Copy code
[global]
index-url = <https://REDACTED@amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/saitama/simple/>

trusted-host = <http://download.zope.org|download.zope.org>
               <http://pypi.org|pypi.org>
               secondary.extra.host
extra-index-url= <https://pypi.org/simple>
                 <http://secondary.extra.host/simple>
d
okie. and if you do the 
config list
thing, does it show that config?
w
Copy code
✗ which pip
/Users/pownissa/Library/Caches/pypoetry/virtualenvs/saitama-2InH2OdX-py3.10/bin/pip

✗ pip config list
global.extra-index-url='<https://pypi.org/simple>\n<http://secondary.extra.host/simple>'
global.index-url='<https://REDACTED@amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/saitama/simple/>'
global.trusted-host='<http://download.zope.org|download.zope.org>\<http://npypi.org|npypi.org>\nsecondary.extra.host'
seems so
d
w
yup that worked:
Copy code
/Users/pownissa/micromamba/envs/metaflow_builder_bcaa5c0f188333ea6204cf10eaed8c900354c27d_0a72266ccb5a2bd3c53d72b276a5031970e91d94/bin/python -m pip config list
global.index-url='<https://REDACTED@amazon-149122183214.d.codeartifact.us-west-2.amazonaws.com/pypi/johnny5/simple/>'
d
excellent — ok, let me try to work on that shortly and push out a fix that will read the value from there. I suspect, btw, that even without specifying METAFLOW_CONDA_PIP_DEFAULT_SOURCE, it should get to the same point (ie: the solving should work sicne it does seem to read the config file). It won’t download it properly though which is what I will fix.
👍 1
I haven’t finished this — I think I figured out a way but haven’t had time to finish yet. I’ll ping you when it’s done (hopefully tomorrow).
🤞 1
w
lmk if you need some help
d
okie — it’s fresh off the press in fix/auth branch. I haven’t had time to test it yet but the change is fairly straightforward. I would greatly appreciate it if you could try it out and let me know if it works for you. I made two small other fixes (it’s in the commit message).
w
tried it out. couple notes: 1. i still need to set METAFLOW_CONDA_DEFAULT_PIP_SOURCE otherwise it doesn't even find my internal package 2. am still getting a bunch of "401 unauthorized" errors. 3. files under 
~/micromamba/pkgs/pip
are now 0B and empty (instead of before where it was a text file) so seems like it's not picking up the settings in pip.conf
im looking at the diff.. do i need to specify 
CONDA_SRCS_AUTH_INFO
?
d
let me get back to you but technically no. it should pick up the stuff in pip conf for the auth part. it won’t pick it up for 1 so that is expected. The errors should now come back up.
w
separate question: is there a directory that has all the dependencies already so i can just zip it up for deployment purposes?
well, this is weird. i set 
CONDA_SRCS_AUTH_INFO
and it managed to download some stuff but a bunch of others are still 0B which is causing the failures
d
I’ll add some debugging messages on it to see what is happening. The auth should be gleaned from the pip conf and the one you add.
w
the one i added is the same one in pip.conf so i can get rid of it (hopefully)
d
I don’t have a good local way of testing just yet (need to fake it I guess)
But will try to look in a bit. The logic is pretty simple: match hostname and if so add basic auth headers.
I’ll add some debug to print thr auth info structure so we can see what’s happening.
w
ok i found one problem. in this code:
Copy code
for url in urls:
        up = urlparse(url)
url was single quoted.. so the urlparse was failing. so i got rid of the quotes using:
Copy code
if url.startswith("'") and url.endswith("'"):
            url = url[1:-1]
i got it to work after disabling the cache. not sure how to clear it properly
d
Ah. That’s interesting for the quote. Ill do more sanitization then.
w
for wahtever reason, even if i delete the files from s3, metaflow still thinks they exist in the cache
d
Ah. You mean there are bad packages in the cache. Yes that makes sense.
You have to do two things to clear it all. Remove the .cnd files and clear everything in packages and envs.
In thr cache.
w
packages and envs under s3?
d
I am not in front of a computer but can point you to it when I get back.
And yes.
There are two folders typically. One called envs and one called packages.
Both need to be cleared out.
w
what if i just deleted the entire conda_env folder?
d
And also locally there is a .cnd file somewhere.
Thst should be fine.
In debug mode it prints the conda root it is going to look in.
But definitely clean out the local .cnd file too else it will look for things there too.
👍 1
Am at my daughter’s pre-k graduation. Should be back soon and will point to exact files. Sorry for the delay.
w
it's all good, i got it working
now i'm trying to figure out how to package up this micromamba environment for deploying -- i think i need something similar to conda-pack because i doubt just zipping up the env will work
d
what do you mean?
metaflow will take care of doing all that for you as long as you have the cache enabled (and if you don’t, it’ll work too but just be a bit slower).
So, to summarize the current issues: • need to sanitize the URLs (that should be an easy fix) • it’s still not by default looking at pip.conf option to download the packages (if pip sees the options, it should just work since I am not doing except calling pip) • does it see the pip.conf values for the auth extraction?
w
only the sanitization issue remains. it's using the pip.conf options. and i dont need to set 
METAFLOW_CONDA_DEFAULT_PIP_SOURCE
or 
METAFLOW_CONDA_SRCS_AUTH_INFO
anymore
metaflow will take care of doing all that for you as long as you have the cache enabled (and if you don’t, it’ll work too but just be a bit slower).
i want to package up my code/dependencies to use in an inference image, so i dont need to bake the code into the image (which is what i'm doing right now). so basically it should work just like how it does in Batch, where there is a base image and it sucks down the code/dependencies. maybe i can just do it the same way that metaflow does it for Batch?
d
so to be clear, if you use the conda environment and you have S3 enabled (which you probably do if you are using batch), what will happen is: • environment is resolved before launching (that’s what this whole thing is) • packages are downloaded to your local machine and then uploaded to S3 • the description of the environment is created and also uploaded to S3 • when it runs on batch, it will: ◦ download micromamba if it doesn’t have it (you can also cache a distribution of micromamba if you want and it will use that) ◦ download all packages cached on S3 (or if you don’t have cache, it will re-download from the web) ◦ create your environment ◦ launch within that. So, you should have no need to do anything else to get it to work. It should just work. At least that’s what we are doing internally (it’s not batch but similar). Our image is “generic” and the conda environment is created “on the fly” when the container launches.
ok for sanitization — I will do that (that’s an easy fix). Glad it works otherwsie, I was really surprised it wasn’t 🙂
a couple more points to clarify: • the cache for this version of conda cannot be the same as the one for the older conda (they are not the same path structure) • packages are located (from most preferred to least preferred): locally (in the conda’s pkg/ directory typically), from the cache, from the web. If you want to clean everything, you should clean all three of those locations. • “pointers” to where packages are are maintained locally in a .cnd file (condav2.cnd in OSS I think) which is in a 
.metaflow
directory in your cwd or along the path to the root as well as remotely in S3. If you want to fully remove things, you should remove both of those as well.
w
Is there a way to get the bootstrap command to do all of the above things? I can see the command in Step Functions but it's pretty gigantic
d
I’m sorry — I am not understanding. It currently does all this. It should work out of the box for sfn and batch.
w
Sorry I wasn't clear. I am creating an image to use in Sagemaker for online inference right now (ie in a Sagemaker endpoint), and I am doing the packaging myself. I want to reuse the Metaflow bootstrap method for that
d
aaaaah. OK, that makes more sense. OK, right, we have a hosting solution that does this internally. Let me DM you a piece of code that allows you to do this for inferencing. Or at least you should be able to adapt it. It’s totally do-able 🙂
13 Views
Open in Slack
PreviousNext
Powered by