I'm having this issue when adding `--with kubernet...
# ask-metaflowm
most-telephone-75272
11/21/2023, 6:52 PMI'm having this issue when adding
--with kubernetesmost-telephone-75272
11/21/2023, 6:53 PMthe flow looks like this with some custom decorators
Copy code
from metaflow import (
FlowSpec,
Parameter,
card,
catch,
conda,
conda_base,
kubernetes,
project,
step,
)
from src.flows.decorators import apt_install_java, poetry_install
from src.flows.utils import notebook
@project(name="dnn")
@conda_base(python="3.10", packages={"poetry": "1.7.1"})
class NotebookFlow(FlowSpec):
...
@step
def start(self):
...
@kubernetes()
@card(type="html")
@catch(var="execute_notebook_error")
@poetry_install()
@apt_install_java()
@step
def execute_notebook(self):
...
@step
def check_execute_notebook_error(self):
...
@step
def end(self):
...
if __name__ == "__main__":
NotebookFlow()most-telephone-75272
11/21/2023, 6:54 PMany idea what's going on? this is the command I'm using:
Copy code
USER=alvaro poetry run python notebook_flow.py --package-suffixes .json,.toml,.ipynb,.lock --environment conda run --notebook_path pricing/DNN/general/DS-737/ARPPU_Model.py --workflow_id "DS-733" --model_id "DS-740" --model_type general --with kubernetesmost-telephone-75272
11/21/2023, 6:55 PMit's failing on pypi bootstrap, but I'm not referencing it anywhere
most-telephone-75272
11/22/2023, 4:57 PMSolved using python 3.10.12 instead of 3.10, it was being converted into float 3.1 at some point
s
salmon-smartphone-62113
12/01/2023, 10:15 PMHi @most-telephone-75272! Would you be up for sharing your implementation of the custom
poetry_installm
most-telephone-75272
12/05/2023, 8:32 AMsure, here it is
Copy code
def poetry_install():
def decorator(function):
@wraps(function)
def wrapper(*args, **kwargs):
Path("/root/.pip").mkdir(exist_ok=True)
copy("pip.conf", "/root/.pip/pip.conf")
run(
[
f"{sys.exec_prefix}/bin/poetry",
"export",
"--without-hashes",
"-E",
"metaflow",
"-f",
"requirements.txt",
"-o",
"/tmp/requirements.txt",
]
).check_returncode()
run([sys.executable, "-m", "pip", "install", "-q", "-r", "/tmp/requirements.txt"]).check_returncode()
return function(*args, **kwargs)
return wrapper
return decoratormost-telephone-75272
12/05/2023, 8:33 AMbasically I couldn't find a way to install poetry because pip, conda, metaflow env... all becomes a mess of libraries location
most-telephone-75272
12/05/2023, 8:33 AMsimplest approach was to export and install with pip
s
salmon-smartphone-62113
12/21/2023, 9:29 PMThanks!
salmon-smartphone-62113
12/21/2023, 9:31 PMWe ended up just using a custom docker with all the poetry stuff installed in the image
12 Views