Hi, with the `@pypi_base` decorator I can point a ...
# ask-metaflow
w
Hi, with the
@pypi_base
decorator I can point a package to a local directory (which works), but subsequent changes to the package do not show up (the resolver seems to think the package is cached). Is there a way to make the decorator pick up future changes to the package, similar to how
develop=True
works?
d
hum. I don’t think there is in either bleeding edge or regular. You are using the extension one right?
It does cache things by design (ie: to give you something reproducible) but I could possible have it check if something changed.
and re-package it.
w
yes, we're using the extension one
d
let me think about it a bit. We can’t go the route of “use stuff in this directory” because that makes it not reproducible but I could make it check. In the meantime, one workaround is to force resolve the enviornment again.
w
i guess symlinking it won't work since the package is also in s3?
d
correct. The package gets built and uploaded to S3
to force resolve, you can do something like:
python ./myflow.py --environment=conda environment resolve --force <step name>
.
I have to check though — it may be too smart for itself and consider the package to be pre-existing — I need to check for this use case.
To be sure I understand, this is just a local folder with a setup.py (for example) and you are trying to build an env with this “package”.
w
Correct, and we want changes to the package to show up on subsequent flow runs. We're trying to test changes to a dependency locally without having to republish our package each time
d
ok, so you would be fine with a solution wher ethe package gets built and pushed to S3 on changes?
w
yeah.. technically you could put the code along with METAFLOW_CODE_URL and just cache the dependencies as usual (dependencies of the local package)
f
@dry-beach-38304, I tried the force resolve path, doesn't work. Code changes to the local package are not reflected in the flow
d
Let me take a look and try this out. I had not thought of this specific use case (it’s valid just hasn’t been thought about).
it’s a balance between figuring out that the package has changed and not doing it every single deploy.