Hi, I've been able to access local

whl

file using the syntax

@pypi(
        python="3.11.5",
        packages={
            "/my/directory/path/mylibrary.whl": "",
        },
    )

using the

metaflow-netflixext

package. I'm having to have

mamba

installed though. Is there any way of configuring things so that

mamba

is not required? For example, setting an env variable to use

conda

instead?

of course

Ah, just googled. Thanks @glamorous-night-24898. Is that

METAFLOW_CONDA_DEPENDENCY_RESOLVER

🙂

i test primarily with mamba so if you face issues with conda/micromamba, let me know and I’ll take a look.

😞 oof

cool — let us know if you have more questions/concerns.

Here's another question @dry-beach-38304. So let's say that the

.whl

file has been updated. Is there a way to force update of the cached environment?

I am actually working on that. Right now not directly. It uses the path to determine if something changed. SO if the name or path changes, it will re-update it but it doesn’t pick it up as is. There was a similar question around using a local directory. I need to fix that but haven’t quite figured out how yet for everything (for a whl file it should be easier to just hash it and figure out if the hash changed).

Hi @dry-beach-38304 I seem to occasionally encounter the "file change update" problem. Is there a fix for that besides renaming? I had resorted to just trying to place everything within the flow file.

Sorry — this is still on my plate (I know I know it’s been a while). I am hoping to get a few of these minor annoyances out soon. I actually started on it uesterday :). Hopefully by this weekend. I can definitely take care of the wheel part (still thinking for directory).

Thanks! I'm wondering if it is possible or advisable to have "helper" methods defined in a FlowSpec derived class?

What do you mean?

Hi @dry-beach-38304, I just meant if it is ok to have a class method separate from a flow step as part of a FlowSpec subclass. for example

def helper(self, x, y):
   etc.

Alright — I am an idiot. A similar problem has come up but it’s not what you are asking (I am also fixing the similar problem). For what you are asking though, yes, there is a way rn to get it to work. You can do something like

python ./myflow.py --environment=conda environment resolve --force <step-name>

. That will re-resolve the environment even if it is already resolved. The related problem is about uploading updated versions of the wheel and that it seems it does already (albeit badly) So am working on that as well but the command above should help.

@dry-beach-38304 you are amazing. Thank you 🙏

does it work?

I haven't tested it, but I very much appreciate the efforts!

What were the issues in 2.11.7?

I was getting the following error in 2.11.7 when attempting to run Amazon AWS job with `@batch`` parameter

2024-03-29 08:36:00.520 [1195/train/8756 (pid 10349)] raise ParamValidationError(report=report.generate_report())
2024-03-29 08:36:00.520 [1195/train/8756 (pid 10349)] botocore.exceptions.ParamValidationError: Parameter validation failed:
2024-03-29 08:36:00.520 [1195/train/8756 (pid 10349)] Missing required parameter in containerProperties.logConfiguration: "logDriver"

@brainy-truck-72938 — this last error is not related to plugin but seems to be to this: https://github.com/Netflix/metaflow/pull/1777/files

yeah, https://github.com/Netflix/metaflow/pull/1777/files is the fix actually, because if you don't populate these values, we don't fill them -- which is the same behaviour as before...

Thanks @freezing-telephone-6958

actually, there's still a slight problem it seems, let me fix it

Thanks 🙂

https://github.com/Netflix/metaflow/pull/1786 is not in 2.11.8, yet to be merged 😕

Thanks @freezing-telephone-6958