https://outerbounds.com/ logo
Join Slack
Powered by
Hi, have a question about local imports with batch...
# dev-metaflow
m
Hi, have a question about local imports with batch: If I have a file structure that looks like: 
flow_type_1
flow_1
flow_specific_utils
flow_type_2
flow_2
flow_specific_utils
utils
general_util
If I have a batch step I know I can import in functions from the 
flow_specific_utils
in 
flow_1
or 
flow_2
and if I have a non-batch step I can alter the python path to allow me to access functions in 
general_util
but is there any way to access these functions in 
general_util
from a batch step?
✅ 1
📓 1
s
Metaflow packages all subdirectories of the flow automatically, so it would work if you had 
flow_1
and 
flow_2
at the top level
u
@victorious-lawyer-58417 would a symlink work?
s
unfortunately we don't follow symlinks currently
m
So in short, there isn’t a way to have this file structure?
Because if flow 1 and flow 2 were actually 100 flows then that could get quite messy
s
right. Let me take a look at a few PRs that we have open that enable symlinks. Would you be open to adding a symlink to the top-level 
utils
from each 
flow_type_1
etc?
m
Yes, this would definitely be an improvement!
s
ok, I'll get back to you soon and let you know if it is feasible. Stay tuned!
m
@bulky-gpu-70936
s
this PR enables symlinks https://github.com/Netflix/metaflow/pull/958 It'll be included in the next release in a few days. I'll ping you when it's out 🙂
👀 1
🙌 1
the latest release of Metaflow, 2.5.2 supports symlinks in packaging 🎉 this allows you to structure projects like @melodic-train-1526 suggested above
Copy code
flow_type_1
    flow1.py
    flow_specific_utils
    common_utils -> ../common_utils
flow_type_2
    flow2.py
    flow_specific_utils
    common_utils -> ../common_utils
common_utils
     __init__.py
     some_module.py
Prior to this release, you couldn't run 
flow1.py
and 
flow2.py
remotely since 
common_utils
wasn't packaged but now it is! 📦 cc @silly-motorcycle-86975 @careful-dress-39510 @worried-flag-22881 who have asked similar questions about structuring projects in the past
🙌 6
👍 2
s
Thank you for the update. Will try it out!
👍 1
w
Awesome!
q
Amazing
m
Thanks @straight-shampoo-11124
👍 1
b
this is awesome!
yayfox 1
b
Out of curiosity I tried this on an ARM based Mac with a larger repo as a simlink, which failed during the creation of the container with the following error 
OSError: [Errno 62] Too many levels of symbolic links:
. It seems to be macOs specific problem for conda, if I interpret this issue correctly: https://github.com/conda/conda/issues/6514 That said, the very same flow with simlink works perfectly under Linux and I am super happy about it! Thank you :)
Open in Slack
PreviousNext
Powered by