https://outerbounds.com/ logo
Join Slack
Powered by
Hi all :sunny: Thanks for Metaflow: it's a pleasur...
# ask-metaflow
b
Hi all ☀️ Thanks for Metaflow: it's a pleasure to use! I am working on the combining 
zarr
data transformation with Metaflow's LinearFlow. All is going well except one key zarr function: when I try to consolidate all my metadata into a Metadata Store inside the flow, I get no error message but the metadata does not get consolidated (i.e. the Metadata Store that should be created does not get created). The 
zarr
function being called in the LinearFlow is: 
zarr.convenience.consolidate_metadata()
Have you stumbled upon such issues before? thanks for your insight ! (I have also created a Stackoverflow question for it here)
✅ 1
v
hi Laure meow wave are you running the flow locally or on `@kubernetes`/`@batch` ? I am not familiar with 
zarr
but if it creates a local file on a remote system, it wouldn't be stored automatically when the task finishes and the container disappears
b
Hi 🤗 thanks for your input ! I'm testing it out locally atm (and once this issue is fixed I will run on a kubernetes cluster)
v
I tried this flow using 
zarr
zarrflow.py
if I enable the 
consolidate_metadata
line, it writes something to the 
zarrdata
datastore because the file size grows, so it seems to be doing something 🙂
maybe it's modifying an existing store in your case too? 🤔
👍 1
b
hi ! thanks for trying it out 😄 what if you put it in a different step?
Hi again @victorious-lawyer-58417 I reworked your example to show you my current issue. The datastore is defined elsewhere and I want to create a flow to consolidate the metadata with a parameter (atm it is commented out to avoid complexity). What you will see if you run it 
python test_zarr_metaflow.py run
is that the metadata does not get consolidated (there is no creation of a 
.zmetadata
file in 
test_case
folder. I also had glitches where the file would appear then disappear: could there be an issue with the current of the Flow where the object gets created then deleted? Thanks a lot !
test_zarr_metaflow.py
v
ok, thanks, let me test this
🤗 1
b
Hi @victorious-lawyer-58417, sorry to ping you again on this but did your test reproduce the issue? 🙂 thanks !
v
sorry for the delay! Here's what happens in 
test_zarr_metaflow.py
: 1. The block of code below 
if __name__ == '__main__':
gets executed for every step 2. 
zarr.convenience.consolidate_metadata('test_case/test_metaflow')
actually creates 
.zmetadata
in the 
start
step correctly 3. but when the 
end
step executes, it re-initializes the directory, removing 
.zmetadata
If you move the code from the 
if
block to the 
start
step, it works correctly.
🙌 1
so you should be able create a flow that takes the location as a parameter and then consolidates it
b
amazing! I will be sure to try it out asap. Thanks for looking into it 🙂 (and sorry for my delayed reply, I was out of office for a while)
v
np! Let me know if it still doesn't work after trying it
b
it now works! (i got side tracked on other work for a few months) thanks for looking into it with me 🙂
👍 1
v
awesome, great to hear 🙂
6 Views
Open in Slack
PreviousNext
Powered by