https://outerbounds.com/ logo
Join Slack
Powered by
Hi, More of a conceptual question, I guess, but c...
# ask-metaflow
m
Hi, More of a conceptual question, I guess, but can someone explain to me what the difference is between a 
DATASTORE
and 
DATATOOL
? For example, in the config there is 
METAFLOW_DATASTORE_SYSROOT_S3
and 
METAFLOW_DATATOOLS_SYSROOT_S3
, but it is not clear to me what is meant by the terms 
DATASTORE
and 
DATATOOL
. Thanks
✅ 1
v
DATASTORE
is the main place where Metaflow stores its artifacts, code etc. It can be local files or an object store like S3 in the cloud
DATATOOLS
refers to the 
metaflow.S3
library which you can use to interact with S3 specifically
m
ok, cool. This is roughly aligned with my initial understanding. However, I am confused why 
DATATOOLS
is referenced in 
METAFLOW_DATATOOLS_SYSROOT_S3
.
What does it mean in this context?
v
the default location that 
metaflow.S3
uses is derived from the datastore location (
METAFLOW_DATASTORE_SYSROOT_S3
) with 
data
suffix appended, hence the name (which could/should be more descriptive)
m
Sorry, I'm still a bit confused. Is this issue still valid? It seems to suggest that 
METAFLOW_DATATOOLS_SYSROOT_S3
is not used.
That would make sense to me and we should use 
DATATOOLS_SUFFIX
instead, (though I guess it should be 
DATASTORE_SUFFIX
?) but still I don't really understand what bits of config go under 
DATATOOLS
and what goes under 
DATASTORE
. On a conceptual level, I mean.
v
let me take a look at the issue..
m
cool, thanks
Looking at this it would seem the 
DATATOOLS
is indeed about configuring whatever client you use to interact with the 
DATASTORE
, while 
DATASTORE
is about the store itself. Then the two above variables are old and can be removed, in the case of 
METAFLOW_DATATOOLS_SYSROOT_S3
, or still a bit confusingly named, in the case of 
DATATOOLS_SUFFIX
. Is that a fair assessment?
v
in 95% of cases(*) 
DATASTORE
is independent from 
DATATOOLS
. 
DATATOOLS
path settings only affect the 
metaflow.S3
client, in particular 
metaflow.S3(run=self)
which uses the default path (*) exceptions: • 
IncludeFile
happens to use the 
DATATOOLS
path internally today, which may/should change in the future without any user-facing effects • Special variables 
DATATOOLS_CLIENT_PARAMS
and 
DATATOOLS_SESSION_VARS
affect S3 datastore behavior, but they need to be touched only in special cases
re: 
DATATOOLS
path variables. You have two ways to configure it: • If you want a fully custom S3 location, you can set 
DATATOOLS_S3ROOT
• If you want to use the main S3 location (set through 
DATASTORE_SYSROOT
) but just change the 
data
suffix, you can set 
DATATOOLS_SUFFIX
does this make sense?
m
Largely, I think so yes. Thanks. So you believe the above issue is no longer valid?
v
checking before I say anything... Stay tuned
yeah, it seems like an open issue still - sorry about it
feel free to suggest a PR if you a quick fix in mind! 🙏
m
cool, thanks for clarifying
👍 1
5 Views
Open in Slack
PreviousNext
Powered by