Hello - really enjoying Metaflow so far but having an issue when trying to trigger an Argo Workflow with an InputFile parameter. Most likely I’m not using the API correctly but wanted to see anyone has any advice .. I’ve tried doing something like

ArgoEvent(name='event', url='<http://localhost:12000/metaflow-event>', payload = { 'config': p }).Publish()

I’ve tried sending p as • a local file path (argo fails with file not found) • the actual content (could not open file) • a cloud based path - not supported any advice is greatly appreciated

AFAIK

IncludeFile

is for including/versioning data from a local file alongside a flow when it's run or deployed, not changed dynamically as an input parameter – e.g. if you use

IncludeFile

and then run

argo-workflows create

the local file will be persisted by metaflow into the backend datastore, and later made available to the flows as a binary blob https://docs.metaflow.org/scaling/data#data-in-local-files where is the file you're trying to load located?

the file that I’m trying to pass in is currently local but could put it anywhere that would work. it’s interesting but this did work which made me think doing the same programmatically would work as well

python flow.py --environment=pypi argo-workflows trigger --config ../config.yml

is the intent that the

config.yml

should be included/frozen at the time of deployment, e.g. when you run

argo-workflows create

? If so, then I think it should work no problem If the goal is to dynamically load a file, e.g. from S3, then I don't believe

IncludeFile

supports that. In that case, you should likely just pass in a str parameter and load the file in the start step

yeah, was hoping to be able to pass in the file dynamically. can definitely capture a string argument instead but the file may be decently large for a command line parameter (currently at 5k)

the file path is that long? I'm not sure I'm following

oh sorry - the content is that long

ah gotcha, yeah you probably would want to pass in the path to the file if it's dynamic, and then in a step you can load the file content

We have sent cloud based s3 path and it works. What kind of error do you get?

we’re using gcp and when I provided a gcp path I got the following error

IncludeFile using a direct reference to a file in cloud storage is no longer supported. Contact the Metaflow team if you need this supported

Just pass it as a string path. You don't need to use IncludeFile in this case,

yeah but we need the contents of the file in the workflow which is running in Argo

ah ok! Why can't you pull it from GCP? It should be fast.

ok, found a way to do it after looking at the include file source .. it uploads the file and then passes information about the uploaded file to the event. be warned that a few pieces are hard-coded (flow name and data client to use GCP)

import sys, os, json

from metaflow.integrations import ArgoEvent
import metaflow.includefile as includefile

def trigger(config_file_path):
	if not os.path.exists(config_file_path):
		print(f"path: {config_file_path} does not exist")
		sys.exit(1)

	flow_name = 'SpecFlow'
	uploaded = includefile.CURRENT_UPLOADER.store(flow_name, config_file_path, True, 'utf-8', includefile.DATACLIENTS['gs'], print)
	config_file = includefile.IncludedFile(uploaded)
	config_desc = json.dumps(config_file.descriptor)
	e = ArgoEvent(name='event', url='<http://localhost:12000/metaflow-event>', payload = { 'config': config_desc })
	e.publish()

out of curiosity, what's the rationale behind this versus just having a string parameter with the cloud path and loading it in your code?

this way we can run it locally as well .. the use case we’re trying to cover is running a simulation based off a configuration that would be on your desktop

most cloud providers have filesystem-like abstractions for blob storage, so I don't see why you couldn't have the input path support both local files and remote files e.g. a simple wrapper that checks if the path is prefixed with

gs://

and then loading from your cloud, otherwise loading the yaml locally