https://outerbounds.com/ logo
Join Slack
Powered by
Hello. I am following chapter 4 of the book Effect...
# ask-metaflow
a
Hello. I am following chapter 4 of the book Effective DS infrastructure and it mentions if we have Qs to ask in this slack (not sure if this channel tho). I am configuring metaflow with aws from the start - following the book and I am asked for METAFLOW_SERVICE_URL but the book does not give instructions on the setup :(
just in case anyone else comes in with the same question. I followed this to create resources on AWS and get all the variables needed: https://github.com/outerbounds/metaflow-tools/tree/master/aws/cloudformation
c
Go to CloudFormation (search for it on the AWS console). Click on your formation. Click on the Outputs tab. Look for the key "ServiceUrl" with the description containing "METAFLOW_SERVICE_URL". Other useful information is also on the Outputs tab. Check the descriptions for Metaflow keys.
Be careful if you're new (like me). I set this up, and it automatically runs the EC2 instances. Even when you shut them down, they auto-boot due to the "Auto Scaling Group." I couldn't figure out how to turn off auto-scaling, so I had to delete the entire formation to be safe. It charged $10 to my credit card after 12 hours, but luckily I caught it before it got out of control. If you're using a personal account for learning, I suggest editing the CloudFormation template to use the cheapest instances across all services. These should work fine for "hello world" and toy examples.
a
yep i tried to lower the dmg haha. I better finish the book fast. 🤭 BTW, do you have a resource that tells step by step to set up metaflow service (and potentially UI) locally (edit: on mac), and a resource that does it on GCP (my most familiar cloud platform)? I have tried to set it up in the past multiple times but I always failed.
c
on their git they go through it for setting up locally https://github.com/Netflix/metaflow for their UI follow this https://github.com/Netflix/metaflow-ui it requires you to set up a service (https://github.com/Netflix/metaflow-service) I had to build the images on my personal mac to get the docker containers up and running on my work mac. used the docker compose development from the service git and changed the image names to be those images. make sure to make a metaflow config folder and yaml file 
mkdir -p ~/.metaflowconfig
Create 
config.json
{   "METAFLOW_DEFAULT_METADATA": "service",   "METAFLOW_SERVICE_URL": "<http://localhost:8080/>" }
cant remember right now if I changed the METAFLOW_DEFAULT_METADATA to service since I just go t it setup with S3. I'm limping through this so take my comments with a grain of salt
but yeah i can get the local UI setup this way and I do this all in my project venv (not sure if this is best practice I asked a question and waiting on response about that). at the moment I can see my metadata on S3 but the DAG still isnt working. honestly not sure what I'm doing wrong. best of luck. if I make a tutorial I will post it here.
a
🙏 thank you so much.
a
@average-engineer-25057 you can also use the sandbox if you wouldn’t like to deploy anything in your account - it’s available at outerbounds.com/sandbox and is totally free!
c
The sandbox is more for determining if you would like to use the MetaFlow setup right? Or is it meant to something that is actually mean to be used in production?
Open in Slack
PreviousNext
Powered by