Hi guys, this is my clickstream process , please give me feedback about what to change in the process: 1. Event api 2. Kinesis 3. Lambda ( boto for decode and create dynamodb) 4. Dynamo db for saving the event 5. S3 for the actual data 6. Docker( code , 7. airflow/argo/AWS Step Functions 8. Sagemaker took data from s3 (old data)+ dynamodb( new data) 9. S3 for results

makes sense. If you don't need to lookup new data in real-time, you could also replace the steps 3-5 with Kinesis Firehose that delivers data directly in S3

hi, some short question : 1."lookup new data in real-time" you mean to use this event for prediction ? 2.what I need to add/change in the process for a real time process ? 3.metaflow replace the sagemaker (in this process ) step. Is this correct?

1. If you need real time (<15mins latency or so) features coming from your event stream in predictions, then using DynamoDB is a good idea. Otherwise it may be possible to do without it. 2. For real-time predictions, you need some place to host your models. You can use Sagemaker Model Hosting or you can deploy an ordinary microservice. 3. You can use Metaflow in steps 7 and 8 above to train models and to precompute predictions, if that's an option. If you need real-time, you can load the model trained in Metaflow to a model hosting service.