Scalable Movie Recommendation System with Monitoring and Continuous Deployment
Built a scalable movie recommendation system for 1M+ users using Kafka for real-time data ingestion, MongoDB for persistent storage, and RESTful API microservices containerized with Docker and deployed via Kubernetes.
Automated model retraining and redeployment every 48 hours via Jenkins pipelines, using MLflow for experiment tracking and model versioning.
Monitored key system metrics using Prometheus and Grafana, including click-through rate (CTR), request latency, and anomaly alerts.
Conducted online A/B testing with Minikube for isolated environments, tracking CTR in real-time and testing for significance with Z-tests.
System Highlights:
NCF model enhanced with dropout, batch normalization, and learnable scaling.
Microservice architecture included data buffer, model service, monitoring agent, and rating drift detector.
CTR and rating drift logs were streamed to Kafka, enabling dynamic analytics dashboards.
Security and fairness audits identified rating manipulation and demographic bias in recommendations.
Outcomes:
Achieved statistically significant CTR uplift over baseline models.
Built a robust versioning and retraining loop for real-world deployment readiness.
Demonstrated full-stack ML Ops skills from data ingestion to metrics-based alerting.