• Built ETL pipelines using Python (pandas, PySpark for large datasets) and SQL to extract from Salesforce API, Stripe webhooks, and event streams, then loaded into a centralized feature store (PostgreSQL with TimescaleDB extension)
• Implemented point-in-time correctness using temporal joins to prevent data leakage—ensured all features reflect information available at prediction time
• Applied multiple imputation strategies:
  • MICE (Multivariate Imputation by Chained Equations) for continuous engagement metrics
  • Mode imputation with missingness indicators for categorical attributes
  • Forward-fill with decay for temporal features like last login date
  • Created explicit "missing" flags as features to capture information in the missingness itself
• Engineered 120+ features including:
  • RFM features (recency, frequency, monetary value) with multiple time windows (7d, 30d, 90d)
  • Behavioral trend indicators: engagement velocity, feature adoption curves, support ticket sentiment scores
  • Interaction terms (e.g., plan_tier × usage_rate) and ratio features (e.g., actual_usage / expected_usage)
  • Temporal aggregations: rolling means, standard deviations, percentile changes
• Implemented feature selection using mutual information, SHAP values, and recursive feature elimination to reduce to 45 core features

• Selected XGBoost after benchmarking against Random Forest, LightGBM, and Neural Networks—XGBoost showed best precision-recall tradeoff and feature interpretability
• Addressed class imbalance using:
  • Scale_pos_weight parameter set to negative/positive class ratio (~20:1)
  • Custom objective function with asymmetric loss (FP cost = 1, FN cost = 5 based on business value)
  • Threshold optimization via precision-recall curve analysis (selected threshold 0.35 vs default 0.5)
• Used stratified 5-fold cross-validation with temporal splitting (train on months 1-10, validate on month 11) to ensure no future information leakage
• Hyperparameter tuning with Optuna (Bayesian optimization):
  • Optimized learning_rate, max_depth, min_child_weight, subsample, colsample_bytree
  • Custom objective: maximize F1-score subject to precision ≥ 0.80 constraint
  • 100 trials with early stopping based on validation F1
• Evaluation metrics: AUC-PR (more informative than AUC-ROC for imbalanced data), F1-score, precision@k, recall@k, and custom business metric (expected value of intervention)

• Model versioning and tracking with MLflow: logged hyperparameters, metrics, artifacts, and model signatures for reproducibility
• Containerization using Docker with multi-stage builds (separate training and inference images) for portability and consistency across environments
• Orchestration via Apache Airflow:
  • Daily scoring DAG: data extraction → feature generation → model inference → result storage → notification
  • Weekly monitoring DAG: drift detection, performance evaluation, alert generation
  • Monthly retraining DAG: data extraction → preprocessing → model training → evaluation → conditional deployment
• Inference optimization:
  • Model serialization using joblib and ONNX for faster loading (reduced cold start from 3s to 200ms)
  • Batch prediction vectorization achieving <50ms p95 latency for single predictions
  • Feature caching layer using Redis for frequently accessed customer profiles
• Monitoring & observability:
  • Prediction drift tracking via PSI (Population Stability Index) on score distributions
  • Feature drift detection using KL divergence between training and production feature distributions
  • Performance degradation monitoring via delayed label feedback (tracked actual churn 30 days post-prediction)
  • Grafana dashboards displaying real-time inference volume, latency percentiles, error rates, and drift metrics
  • PagerDuty integration for critical alerts (drift exceeds threshold, performance drops, pipeline failures)
• A/B testing framework:
  • Randomized 50/50 split of predicted churners into treatment (received intervention) and control (no intervention)
  • Tracked conversion rates, retention lifts, and statistical significance using two-proportion z-test
  • Implemented guardrail metrics to ensure no negative impact on non-targeted customers

Business Impact:

• 15% relative reduction in churn rate (from 4.2% to 3.57% monthly churn) measured via randomized A/B test over 90-day period, statistically significant with p < 0.001
• 7-14 day prediction window providing actionable early warning—model predicts churn probability 1-2 weeks before actual churn event occurs
• Retention team efficiency improved by 40%—reduced outreach volume by 60% while maintaining higher intervention success rate
• ROI of 5.2:1 on retention campaigns: $520K in retained revenue per $100K spent on targeted interventions (calculated using LTV estimates)

Technical Achievements:

• Successfully handled severe class imbalance (1:23.5 ratio) through ensemble of techniques: weighted loss, threshold optimization, and cost-sensitive learning
• Built production-ready data pipelines processing 500K+ customer records daily with 99.8% uptime and <5 minute SLA for batch scoring
• Achieved <50ms p95 inference latency enabling potential real-time use cases (current implementation uses nightly batch scoring)
• Implemented automated retraining pipeline with monthly model refresh, automated validation, and conditional deployment based on performance thresholds
• Deployed comprehensive monitoring with drift detection, performance tracking, and automated alerting reducing MTTR from days to hours
• Feature engineering pipeline reduced dimensionality from 300+ raw features to 45 core features while maintaining 95% of predictive power (measured via ablation studies)
• Model interpretability via SHAP values enabled business stakeholders to understand top churn drivers: low engagement velocity (SHAP = 0.18), decreasing usage trends (0.15), and increased support tickets (0.12)