Multi-Source ETL Pipeline (Apache Airflow):

• Salesforce extraction:
  • REST API integration using simple-salesforce Python library
  • Objects: Lead, Contact, Account, Opportunity, Task, Event, Campaign
  • CDC via LastModifiedDate field for incremental extraction (daily sync)
  • Custom fields: Industry vertical, employee count, annual revenue, tech stack
• Marketo integration:
  • REST API bulk extract endpoints for activity logs
  • Activity types: Email sent/opened/clicked, web page visited, form filled, program membership
  • Polling frequency: Hourly for web activities, daily for email engagement
• Uberflip (content platform):
  • JavaScript tracking pixel data exported to S3, ingested via boto3
  • Metrics: Content views, average time-on-page, scroll depth, downloads, shares
  • Content tagging: Product category, buying stage (awareness/consideration/decision), topic
• Zendesk (support):
  • Zendesk API for ticket data (ticket ID, subject, description, status, priority, tags)
  • Ticket categorization: Pre-sales inquiry, technical question, pricing request, feature request
• ON24 (webinar platform):
  • Event API for registration, attendance, engagement data (polls, Q&A, resource clicks)
  • Engagement scoring: Time attended, questions asked, resources downloaded

Data Warehouse (Snowflake):

• Dimensional model:
  • Fact tables: fact_web_activity, fact_email_activity, fact_content_consumption, fact_support_tickets, fact_events
  • Dimension tables: dim_contact, dim_account, dim_content, dim_campaign, dim_date
  • Bridge table: bridge_contact_account for complex org hierarchies
• Aggregated feature tables:
  • agg_contact_engagement_7d, agg_contact_engagement_30d, agg_contact_engagement_90d
  • Pre-computed rolling windows for velocity metrics (engagement acceleration/deceleration)

Behavioral Feature Engineering (200+ features):

• Web engagement features:
  • Session counts (7d, 30d, 90d), page views, unique pages, avg session duration
  • High-intent page visits: pricing_page_views, product_comparison_views, demo_request_page_views, careers_page_views (hiring signal)
  • Content depth: documentation_views, technical_whitepaper_downloads, customer_case_studies_read
  • Recency: days_since_last_visit, visit_frequency_trend (increasing/stable/decreasing)
• Email engagement features:
  • Open rate, click-through rate, reply rate (strongest intent signal)
  • Engagement velocity: time_to_first_open, time_to_first_click
  • Campaign responsiveness: % of emails opened in last 10 sent
  • Engagement decay: Exponentially weighted moving average of opens/clicks
• Content consumption features:
  • Total content pieces consumed (30d, 90d), content depth score (avg time per asset)
  • Buying-stage progression: % of content in awareness/consideration/decision stages
  • Topic affinity: TF-IDF vectorization of consumed content titles/descriptions → cosine similarity to product categories
  • ROI calculator usage (binary + usage count)
• Support interaction features:
  • Pre-sales ticket count, technical inquiry count, pricing inquiry count
  • Ticket priority distribution (high/medium/low)
  • Response engagement: avg time to response, % of responses that get follow-up questions
• Event engagement features:
  • Webinar registration count, attendance rate (attended/registered), avg attendance duration
  • Active participation: poll_responses, questions_asked, resources_clicked_during_webinar
  • Event type: Product demo webinar, industry trends, customer success stories
• Multi-stakeholder features:
  • Contact count from same account, unique job function count (sales, IT, finance, procurement)
  • Executive involvement: binary flag if C-level or VP engaged
  • Buying committee size proxy: # of unique contacts engaged in last 30 days
• Temporal/velocity features:
  • Engagement acceleration: (activity_last_7d / activity_prev_7d) - 1
  • Bursty behavior detection: Coefficient of variation of daily activity
  • Days since first touch, days since last touch

Text Data Sources & Preprocessing:

• Collected text corpus:
  • Inbound emails to sales/support (n=450K messages)
  • Support ticket descriptions (n=85K tickets)
  • Webinar Q&A transcripts (n=12K questions)
  • Form submission text fields (n=180K responses)
• Preprocessing pipeline:
  • Text cleaning: Lowercasing, removing HTML/URLs, punctuation normalization
  • Tokenization: SpaCy tokenizer with custom enterprise tech vocabulary
  • Stopword removal: Extended NLTK stopwords + custom domain stopwords
  • Lemmatization: SpaCy lemmatizer for morphological normalization

Intent Classification Model:

• Approach: Supervised classification using labeled training data
  • Manual labeling: 15K emails labeled by sales ops into 5 intent categories:
    • Awareness (34%): General inquiries, information gathering
    • Consideration (28%): Feature questions, use-case discussions, competitive comparisons
    • Evaluation (18%): Pricing requests, demo scheduling, technical deep-dives
    • Procurement (12%): Budget/timeline discussions, contract terms, implementation planning
    • Other (8%): Support, feedback, non-sales
  • Inter-annotator agreement: Cohen's kappa 0.78 (substantial agreement)
• Model architecture: Fine-tuned BERT
  • Base model: bert-base-uncased (110M parameters)
  • Fine-tuning: Added classification head (768 → 256 → 5 with dropout 0.3)
  • Training: 5 epochs, learning rate 2e-5, batch size 16, Adam optimizer
  • Validation: 80/20 train/validation split, stratified by intent class
  • Performance: 87% accuracy, weighted F1 0.86
• Feature extraction from text:
  • Intent probability distribution: [P(awareness), P(consideration), P(evaluation), P(procurement), P(other)]
  • Max intent probability: Highest scoring intent class probability
  • Intent confidence: Entropy of probability distribution (lower = more confident)

Keyword/Pattern Extraction:

• Regex patterns for high-signal phrases:
  • Budget mentions: r'\b(budget|funding|approved|allocated)\b' (case-insensitive)
  • Timeline urgency: r'\b(Q[1-4]|ASAP|urgent|deadline|by [month])\b'
  • Procurement process: r'\b(RFP|RFI|proposal|contract|procurement|purchasing)\b'
  • Decision-maker involvement: r'\b(CIO|CTO|VP|Director|executive|C-level)\b'
  • Competitor mentions: r'\b(NetApp|Dell|EMC|Pure Storage|IBM)\b'
• Binary flags: has_budget_mention, has_timeline_mention, has_procurement_mention, has_competitor_mention

Sentiment Analysis:

• Model: VADER (Valence Aware Dictionary and sEntiment Reasoner) for business text
  • Compound sentiment score: -1 (most negative) to +1 (most positive)
  • Positive/neutral/negative probability distribution
  • Applied to email body, support ticket descriptions
• Features: email_sentiment_compound, ticket_sentiment_compound, avg_sentiment_last_5_interactions

• Firmographic enrichment (Clearbit API):
  • Company size (employee count bucketed: 1-50, 51-200, 201-1000, 1000+)
  • Industry/sector (NAICS codes mapped to high-level categories)
  • Annual revenue estimate (bucketed: <$10M, $10M-$100M, $100M-$1B, >$1B)
  • Location (country, state, metro area for geo-specific trends)
  • Company tags: Public/private, tech stack indicators
• Technographic intent data (Bombora, 6sense):
  • Topic-based intent scores: Companies researching "data storage", "hybrid cloud", "data management" across B2B web
  • Intent surge detection: Week-over-week % increase in research activity on relevant topics
  • Competitive intent: Research activity on competitor products
• Company news/events (Web scraping + APIs):
  • Funding announcements: Crunchbase API for Series A/B/C rounds, M&A activity
  • Leadership changes: New CIO/CTO appointments scraped from LinkedIn, company press releases
  • Business expansion: Office openings, headcount growth (LinkedIn employee count tracking)
  • Earnings/revenue milestones: Public company earnings calls mentioning IT investments
• Regulatory/compliance triggers:
  • Industry-specific compliance deadlines (GDPR, HIPAA, SOX, CCPA) driving data infrastructure upgrades
  • Binary flags: is_healthcare (HIPAA), is_financial_services (SOX), is_european (GDPR)
• Competitive intelligence:
  • Contract expiration estimates: Typical contract lengths (3-5 years) × last purchase date
  • Competitor partnership announcements: Web scraping for "NetApp partnership" type announcements

Target Variable Definition:

• Binary classification: y = 1 if contact/account converted to qualified opportunity within 90 days, else 0
• Qualified opportunity criteria: Stage ≥ "Discovery" in Salesforce, deal value ≥ $50K, valid next steps documented
• Prediction window: 90 days (aligned with quarterly sales planning cycles)

Training Dataset Construction:

• Observation grain: Contact-level (one row per contact per scoring date)
• Historical data: 24 months (Jan 2022 - Dec 2023)
• Sample size: 280K contact-observations (18K positive, 262K negative)
• Class imbalance: 6.4% positive rate (1:14.5 ratio), addressed via SMOTE + class weighting
• Feature count: 247 features after engineering + selection
• Train/validation/test split: 60%/20%/20% with temporal stratification (no data leakage)

Model Selection & Benchmarking:

• Evaluated multiple algorithms:
  • Logistic Regression (L2): AUC-ROC 0.76, Precision@20% 0.58
  • Random Forest: AUC-ROC 0.82, Precision@20% 0.71
  • XGBoost: AUC-ROC 0.88, Precision@20% 0.79 ✓ SELECTED
  • LightGBM: AUC-ROC 0.87, Precision@20% 0.77
  • Neural Network (3-layer): AUC-ROC 0.84, Precision@20% 0.74
• XGBoost selected for: (1) best performance, (2) feature importance interpretability, (3) fast inference

XGBoost Configuration & Tuning:

• Hyperparameter optimization via Optuna (Bayesian optimization):
  • Search space: learning_rate [0.01-0.3], max_depth [3-10], min_child_weight [1-10], subsample [0.5-1.0], colsample_bytree [0.5-1.0], gamma [0-5]
  • Optimization objective: Maximize AUC-PR (more appropriate for imbalanced data than AUC-ROC)
  • Cross-validation: 5-fold stratified CV with temporal blocking
  • Trials: 200 iterations with early stopping (no improvement after 30 trials)
• Best hyperparameters:
  • learning_rate: 0.05, max_depth: 7, min_child_weight: 5, subsample: 0.8, colsample_bytree: 0.7, gamma: 1.5
  • scale_pos_weight: 14.5 (inverse of class ratio to handle imbalance)
  • objective: binary:logistic, eval_metric: aucpr, n_estimators: 500 (with early stopping)

Handling Class Imbalance:

• SMOTE (Synthetic Minority Over-sampling): Generated synthetic positive examples to balance training data (not applied to validation/test)
• Class weighting: scale_pos_weight parameter gives higher loss penalty to false negatives
• Threshold optimization: Selected threshold 0.38 (vs default 0.5) to maximize F2-score (emphasizes recall)

• Primary metrics (held-out test set):
  • AUC-ROC: 0.88 (excellent discrimination ability)
  • AUC-PR: 0.74 (strong performance despite class imbalance)
  • Precision@20%: 79% (of top 20% scored contacts, 79% converted)
  • Recall@20%: 62% (top 20% captures 62% of all converters)
  • F1-score: 0.71 (at optimized threshold 0.38)
• Calibration analysis:
  • Brier score: 0.09 (well-calibrated probability estimates)
  • Calibration curve: Strong alignment between predicted probabilities and observed conversion rates
  • Implication: Scores can be interpreted as true probabilities (e.g., score 0.75 → 75% chance of conversion)
• Feature importance (SHAP analysis):
  • Top 10 features by mean |SHAP value|:
    • pricing_page_views_30d: 0.24 (strongest signal)
    • intent_procurement_probability: 0.19 (NLP-derived)
    • email_reply_rate: 0.17
    • multi_stakeholder_count: 0.15
    • demo_request_count: 0.14
    • ROI_calculator_usage: 0.13
    • engagement_acceleration_7d: 0.12
    • technographic_intent_surge: 0.11 (external signal)
    • executive_involvement: 0.10
    • support_ticket_pricing_inquiry: 0.09
  • Insight: High-intent behaviors (pricing, procurement language, demos) are strongest predictors

• Batch scoring pipeline (Airflow DAG):
  • Schedule: Daily at 3 AM UTC
  • Steps: extract_features → load_model → score_contacts → write_to_salesforce → generate_alerts
  • Scope: All active contacts (n≈45K) scored daily
  • Latency: <15 minutes for full scoring run
• Salesforce integration:
  • Custom fields: Propensity_Score__c (0-100), Propensity_Tier__c (Hot/Warm/Cold), Last_Scored_Date__c
  • Bulk API v2 for efficient writes (batches of 10K records)
  • Score history tracking: Append-only table for trend analysis
• Tiered scoring buckets:
  • Hot (score ≥75): Top 8% of contacts, immediate sales engagement recommended
  • Warm (50-74): Next 22%, prioritized outreach within 5 days
  • Cold (<50): Bottom 70%, remain in marketing nurture
• Automated alerting:
  • Score surge alerts: Email + Slack notification when contact score jumps >20 points in 7 days
  • New hot account alerts: Daily digest of contacts entering Hot tier
  • Sales rep assignment: Hot accounts auto-assigned to territory owner via Salesforce workflow
• Model versioning & governance (MLflow):
  • Model registry: All trained models logged with metadata, metrics, hyperparameters
  • A/B testing: 10% of contacts scored by challenger model, 90% by production
  • Champion-challenger evaluation: Monthly comparison, promote if challenger AUC-PR >2pp better
• Monitoring & drift detection:
  • Feature drift: KL divergence on feature distributions (alert if KL >0.2)
  • Prediction drift: PSI (Population Stability Index) on score distribution
  • Performance monitoring: Weekly conversion rate tracking for Hot tier (alert if <70%)
  • Delayed label feedback: 90-day lookback to compare predictions vs actuals
• Retraining cadence:
  • Scheduled: Quarterly retraining on rolling 24-month window
  • Triggered: On-demand if performance degrades >5% or drift detected
  • Validation: Hold-out test on most recent quarter, deploy only if metrics maintained

Quantified Business Outcomes (12-Month Period):

• Conversion rate improvement: Hot tier (score ≥75) converts at 42% vs 15% baseline (2.8x lift, p<0.001 via two-proportion z-test)
• Pipeline generation: $12.4M incremental qualified pipeline attributed to early identification of high-propensity accounts
• Sales cycle reduction: Average time-to-close decreased from 147 days to 97 days for AI-scored opportunities (34% reduction)
• Win rate improvement: 31% win rate on Hot-scored opportunities vs 23% on unscored (8pp improvement)
• Deal size impact: Average deal value $385K for Hot tier vs $326K baseline (18% larger, likely due to earlier engagement enabling broader solution scoping)
• Sales productivity: 40% effective capacity increase—same team focusing on high-propensity accounts achieved output equivalent to 1.4x headcount
• Forecast accuracy: Quarterly revenue forecast accuracy improved from 68% to 87% using propensity-weighted pipeline

Technical Achievements:

• Built multi-source data integration pipeline ingesting from 5 systems (Salesforce, Marketo, Uberflip, Zendesk, ON24) processing 2M+ events/month
• Engineered 247 behavioral features across web, email, content, support, events with temporal rolling windows (7d, 30d, 90d)
• Deployed BERT-based NLP model for intent classification achieving 87% accuracy on 5-class task (awareness/consideration/evaluation/procurement/other)
• Integrated external market signals (firmographic enrichment, technographic intent, company news/events) augmenting first-party behavioral data
• Developed XGBoost ensemble achieving AUC-ROC 0.88, AUC-PR 0.74, Precision@20% 79% on held-out test set
• Implemented production ML pipeline with daily batch scoring (45K contacts in <15min), Salesforce integration, automated alerting
• Established comprehensive MLOps:
  • Model versioning via MLflow with champion-challenger A/B testing
  • Multi-dimensional monitoring: feature drift (KL divergence), prediction drift (PSI), performance degradation
  • Quarterly automated retraining on rolling 24-month window with validation gates
  • Explainable AI via SHAP values for sales team trust and model debugging
• Achieved production SLA: 99.7% uptime, <15min scoring latency, 97% feature availability

Feature Importance & Model Insights:

• Top predictive signals (SHAP analysis):
  • Pricing page visits (SHAP 0.24): Single strongest predictor—viewing pricing indicates decision-stage interest
  • NLP procurement intent (SHAP 0.19): Language indicating budget/timeline discussions highly predictive
  • Email reply rate (SHAP 0.17): Active two-way engagement stronger signal than passive opens/clicks
  • Multi-stakeholder involvement (SHAP 0.15): Multiple personas from same account indicates buying committee formation
  • Demo requests (SHAP 0.14): Hands-on evaluation request signals serious consideration
• Interaction effects discovered:
  • Pricing page views + executive involvement: 2.1x lift vs pricing views alone
  • Support inquiries + technographic intent surge: 1.8x lift (dual signal validation)
  • Email engagement velocity + content depth: 1.6x lift (sustained, deep engagement)
• NLP model contribution:
  • Adding NLP features improved AUC-PR from 0.68 to 0.74 (9% relative gain)
  • Procurement intent classification particularly valuable: Precision@10% of 91% (9 of 10 contacts with procurement intent convert)
  • Competitor mentions in text correlated with 2.3x higher conversion (competitive displacement opportunities)
• External signal value:
  • Technographic intent surges: 3.2x conversion rate when B2B research activity spikes
  • Funding announcements: 1.9x conversion within 90 days of Series B+ round
  • New CIO/CTO appointments: 2.1x conversion within 180 days of leadership change

A/B Test Validation:

• Experimental design:
  • Randomized controlled trial: 50% of sales territories used propensity scores (treatment), 50% business-as-usual (control)
  • Duration: 6 months (sufficient for multiple full sales cycles)
  • Sample: 8,400 contacts in treatment, 8,200 in control (balanced)
• Primary outcome: Conversion rate
  • Treatment: 28.4% conversion rate
  • Control: 14.7% conversion rate
  • Lift: +13.7pp absolute, 93% relative (p<0.001 via two-proportion z-test)
• Secondary outcomes:
  • Time-to-close: Treatment 98 days vs Control 149 days (34% faster, p<0.001 via t-test)
  • Sales rep satisfaction: +28 NPS points for treatment group (p<0.01 via Mann-Whitney U test)
  • Pipeline value: $6.2M treatment vs $3.8M control (63% higher, p<0.01)

Production Performance Tracking (Post-Launch):

• Model stability: AUC-PR maintained 0.72-0.76 range over 12 months (no significant degradation)
• Calibration drift: Brier score 0.09-0.11 (well-calibrated throughout)
• Feature drift: KL divergence <0.15 for all features (below 0.2 alert threshold)
• Scoring latency: 99th percentile <18 minutes (well within 30min SLA)
• Data quality: 97.3% average feature availability (>95% target)