Uplift models: S-learner baseline and causal forest
Trained on
Criteo's uplift benchmark,
13,979,592 rows from a real randomized ad-exposure incrementality test. Full writeup and
every gate:
github.com/Bauxitiego/uplift-modeling.
Only two of the four methods evaluated are published here. T-learner and X-learner lost to
the naive S-learner baseline on both targets tested - publishing them as ready-to-use
artifacts would imply they're worth deploying, which the results say they aren't. S-learner
(the baseline nothing but the causal forest beat) and CausalForest (the one method that did)
are the two ends of the actual finding.
Results these models produced
| method | visit (qini) | conversion (qini) |
|---|
| S-learner | 2938.00 | 337.96 |
| CausalForest | 2991.39 | 350.78 |
| random targeting | 188.69 | -7.78 |
3M training rows, 1M held out, same split for both methods. Causal forest beats the baseline
on both targets - modestly (+1.8% visit, +3.8% conversion), but real and consistent in
direction - at a real cost: 500-700s to train against 4s for the S-learner.
Loading a model
S-learner: plain XGBoost, saved in XGBoost's own native format. No pickle, nothing to
disclose about trust.
1from xgboost import XGBRegressor
2model = XGBRegressor()
3model.load_model("s_learner_visit/model.json")
4# tau_hat(x) = model.predict(x with treatment=1) - model.predict(x with treatment=0)
CausalForest: joblib, not a safer format - checked directly, econml's CausalForest has
Cython-based tree internals that skops cannot reconstruct (its Tree.__cinit__ needs
constructor arguments skops has no way to supply). This is the one model in this repo that
needs a trusted-source caveat on load. Only load causal_forest_*.joblib files from a source
you trust, same as any pickle-based artifact.
1import joblib
2cf = joblib.load("causal_forest_visit/causal_forest.joblib")
3tau_hat = cf.predict(X).flatten()
Each directory also has metadata.json with the exact train/test split size, seed, and
feature list the model was fit with.
Honest scope
Meta-learners (Kunzel et al. 2019) and causal forests (Athey & Wager 2019) are established
methods, not novel here. What's real: the meta-learners were implemented directly rather than
called from a pre-built uplift library, validated against synthetic ground truth before being
trusted on real data (where, by the fundamental problem of causal inference, no individual
ground truth is ever available), and the evaluation metric itself was verified correct with
dedicated tests before being relied on as the only judge.
License
Apache 2.0 for code and these model weights. The underlying Criteo data is not redistributed
here - it's already published at its source, linked above, under its own terms.