Three expert algorithms, time-aware splits: Phase 2 gets smarter

Two of the quieter, deeper upgrades we've shipped this week change results in a very practical way. Your models will usually be more accurate, and they'll be more honest about how accurate they actually are on future data. Here's what changed and why it matters.

Three model families, one best answer

JITM.ai now trains three different families of gradient-boosted tree models side by side: XGBoost, LightGBM, and CatBoost. Think of these as three expert mathematicians who solve the same problem in slightly different ways. Each has strengths on certain shapes of data: XGBoost is the classic workhorse, LightGBM is unusually fast on wide tables with lots of columns, and CatBoost handles messy categorical fields (things like "city" or "product_category") with less fuss. Instead of betting everything on one algorithm, JITM.ai runs all three through its own search and combines the best from each into an ensemble. In practice, combining them smooths out the weirdness where one family happens to overfit a particular corner of the data.

Phase 2 cross-validation, not just a holdout

Previously, Phase 2 evaluated models on a single held-out slice of data. That's a fine first signal, but it can be lucky or unlucky depending on which rows happen to land in the holdout. Phase 2 now uses k-fold cross-validation: we split your training data into folds, train a model on all folds except one, and evaluate on the one left out. We rotate through every fold, which gives several independent estimates of real-world performance instead of one. Every fold model is kept and combined at inference time. This is where "fold bagging" comes from: each fold produces a slightly different model, and averaging their predictions tends to be more robust than any single one.

For ML practitioners: what we actually do

We use stratified k-fold for classification (every fold carries the same class balance) and plain k-fold for regression. When a grouping column is present (sessions, users, races), we switch to group-aware folds so the same group never appears in both train and validation. When a datetime axis is detected, we switch to forward-only time splits. All of this is selected automatically from the dataset rather than asked of the user.

Time-aware splits for time-aware data

Cross-validation is a problem when your data has a time order. If your dataset is daily sales, sensor readings, or races across a season, randomly shuffling rows into folds lets a model cheat: it sees the future during training and looks brilliant on the holdout, then fails in production. This is the single most common silent bug in real-world ML. JITM.ai now detects a time column automatically. When it finds one, validation switches to chronological mode: the holdout is always the most recent slice of data, and each fold trains on everything before a cutoff and validates on what comes right after. The model never sees any row more recent than what it's predicting.

Leakage checks stay on

Our leakage detection still runs on every model. If training results look suspiciously better than validation, we flag it as likely leakage (future-into-past, target-derived features, or ID leakage) without blocking the run, so you can decide what to do.

What you should do

Nothing. If your data has a time dimension, JITM.ai will notice. The improvements apply to every new model you train from now, and existing models keep serving predictions from their original training until you retrain them.