Head to head with H2O and AutoGluon on a brutally hard problem
We put JITM.ai up against two of the most respected open-source AutoML systems on the short-term direction of Bitcoin. Same data, same features, same hold-out. Here is how it went, including the honest caveats.
AI agents are about to manage the world's data: your calendar, your health records, a company's entire customer base. But there's a gap nobody is filling. When an agent needs to predict something, where does the model come from? Today the answer is ugly. You hire a data science team, stand up a cluster and wrangle dependencies for a week. Prediction is still a project, not a utility. We think it should be a utility, and that is what JITM.ai is: upload a dataset, train, and call a live prediction endpoint. Agents can do all three over MCP. The obvious question is whether a model built that simply can hold its own against the heavyweights. So we tested it.
The test: a problem that humbles everyone
We put JITM.ai up against H2O (the enterprise standard) and AutoGluon (Amazon's) on one of the hardest prediction problems there is: the short-term direction of Bitcoin, from live exchange data. Same features, same data, same target, and the same hold-out period, which none of the three ever saw during training.
The setup told the first story
Before a single number, the experience was the headline. H2O needed a Java runtime and a Docker cluster. AutoGluon needed its own Linux server, gigabytes of dependencies and a multi-hour install fight. JITM.ai was upload, train, predict: one interface, a few minutes. Same league of capability, a fraction of the friction.
The results
On the held-out future data, here is how the three ranked on signal strength (the information coefficient, which measures how well predictions track what actually happened; higher is better) and directional accuracy (how often each one called the direction of the move correctly).
| Platform | Signal strength (IC) | Directional accuracy |
|---|---|---|
| JITM.ai | 0.145 | 54.8% |
| AutoGluon | 0.106 | 53.9% |
| H2O | 0.097 | 54.4% |
JITM.ai came out on top, with the strongest signal of the three. The directional accuracy gaps are small; the signal gap is not. Under the hood it blended gradient-boosted trees with TabM, a modern neural network built for tabular data: the kind of model team you would normally need a specialist to assemble. Nobody configured that. The platform chose it.
The honest part
Bitcoin's 5 to 30 minute direction is close to random. Every model here is mining a faint edge, not printing money. That is the point: on a problem this unforgiving, JITM.ai matched and edged out the giants. It is a single dataset, so treat it as evidence rather than proof, but the signal is clear: JITM.ai plays in this league.
What has happened since
When we first published this, the next step was the rest of the prediction layer: ground-truth feedback and models that improve as reality arrives. That is now live as JITM Evergreen. Send back what actually happened, and when the world moves, a challenger has to beat your live model on fresh outcomes before it can take over, with your approval.
Got a hard problem?
We like working with teams on genuinely hard prediction problems. If you have one, get in touch at hello@jitm.ai or through our contact page. We would love to help.