TabPFN and TabICL vs. tuned XGBoost: the model that doesn't train won 14/14
Tabular foundation models (TabICL and TabPFN) outperformed tuned XGBoost across fourteen datasets in inference-only tests, bypassing hyperparameter tuning.
summary is enough The article provides a useful benchmark of tabular foundation models against XGBoost, but commenters rightly flag methodological quirks and a heavily AI-generated writing style.
summary
An author benchmarks tabular foundation models (TabPFN and TabICL) against tuned XGBoost on fourteen datasets, finding that these models require no gradient descent training and win on AUC metrics while saving compute time. However, the evaluation reveals limits on wide tables with hundreds of columns and highlights friction like licensing walls for newer model versions. Commenters heavily criticize the breathless, LLM-generated writing style and raise valid methodological critiques regarding metric mismatches and training time comparisons.
key points
- TabICL and TabPFN perform in-context learning without gradient descent, beating tuned XGBoost on 14 of 14 datasets by AUC in this benchmark.
- The models eliminate the hyperparameter tuning bottleneck, drastically reducing end-to-end execution time compared to XGBoost search.
- Performance degrades significantly on wide tables with hundreds of columns, such as chemical descriptor datasets like Bioresponse.
who should read it
Data scientists and machine learning engineers evaluating alternative approaches for tabular data modeling.
discussion skeptical
consensus
Commenters generally acknowledge that tabular foundation models perform surprisingly well in practice, but express widespread annoyance at the breathless, LLM-generated writing style of the blog post.
pushback
Technical commenters point out methodological flaws, such as tuning XGBoost for accuracy while evaluating on AUC, and question whether these models truly scale to massive industrial datasets.
XGBoost’s search optimized accuracy, and afterwards I also compare by area under the curve. Which means the “fourteen of fourteen on AUC” is against a boosting model that was not tuned for that metric. Tuning it for AUC would probably improve it there; I did not measure that.
So, not a fair test?
I also did not get why Xgboost had to count its training time for the inference. You only train once. I guess in some scenario, where someone says, "I need the best model now, you have five minutes on this singular dataset", but I have never been in that situation.
I would feel better if scripts …
It highlights specific methodological flaws in the benchmark, such as metric mismatch for XGBoost tuning and questionable training-time assumptions.
Generated by an LLM from the article and thread. Check the sources before relying on it.