NVIDIA Releases Kumo Tabular Foundation Model for Zero-Shot Tabular Prediction

NVIDIA Releases Kumo Tabular Foundation Model for Zero-Shot Tabular Prediction NVIDIA has released Kumo Tabular, an open-source tabular foundation model available on Hugging Face under the OpenMDW-1.1 license. Pretrained solely on synthetically generated tables, the Transformer-based model performs classification and regression in a single forward pass without task-specific training, tuning, or manual feature engineering. Kumo Tabular ranges from 28M to 215M parameters and operates via in-context learning to predict query row labels from labeled context rows in a single pass. The model architecture uses cell embeddings with Fourier features, column and row attention with row compression, in-context Transformer attention with Test-GQA, and a length-aware attention temperature. Pretraining used tens of millions of procedural tables generated from Structural Causal Models featuring realistic messiness like missing values, coarsened features, and heavy tails. Across benchmarks including TabArena, BeyondArena, TALENT, and ScoringBench, Kumo Tabular set state-of-the-art results on the accuracy-efficiency Pareto front. Native operation is limited to numerical and categorical features and up to 10 classes in a single pass, though built-in pre-processing and error-correcting output codes extend capability. Architecture and In-Context Inference Kumo Tabular translates tabular in-context learning into a specialized Transformer workflow. Numerical and categorical cells are converted to tokens using Fourier features with learned frequencies and special handling for missing values. Multiple alternating layers of column attention (scaling linearly with rows) and row attention (using rotary positions and four learnable [CLS] tokens) compress each row into an embedding. A final Transformer performs in-context learning: context rows attend to each other, while query rows attend exclusively to context rows using Test-GQA. To maintain attention sharpness across tens of thousands of rows, query representations are scaled using a learned, length-aware attention temperature. Synthetic Pretraining Pipeline The model is pretrained exclusively on artificial datasets generated via Structural Causal Models (SCMs). A procedural generator creates random causal graphs with diverse node mechanisms, converts selected nodes into categorical or numerical features or targets, clips outliers, and discards unlearnable data via tree-ensemble verification. Realistic imperfections—including heavy-tailed regression targets, duplicate rows with conflicting labels, and multi-pattern missing values—are injected directly. Training proceeds across three stages expanding context length from 1,024 rows to 60,000 rows across up to 100 columns, exposing model sizes (Small, Medium, Large) to 35, 71, and 137 million synthetic tables respectively. Benchmark Performance Evaluated against tree models, AutoGluon, and recent tabular models on the TabArena benchmark, Kumo Tabular achieved first place overall with an ELO of 1950, running 17 times faster than LimiX-2 on a single RTX 6000 Pro setup. It additionally ranked first on BeyondArena (1418 ELO, 7.78% Improvability score) and secured top overall ranks on TALENT across classification accuracy, log-loss, and regression RMSE. On ScoringBench's predictive distribution evaluation, the Large and Medium variants ranked first and second. Deployment and Known Limitations The model is deployed through NVIDIA's GPU-native structured-data-models library, accepting pandas DataFrames directly. Current limitations include native support restricted to numerical and categorical data, though timestamps, text, and images can be preprocessed into standard features. Single-pass inference natively covers up to 10 classes, using error-correcting output codes to scale higher. Prediction accuracy can degrade when evaluating distributions differing significantly from context rows or training ranges.

英伟达发布开源表格基础模型 Kumo Tabular 英伟达在 Hugging Face 开源了表格数据基础模型 Kumo Tabular。该模型基于 Transformer 架构并通过纯人工合成数据预训练，支持在单次前向传播中完成分类与回归任务，无需训练、微调或特征工程。 模型提供 28M 至 215M 三种参数规模，基于 OpenMDW-1.1 协议开源并商用友好。 采用单元格嵌入、行列注意力交互及基于上下文学习的 Transformer 架构。 完全基于结构因果模型（SCM）生成的数千万个人工表格进行多阶段预训练。 在 TabArena、BeyondArena、TALENT 及 ScoringBench 四项基准测试中均位居榜首。 表格上下文学习架构与机制 Kumo Tabular 采用专为表格结构设计的 Transformer 架构。单元格嵌入通过可学习傅里叶特征编码数值与类别特征并原生处理缺失值；随后通过行列注意力交替压缩出每行表征；最后在行表征上执行上下文学习，待预测查询行仅单向关注上下文行，其键值缓存可复用。模型还引入随键数量对数变化的长度感知注意力温度，以保障大表格场景下的注意力敏锐度。 合成因果数据预训练与基准表现 该模型完全在结构因果模型（SCM）采样的合成数据上预训练，注入了缺失值、高基数类别及重尾分布等现实缺陷。预训练分三阶段逐步将上下文扩展至 60,000 行，三种规模分别训练了约 3500 万、7100 万与 1.37 亿个合成表格。在 TabArena 评测中，Kumo Tabular 取得 1950 的 ELO 评分且运行速度达到 LimiX-2 的 17 倍，并在 BeyondArena、TALENT 和 ScoringBench 上均刷新记录。 适用范围与使用限制 Kumo Tabular 原生支持数值与类别型特征，文本、图像及时间戳等需经预处理转换；单次前向推理支持最多 10 个类别，库内结合纠错输出码可扩展至更多类别。当测试表格分布与训练范围差异极大或查询行分布出现偏移时，预测准确率可能会下降。

Highlights (TL;DR) NVIDIA Kumo Tabular, part of the NVIDIA Kumo Structured model collection, is an open foundation model for tabular data now available on Hugging Face . Given a table of labeled rows, it predicts the labels of new rows in a single forward pass, with no training, no tuning, and no feature engineering, for both classification and regression. It was pretrained only on artificial data, comes in three sizes (28M to 215M parameters), runs through our open-source library , and is released under the OpenMDW-1.1 license for commercial use. It ranks first on the four benchmarks TabArena , BeyondArena , TALENT and ScoringBench . Model Code: https://github.com/NVIDIA/structured-data-models Model Weights: https://huggingface.co/nvidia/Kumo-Tabular The Shift to Tabular Foundation Models Tabular data is the backbone of enterprise machine learning. Customer records, transactions, sensor logs, claims, and orders all live in tables, and predicting churn, default, demand, or price from them is among the most common machine learning tasks in industry. For two decades, this work has been done with gradient-boosted trees, and it has worked well. But the lifecycle around those models has barely changed. Every new question means collecting labels, engineering features, searching hyperparameters, validating, and deploying a model that knows nothing about tables in general and learns each task from scratch. Large Language Models showed a different way of working with new tasks. Given a few examples in the prompt, a pretrained model solves the task without updating a single weight. This is in-context learning , and it applies to tables just as well as to text: a model pretrained on millions of tables can read a labeled table as its context and predict the labels of new rows directly. Today, we are releasing NVIDIA Kumo Tabular ( GitHub , HuggingFace ), an open foundation model for tabular classification and regression. Given a table with labeled rows and the rows you want predictions for, Kumo Tabular returns class probabilities or numeric predictions in a single forward pass. How Kumo Tabular Works Kumo Tabular is a Transformer built around the structure of a table, utilizing column, row and in-context attention as introduced in TabICL and TabPFN . To predict a label it has to do three things: (1) understand what each value means within its column, (2) understand how the columns of a row interact, and (3) relate the context rows with existing labels to the query rows with unknown labels. Kumo Tabular achieves this as follows: Cell Embedding: A group of cells becomes a token. Numerical and categorical values pass through Fourier features, sines and cosines of learned frequencies, with separate weights for each type. Missing values need no imputation and are treated specially. Finally, every token in the context receives a label embedding. Row Embedding: We then turn each row into an embedding by alternating two kinds of attention multiple times. Column attention looks down a single column and learns what a value means in the distribution of its column, e.g. , whether a 42 is typical or extreme, via induced self-attention. Its cost therefore grows linearly with the number of rows. Row attention looks across the tokens of a single row and learns how features interact, with rotary positions to tell columns apart. Four learnable [CLS] tokens join each row and act as the final readout of a row. After this row compression, the cost of the final stage no longer depends on the number of columns. In-context Learning: A final Transformer operates on the row embeddings. Context rows attend to each other, while query rows attend to context rows only. Each prediction therefore depends only on the context and on the row itself, not on which other rows are scored alongside it. Because the context never looks at the queries, its keys and values are computed once and can be reused for follow-up predictions. Query rows utilize Test-GQA, which shrinks the cache that every prediction reads. A head turns each query row into class probabilities for classification and 999 quantiles for regression, from which a point prediction and an uncertainty estimate follow. Length-aware Attention Temperature: Softmax attention spreads out as the number of keys grows. Attention that is sharp over a few hundred rows can dissolve over tens of thousands, which is exactly the situation when a table at inference is much larger than a typical training table. Kumo Tabular therefore scales every query by a temperature that grows with the logarithm of the number of keys, with a coefficient learned separately for each attention head. The result is attention that stays sharp as tables grow longer or wider. How Kumo Tabular was Built Kumo Tabular is pretrained entirely on artificial tables. Each training table is sampled from a Structural Causal Model (SCM) in the six steps shown below: We first draw a configuration for the whole table, from its size and task to its mechanisms and missingness. A random causal graph then links hidden variables, evaluated from root to leaf via randomly drawn functions at every node ( e.g. , linear maps, small neural networks, trees or Gaussian processes). Some nodes become numerical or categorical columns, one becomes the target, and the rest stay hidden, like the unmeasured causes behind real data. Post-processing correlates groups of columns, clips outliers, and injects missing values, and a quick tree-ensemble check discards any table without a learnable signal. Because the generator is a procedural sampler rather than a trained model, it produces an endless supply of tables, each with a new graph and new mechanisms. Real-world tables are messy, so we built more of their imperfections into the generator. Values go missing in several patterns, some features are coarsened so that duplicate rows may disagree on their label, some categorical columns carry many levels, and regression targets can be heavy-tailed. A model that has seen millions of such tables learns to handle these imperfections without any cleanup. On every artificial table, the model sees most of the rows with their labels as context and learns to predict the labels of the remaining rows, with a cross-entropy loss for classification and a quantile loss for regression. Classification and regression are trained as separate models. Similarly to TabICLv2, training runs in three stages. The first and longest stage uses tables of 1,024 rows and up to 100 columns and teaches the model what tables look like. The second stage varies the context from 400 to 10,240 rows, and the third extends it to 60,000 rows, still with up to 100 columns. In total, Kumo Tabular-Small/Medium/Large saw about 35/71/137 million artificial tables. Our training recipe and artificial data generators will be released soon. Performance We ran all three Kumo Tabular sizes with default settings against the full TabArena leaderboard, spanning tuned gradient-boosted trees, AutoGluon, and the latest tabular foundation models. Kumo Tabular ranks first overall with an ELO of 1950 while running 17 faster than LimiX-2 under a uniform single RTX 6000 Pro evaluation setup. Across all three model sizes, Kumo Tabular establishes a new state-of-the-art on the accuracy-efficiency Pareto front: We also evaluated Kumo Tabular on BeyondArena , TALENT and ScoringBench . On BeyondArena, Kumo Tabular reaches an ELO of 1418 with an Improvability score of 7.78%, placing first on the leaderboard. On TALENT, it achieves the top overall ranking across classification accuracy, classification log-loss, and regression RMSE, with average ranks of 6.67, 3.98, and 4.22. On ScoringBench, a benchmark for predictive distributions, Kumo Tabular-Large and Medium rank first and second on average rank. Limitations Kumo Tabular works on numerical and categorical columns only, while text, images, or timestamps can be turned into features via built-in pre-processing recipes. A single forward pass covers up to 10 classes, which the library extends to any number of classes with error-correcting output codes. Accuracy may degrade on tables far beyond the training ranges or when the query rows come from a different distribution than the context rows, so, as with any predictive model, validate accuracy and calibration on your own held-out data before deployment. Demo Kumo Tabular runs via NVIDIA's newly released GPU-native library for structured-data-models . The library downloads the weights from the Hub on first use and provides the preprocessing, ensembling, and many-class handling used in our evaluations. The code below is all it takes to go from a pandas.DataFrame to a prediction: import sdm # structured-data-models # Tensorize tabular data: table = sdm.TableTensor.from_pandas(pd.read_csv(...), device= "cuda" ) na_mask = table[ "target" ].isnan() model = sdm.models.KumoTabular(device= "cuda" ) pred = model( # In-context examples (features/targets): x_context=table[~na_mask].drop_columns( "target" ), y_context=table[~na_mask, "target" ], # Prediction examples (features): x_query=table[na_mask].drop_columns( "target" ), ) Start Building with Kumo Tabular Kumo Tabular is released under the OpenMDW License Agreement, version 1.1 . NVIDIA believes Trustworthy AI is a shared responsibility, and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report model quality, risk, security vulnerabilities, or NVIDIA AI concerns here . Model Code: https://github.com/NVIDIA/structured-data-models Model Weights: https://huggingface.co/nvidia/Kumo-Tabular Acknowledgements We thank David Holzmüller for contributing significant ideas and ablations to Kumo Tabular. We thank Vignesh Kothapalli for his help on Kumo Tabular during his internship.