OrionMoE-Small-v1.0: Rethinking How Tabular Models Spend Their Computation

To our knowledge, OrionMoE is the first tabular foundation model whose experts form the core of the backbone and in which sparse expert routing is explicitly conditioned on a learned representation of the dataset.

It is also the first MoE tabular foundation model to stand alongside frontier dense models: in its first version, at 37.7M parameters, it rates within 31 Elo of TabPFN-3 on TabArena, above it on OpenML-CC18, and above Xiaomi-TabLDM, the only other MoE model in our comparison, on every suite where both are reported.

In-context tabular foundation models such as TabPFN (Hollmann et al., 2025), TabICL (Qu et al., 2025), TabDPT (Ma et al., 2024) and LimiX (Zhang et al., 2025) have shown that a single pretrained network can be competitive across very different datasets. Most of them share one architectural property: the feed-forward transformations inside each block are dense and applied identically to every token, whichever feature or dataset it comes from. A new dataset changes the representations, through attention and context-dependent encoders, but not the computation applied to them.

Tabular data makes this a real constraint. Features can be continuous measurements, categorical codes, counts or ordinal quantities, with different distributions, missingness patterns and relationships to the target, and datasets differ in size, dimensionality and label balance. Context-aware encoders handle part of this by giving the same value different embeddings in different datasets. We hypothesize that the allocation of computation should depend on the data as well: different features and datasets may be better served by different learned transformations.

If we take that hypothesis seriously, at least three architectural directions open up. They are complementary rather than competing.

OrionMoE-Small-v1.0 places two bets: on the first and, centrally, on the third. Adaptive information exchange remains an open and promising direction that we expect to return to.

Images share a grid. Text shares a vocabulary. Tables share almost nothing. One table mixes continuous measurements, categorical codes, sparse counts and missing values. Another, with the same number of columns, obeys entirely different statistical relationships. Consider the number 10: it might be an age, a count, a sensor reading, or an arbitrary label for a category. Its meaning lives in the column’s distribution, its relationships to other variables, and the labeled examples around it.

The field’s dominant response has been to adapt representations: encode each cell in the context of its column and its dataset, so that two identical values can mean different things. That is necessary, and OrionMoE invests heavily in it. But it may not be sufficient. When every dataset flows through the same dense transformations, a single set of weights must serve skewed and symmetric distributions, binary and high-cardinality features, ten-row and ten-thousand-row contexts, balanced and badly imbalanced targets. All of that diversity competes for the same shared capacity.

Our hypothesis is simple to state. If heterogeneity is the defining property of tabular data, a model should be able to adapt not only what it represents, but how it computes.

A sparse MoE layer replaces one dense feed-forward network with a set of expert networks and a router that sends each token to a small subset of them (Shazeer et al., 2017; Fedus et al., 2022). The model gains a larger repertoire of transformations while evaluating only a few experts per token. In language models the router usually decides from the token’s own representation, which works because tokens carry broadly consistent meaning across inputs.

Tabular feature tokens do not. A router that sees a single token cannot tell whether it comes from a skewed column, a high-cardinality categorical, or a dataset with a rare target class. OrionMoE therefore asks its router a richer question. It conditions expert selection on two signals at once: the local representation of each cell, and a learned summary of the dataset derived from the labeled support examples. Two tokens with similar local representations can then be sent to different experts when they occur in different datasets, and tokens from different datasets with similar computational needs can share experts.

MoE has already been applied to tabular data. MixturePFN (Xu et al., ICLR 2025) uses a mixture of in-context prompters to scale TabPFN-style models to larger datasets. Xiaomi-TabLDM (2026) replaces the feed-forward layer of selected layers in its in-context predictor with MoE. MoE components also appear in task-specific tabular networks, for example as a projection head for masked-feature pretraining in MaskTab (2026). Table 1 compares OrionMoE-Small with the two in-context approaches. Of the three, OrionMoE-Small is the only one whose experts form the core of the representation backbone.

Most tabular foundation models adapt only their representations. OrionMoE-Small-v1.0 also adapts the computation applied to them: for every table, in a single forward pass, it selects which experts process each feature token. Three design choices make this possible:

The most interesting question about a dataset-aware router is whether it uses the dataset at all. If routing collapsed to a few universal experts, or ignored the dataset register, the architecture would be MoE in name only.

It does not appear to. Across our routing analysis, expert selection is statistically associated with properties of the dataset: context size, dimensionality, class imbalance, and feature composition. Associations are measured as standardized effects between each characteristic and how often an expert is selected, and are tested with Benjamini–Hochberg correction for false discovery.

The patterns are distinct rather than uniform. One expert is selected more often as context size grows but less often as dimensionality rises, and is disfavored on binary-only and numeric-only tables relative to mixed-type ones. Another shows the opposite tendency on dimensionality and is favored on binary-only tables. A third becomes markedly less active as dimensionality increases. Different experts, in other words, appear to have settled into different statistical regimes.

These are associations, not mechanisms. They do not yet tell us what each expert computes, or how much of the model’s accuracy depends on this division of labor. But they are exactly the signature one would hope to see: a router that was given access to the dataset, and was never told what to do with it, learned to send different kinds of tables down different paths.

We evaluated OrionMoE-Small-v1.0 on five benchmark suites (TabArena, BeyondArena, OpenML-CC18, TALENT and BCCO) against recent tabular foundation models and established baselines. OrionMoE-Small-v1.0 is the first version of a new architecture, and it asks whether a backbone built on dataset-conditioned experts can stand alongside frontier dense tabular foundation models at a modest parameter budget. The results suggest it can. Figure 6 places every model by size and rating on each suite; Table 2 gives the full numbers, grouped by size.

The table also shows clear headroom. Several models in our own size class, including Causilo, Kumo-Tabular-Small and EXAONE-Tabular, score higher on TabArena. This is version 1 of a new architectural direction, and we report the full picture because both what it achieves and how far it can still go are part of the result.

The familiar story of foundation models is a story of scale. Scale will keep mattering for tabular models too. But OrionMoE-Small-v1.0 points to a second axis of progress that has received far less attention: not how much computation a model has, but how it allocates that computation across the datasets it meets.

Tables are heterogeneous by nature, in their features, their distributions and the relationships that govern them. Perhaps the models that learn from them should be heterogeneous too: able to read a dataset, recognize the regime it belongs to, and route it through the transformations best suited to it.

OrionMoE-Small-v1.0 is our first step in that direction and, to our knowledge, the first MoE tabular foundation model to stand alongside frontier dense models.

Its competitive results at 37.7 million parameters, and experts that discovered dataset-dependent roles without being told to, make us confident the direction is worth pursuing seriously.

We do not think the next leap in tabular foundation models has to come only from larger or better-tuned Transformers. It may come from rethinking which computation each table deserves. If you work on tabular learning, foundation models, sparse architectures or real-world machine learning, we invite you to test that idea with us.

OrionMoE-Small-v1.0, the first release of the OrionMoE family, is available today through early access to the hosted Lexsi API. Researchers, developers and organizations can evaluate it on their own classification datasets, and we are especially interested in feedback from settings where tables are small, heterogeneous or imbalanced. Our research and benchmark findings are public in the technical report.