Theory of Mind or Theory of Benchmarks?

1. Introduction & Background

Theory of Mind (ToM), the capacity to attribute beliefs, desires, intentions, and emotions to others, is foundational to human social cognition (Beaudoin et al. 2020) and matters for deploying LLMs in agentic and social settings. The practical stakes are concrete: when LLMs are equipped with explicit ToM representations linking percepts, beliefs, desires, and actions, embodied agents achieve $3\times$ more task milestones and collect $2.3\times$ more unique items than ToM-free baselines in open-ended environments (Lică et al. 2026). Yet if the benchmarks used to certify ToM capability are themselves psychometrically unsound, the downstream decisions they support, such as which models are deployed in social roles and which are trusted in autonomous agentic pipelines, rest on measurement that has never been validated. ToM benchmark scores are used to argue that certain models possess social intelligence, are safe for human-facing roles, or have crossed a threshold of genuine mental-state reasoning (Kosinski 2024; Chen et al. 2024). These claims rest on an assumption that has gone largely unexamined: that ToM, as operationalized in LLM benchmarks, is a psychometrically coherent construct that different measurement instruments will assess consistently, and whose published item counts reflect genuine measurement capacity.

The measurement problem.

The empirical record casts doubt on both the reliability and the construct validity of current ToM benchmarks. Kosinski (2024) reported GPT-4 solving 95% of classic false-belief tasks and argued ToM-like ability had emerged spontaneously from large-scale training; Ullman (2023) replied that small, logically irrelevant perturbations collapsed performance, suggesting surface pattern matching rather than genuine belief attribution. This exchange spawned a wave of new ToM benchmarks but did not resolve the underlying measurement question: do aggregate accuracy scores reflect a coherent latent construct, or an amalgam of loosely related surface tasks unified by name rather than by measurement? A structural critique sharpens the concern: Riemer et al. (2024) argue that the dominant paradigm of LLM ToM evaluation is broken at the design level, because benchmarks developed for human subjects implicitly assume self-consistent reasoning across questions, an assumption known not to hold for current LLMs. They distinguish literal ToM (predicting the behavior of others from static context, what virtually all existing benchmarks test) from functional ToM (adapting to a partner in-context following a rational response to their behavior), and demonstrate empirically that strong literal ToM performance does not imply strong functional ToM performance, nor vice versa. More recent empirical work compounds the concern: Jones, Trott, and Bergen (2024) found uneven LLM performance across a six-experiment ToM battery, and Tong et al. (2026) reported substantial heterogeneity across 22 models and 46 paradigms. Both results are consistent with ToM not functioning as a unified capacity in LLMs, but neither study applied formal psychometric analysis to their task sets. Bean et al. (2026) showed, in a systematic review of 445 LLM benchmarks, that leaderboard scores are routinely treated as evidence of coherent underlying capability without any structural validity check. ToM benchmarks are no exception.

Addressing this requires tools from measurement science. Construct validity (Cronbach and Meehl 1955; Salaudeen et al. 2025) asks whether an instrument measures what it claims; a benchmark may produce stable, reproducible scores that consistently measure the wrong thing. Convergent validity, the degree to which two instruments claiming to measure the same construct agree in their assessments, is its canonical operationalization for cross-instrument comparison (Perlitz et al. 2024). Classical test theory (CTT) item discrimination (Furr and Bacharach 2014; Edelen and Reeve 2007) quantifies how much each item contributes signal versus noise to model-level ability estimates; the attenuation bound ($r_{XY} \leq \sqrt{\rho_{XX}\rho_{YY}}$) implies that low-reliability instruments cannot, in principle, exhibit high cross-benchmark convergence regardless of how genuine the underlying construct is (Furr and Bacharach 2014). Together, these tools allow us to distinguish two explanations for benchmark divergence: measurement noise (low-discrimination items pulling rankings in opposite directions) versus genuine construct fragmentation (the benchmarks operationalizing fundamentally incompatible notions of ToM). We refer to the first as Scenario A and the second as Scenario B. Lalor, Wu, and Yu (2016) established IRT as a principled framework for NLP benchmark item analysis; Burnell et al. (2023) argued for Kendall’s $\tau_b$ as the appropriate rank-correlation statistic at modest model counts.

This paper.

We apply this psychometric framework to ToMBench (Chen et al. 2024), ToMATO (Shinoda et al. 2025), and CogToM (Tong et al. 2026), three Theory-of-Mind benchmarks sharing the AToMS five-category taxonomy (Beaudoin et al. 2020) but embodying sharply opposed designs (hand-authored vignettes, multi-turn role-played conversations, and cognition-inspired task paradigms). We run a three-stage pipeline on a panel of $M{=}27$ models spanning 1.0B–72.7B parameters: (1) a cross-benchmark convergent-validity test across all three pairwise comparisons, (2) CTT item-discrimination filtering, and (3) a repeated cross-fit re-test, run independently on each benchmark pair to avoid selection bias. We find that all five mental state categories achieve strong convergent validity pre-filter in every pairwise comparison, with item filtering yielding a positive improvement in agreement for 14 of the 15 pairwise-category tests. The sole exception is Belief on the ToMBench$\leftrightarrow$CogToM pair, where filtering produces no significant change. An exploratory factor analysis reveals a single common factor rather than five separable subconstructs, with a large unexplained residual. These findings support the interpretation that ToM benchmarks measure a shared construct once measurement noise is removed, but that roughly a quarter to a third of published items contribute no discriminative signal.

2. Methods

The study uses five benchmarks in total. Three are Theory-of-Mind instruments sharing the AToMS five-category taxonomy (Belief, Intention, Desire, Emotion, Knowledge) but differing in design: ToMBench (Chen et al. 2024) (2,860 hand-authored MCQ items spanning 31 fine-grained abilities); ToMATO (Shinoda et al. 2025) (5,401 items directly organized into the five AToMS categories); and CogToM (Tong et al. 2026) (8,513 items spanning 46 cognitive-science paradigms, organized into seven categories: the five AToMS categories plus Percept and Non-Literal). Item prompts are formed using each benchmark’s published template. For the cross-benchmark convergent-validity analyses, ToMBench items are restricted to the five shared categories via a 31$\to$5 ability mapping (yielding 2,112 items; Appendix C), and CogToM items are restricted to its five AToMS-compatible categories; the Percept and Non-Literal categories are excluded from cross-benchmark analyses.

Two general language understanding benchmarks serve as discriminant-validity controls: HellaSwag (Zellers et al. 2019), a commonsense reasoning benchmark (9,609 items, story-continuation MCQ with adversarially generated foils); and MMLU-Pro (Wang et al. 2024), a challenging multiple-choice knowledge benchmark across 14 disciplines (12,032 items).

Model panel. We evaluated $M=27$ open-weights instruction-tuned LLMs spanning 1.0B–72.7B parameters across ten architecture families, including Qwen-2.5 ({1.5B, 3B, 7B, 32B, 72B}), Qwen-2 (72B), Llama-3.2 ({1B, 3B}), Llama-3.1 ({8B, 70B}), Llama-3 (8B), Gemma-3 (1B), Gemma-2 ({2B, 9B, 27B}), Phi ({3.5-mini, 3-medium-14B}), Mistral ({7B-v0.3, Nemo-12B, Small-3-24B}), Yi-1.5 ({9B, 34B}), OLMo-2 ({7B, 13B}), Granite-3.1-8B, SmolLM2-1.7B, and Falcon3-10B (Table 1). All models are served via vLLM (Kwon et al. 2023) at temperature 0 through Modal, with each model’s tokenizer chat template applied. ToMATO prompts that exceeded each model’s context window with the 8-token answer budget were left-truncated (preserving system prompt and final user question). The full $M \times \text{items}$ binary response matrix is materialized in three parquet files (one per ToM benchmark). Parse failures are coded as 0 (incorrect) in all accuracy calculations; parse-failure rates are below 3% for all models on all three ToM benchmarks (max 2.55% for Yi-1.5-34B on CogToM), with models exceeding 2% noted in Table 1 ($\dagger$).

2.1 Stage 1: cross-benchmark convergent validity

For each shared AToMS category, we compute Kendall’s $\tau_b$ rank correlation between mean-accuracy model orderings for all three pairwise benchmark comparisons (ToMBench$\leftrightarrow$ToMATO, ToMBench$\leftrightarrow$CogToM, ToMATO$\leftrightarrow$CogToM), with 95% CIs from 1,000 paired item-bootstrap iterations per pair. Holm–Bonferroni correction at $\alpha=0.05$ adjusts for the five simultaneous category tests within each pair. We adopt $\tau_b > 0.5$ with lower CI bound $> 0.3$ as the acceptance criterion for substantive convergent validity. Each pair’s aggregate $\tau_b$ is the unweighted mean of its five per-category values; the overall aggregate reported is the unweighted mean across all three pairs.

2.2 Stage 2: CTT item-discrimination filter

Under CTT, items with near-zero variance across the panel, or items that fail to correlate with overall model performance, contribute mostly measurement noise rather than discriminative signal. We apply a fixed item-filtering rule symmetrically. Item $i$ is dropped if (a) $\operatorname{var}(\mathbf{x}_i)=0$ across the panel, or (b) its leave-one-out classical-test-theory point-biserial discrimination $d_i = \operatorname{corr}(x_{m,i},T_{m,-i})$ falls below 0.15, where $T_{m,-i}=\sum_{j\neq i}x_{m,j}$ is model $m$’s total score excluding item $i$. The 0.15 cutoff is the standard psychometric threshold below which items are flagged as uninformative (Furr and Bacharach 2014; Edelen and Reeve 2007). We use point-biserial rather than 2PL IRT discrimination because even at $M=27$ the hierarchical Bayesian 2PL would require larger sample sizes for stable parameter estimation (König, Spoden, and Frey 2020; Fujimoto and Neugebauer 2020); CTT discrimination remains consistent and interpretable at this scale.

2.3 Stage 3: repeated cross-fit re-test on filtered benchmarks

To address potential selection bias where items are selected for discrimination on the same panel used to measure convergence, we employ a repeated stratified $K$-fold cross-fit design (K=5 folds, 10 random seeds, 50 folds total), splitting on items rather than models, run independently for each of the three benchmark pairs. For each pair and category, the items contributed by both benchmarks are pooled and partitioned into 5 folds. Within each fold: (i) a discrimination mask is fit on the pooled training items (the other K-1 folds) against an anchor score derived from those training items and thresholded at $d_i\ge0.15$; (ii) each held-out test item’s discrimination against that anchor is computed and thresholded the same way; (iii) each benchmark’s model accuracies are recomputed restricted to its own surviving test items, and $\tau_b$ is computed between the two benchmarks’ accuracy vectors both before (all test items) and after (filtered test items) applying the mask. Because items have no cross-benchmark correspondence, the mask is fit and applied entirely within the pooled item pool of each pair, so no item-level alignment across benchmarks is required, and the filter is never fit and evaluated on the same items. The headline statistic per pair and category is $\Delta_{c} = \tau_{b,c}^{\text{after}} - \tau_{b,c}^{\text{before}}$, averaged across the 50 folds, with bootstrap CIs and $p$-values from the fold distribution (Holm–Bonferroni-adjusted within each pair across the five categories). Category-level and aggregate summary statistics combine the three pairs by taking the unweighted mean of the three pairwise $\Delta_{c}$ values. Because this averages only three estimates, we report a conservative interval rather than a pooled bootstrap CI: each pair contributes its own fold-level CI, and the conservative per-category interval is the widest span across the three pairs’ CIs (i.e., the minimum lower bound and maximum upper bound). The aggregate conservative CI is, in turn, the widest span across the five categories’ conservative intervals. A consistently positive $\Delta$ across pairs and categories supports Scenario A (apparent benchmark divergence reflects measurement noise); a persistent or reversed $\Delta$ for a given category supports Scenario B for that category specifically (the benchmarks operationalize that mental-state category in genuinely incompatible ways despite the shared taxonomy). Per-pair, per-category sample loss is reported in Appendix F (Table 6); across all three pairs, no fold was excluded for insufficient sample size.

2.4 Baselines and controls

The audit uses two within-design baselines: (i) the raw cross-benchmark $\tau_b$ at Stage 1 serves as the baseline against which the filter-induced Stage-3 $\tau_b$ is compared, and (ii) within each of the three pairwise comparisons, the two benchmarks serve as each other’s baseline within shared AToMS categories (no single benchmark, including CogToM, is privileged as ground truth). No external single-model baseline is used; the validity claim is purely cross-instrument.

Mean overall accuracies of 0.607 (ToMBench), 0.619 (ToMATO), and 0.660 (CogToM) place the panel well above chance (25% on each benchmark) and well below ceiling, providing operating range for both convergent-validity and item-discrimination analyses. CogToM’s higher mean accuracy is broadly consistent across models rather than driven by a few outliers.

3. Results

3.1 Stage 1: all five categories achieve convergent validity

Per-pair, per-category Kendall $\tau_b$ statistics are reported in Table 2 (Appendix B), and shown as pairwise estimates in the left panel in Figure 1. All 15 pair$\times$category cells satisfy the acceptance criterion (Holm-adjusted within each pair): $\tau_b$ ranges from $0.584$ (Belief, ToMATO$\leftrightarrow$CogToM) to $0.817$ (Emotion, ToMBench$\leftrightarrow$CogToM), and every cell exceeds both the $\tau_b>0.5$ threshold and the lower CI bound $>0.3$ criterion. Each pair’s own aggregate is similar in magnitude: ToMBench$\leftrightarrow$ToMATO $\tau_b=0.697$, ToMBench$\leftrightarrow$CogToM $\tau_b=0.710$, ToMATO$\leftrightarrow$CogToM $\tau_b=0.699$. Emotion is the most convergent category in all three pairs ($\tau_b>0.78$ throughout); Belief is the least convergent in all three ($\tau_b=0.58$–$0.66$). Averaging across all three pairs, the overall aggregate is $\tau_b=0.702$.

3.2 Stage 2: substantial item dropout from discrimination filter

Figure 2 summarises full-benchmark item retention after the discrimination filter. ToMBench: 66.4% of all items survive the filter (1,898/2,860), and 8.0% have zero variance across the panel; within the five shared AToMS categories used for cross-benchmark analysis, 67.7% survive (1,429/2,112). ToMATO: 72.7% survive (3,926/5,401), and 4.6% have zero variance; ToMATO has no items outside the five shared categories, so its full-benchmark and five-category retention coincide. CogToM: 73.0% survive (6,215/8,513), and 7.0% have zero variance; within its five AToMS-compatible categories (Section 2), 75.9% survive (4,362/5,747). On all three benchmarks the median $d_i$ exceeds the threshold, so most items do discriminate, yet 27–34% of items fall below threshold or have zero variance and contribute no discriminative signal. The published item counts overstate effective measurement capacity by roughly a quarter to a third (34% ToMBench, 27% ToMATO, 27% CogToM).

3.3 Stage 3: an item-level cross-fit, run independently on each pair

Re-computing $\tau_b$ on the held-out folds of the repeated 5-fold cross-fit (50 folds per pair, 150 fold-results per category in total) yields Table 3 (Appendix B) and the right panel of Figure 1. The cross-fit addresses selection bias by ensuring the discrimination anchor is estimated from training items before being applied to held-out items, and this is done independently within each of the three pairs. 14 of the 15 pair$\times$category cells show a significant positive $\Delta$ (Holm-adjusted bootstrap CI excluding zero); the sole exception is Belief on ToMBench$\leftrightarrow$CogToM ($\Delta=-0.007$, $p=0.088$, CI $[-0.015,+0.001]$), which is indistinguishable from zero rather than a significant reversal.

Averaged across the three pairs, the gains are largest for Knowledge ($\Delta{=}{+}0.082$) and Desire ($\Delta{=}{+}0.079$), moderate for Emotion ($+0.059$) and Intention ($+0.030$), and smallest and least robust for Belief ($+0.009$), the only category where one of the three pairs (ToMBench$\leftrightarrow$CogToM) shows no gain. The aggregate improvement is $\Delta{=}{+}0.052$ with a conservative 95% interval of $[-0.01,+0.13]$ (Section 2.3); the interval is wide because it is built from only three pairwise estimates rather than a pooled bootstrap, and it should be read alongside the fact that 14 of 15 underlying cells are individually significant and positive, not as evidence of a null aggregate effect. This is achieved while discarding 27.3% of held-out items on average across all pairs and categories (22.1% for Desire to 31.2% for Emotion; per-pair, per-category loss in Appendix F, Table 6), with no fold excluded for insufficient sample size; the gain in convergence comes from removing low-signal items. This supports Scenario A for four of the five categories: apparent benchmark divergence partially reflects measurement noise rather than fundamental incompatibility. Belief is the one category where the evidence is mixed: convergence improves robustly between ToMBench and ToMATO but not between ToMBench and CogToM, suggesting its cross-instrument fragility may be specific to how CogToM operationalizes belief-tracking items rather than a general property of the category.

3.4 Factor structure: one dominant factor

We ran EFA (principal-axis factoring, varimax rotation, factor count set by parallel analysis) and standardized PCA on each benchmark individually and on the pooled three-benchmark item set (ToMBench, ToMATO, and CogToM combined), to test whether the data resolve into five AToMS subconstructs or a single dominant dimension. Sampling adequacy on ToMBench is excellent (KMO$=0.95$, Bartlett $p<0.001$); for the pooled set ($p=15{,}705$ items $\gg M=27$ models), the item-by-item correlation matrix is necessarily rank-deficient, so KMO/Bartlett return a degenerate rather than meaningful result there, and we instead rely on EFA convergence and variance explained as evidence of factorability. Results are summarized in Table 4 (Appendix B).

Parallel-analysis EFA retains a single common factor on every matrix (ToMBench, ToMATO, CogToM, and the pooled three-benchmark set), not five, accounting for 19.5–24.6% of total variance (38.5% of common variance on ToMBench), positive evidence for a unified ToM construct over five separable mental-state subconstructs. CogToM’s single factor explains 24.6%, somewhat higher than ToMBench (19.5%) and ToMATO (20.2%), consistent with its broader and more cognitively diverse paradigms loading on the same underlying dimension. PCA scree plots show a consistent elbow at the second component on every matrix (PC1 $19.5\%$, PC2 $9.5\%$ on ToMBench; PC1 $20.2\%$, PC2 $10.0\%$ on ToMATO; PC1 $24.6\%$, PC2 $12.6\%$ on CogToM; PC1 $21.3\%$, PC2 $11.0\%$ on the pooled set); Kaiser’s eigenvalue${>}1$ rule over-extracts at $M=27$, retaining 26 components, so we rely on parallel analysis and the scree elbow instead. We hypothesize, but do not claim to establish, that the second component reflects a partial separation between belief-tracking and intention/social-reasoning items. The leading component captures only $\sim$20–25% of total variance on every matrix, leaving $\sim$75–80% of variance unexplained.

3.5 Discriminant validity: ToM vs. general language understanding

We conducted discriminant validity testing comparing three ToM benchmarks (ToMBench, ToMATO, CogToM) with general language understanding benchmarks (HellaSwag and MMLU-Pro) on the same $M=27$ panel, to test whether ToM benchmarks measure a distinct construct from general language ability. Full pairwise statistics are in Appendix G.

Within vs. cross-construct correlations. The within-ToM correlations are consistently strong across all three pairwise comparisons (mean $\tau_b=0.780$), exceeding both the six cross-construct correlations (each ToM benchmark against each general benchmark, mean $\tau_b=0.706$) and the within-general correlation (HellaSwag$\leftrightarrow$MMLU-Pro, $\tau_b=0.738$; Table 7). The within-ToM/cross-construct gap is about 0.074, and within-ToM convergence also exceeds within-general convergence.

Partial correlation analysis. To isolate the unique ToM component, we computed partial Kendall $\tau_b$ for each within-ToM pair, controlling for MMLU-Pro, HellaSwag, and both jointly (Table 8). All nine partial correlations remain significant despite shrinkage. Averaged across the three pairs, controlling for both general benchmarks leaves $\tau_b=0.525$ (33% shrinkage). The pattern is uneven: ToMBench$\leftrightarrow$CogToM is the least confounded (18.8% shrinkage) and ToMBench$\leftrightarrow$ToMATO the most (41.1%). HellaSwag (commonsense continuation) absorbs more of the ToM convergence than MMLU-Pro (academic knowledge) in every pair, consistent with ToM benchmarks loading more on everyday social/commonsense reasoning than on factual recall.

Variance decomposition. As a complementary, per-benchmark view, we regressed each ToM benchmark’s overall accuracy on a composite general-ability score (mean of standardized HellaSwag and MMLU-Pro accuracy); the regression $R^2$ gives the share of variance attributable to general ability. ToMATO is the most general-dependent (85.4% general, 14.6% unexplained), CogToM is intermediate (74.7%/25.3%), and ToMBench retains the largest unexplained component (69.7%/30.3%). This is consistent with the partial correlations above, since the two highest-shrinkage pairs both involve ToMATO while the one pair excluding it (ToMBench$\leftrightarrow$CogToM) has the lowest shrinkage.

Interpretation. A coherent ToM construct does exist and is not entirely reducible to general language ability, even as it remains partly confounded with it (some of which may reflect shared MCQ formatting rather than pure construct overlap).

4. Discussion

Across five analyses a consistent picture emerges. ToMBench, ToMATO, and CogToM show substantially concordant model orderings within every AToMS category and across all three pairwise comparisons, and a single common factor underlies all three, individually and pooled. ToM as operationalized in these benchmarks therefore behaves as a coherent, largely unidimensional construct rather than five separable mental-state abilities.

An item-level cross-fit, run independently on each of the three benchmark pairs to avoid fitting and evaluating the discrimination filter on the same items, shows that convergence improves once low-discrimination items are removed in 14 of 15 pairwise-category tests: the gain survives held-out evaluation, so where it appears it reflects genuine measurement-noise reduction rather than overfitting to the panel. The one exception is Belief on the ToMBench$\leftrightarrow$CogToM pair, where filtering produces a small, non-significant change rather than a gain. This is a localized, category-specific caveat, not a general failure of the filtering approach.

Three further cautions temper this otherwise positive reading. First, roughly a quarter to a third of published items carry no discriminative signal, so leaderboard item counts overstate effective measurement. Second, the construct these benchmarks measure remains confounded with general language ability: averaged across all three within-ToM pairs, controlling for HellaSwag and MMLU-Pro shrinks convergence by 33%, with the two pairs involving CogToM retaining more of their raw convergence. Third, the three-pair aggregate’s conservative confidence interval is wide and should not be over-interpreted in isolation from the underlying pairwise significance pattern. ToM scores on current benchmarks therefore partly track general capability; pruning low-discrimination items sharpens them for most categories, while more fully separating mental-state reasoning from general ability remains an open problem for benchmark design.

This audit is also a necessary, if unglamorous, precursor to the broader interactive-measurement agenda: before a functional or interactive ToM evaluation can attribute behavioral change to a model’s in-context adaptation rather than to noise, the literal item pools it builds on must already be known to carry signal. Treating static benchmarks as already-solved infrastructure, rather than auditing them with the same rigor demanded of adaptive or strategic evaluation settings, risks inheriting, rather than diagnosing, measurement problems once interaction, optimization pressure, or temporal drift are layered on top.

4.1 Limitations

With $M=27$ below the sample sizes at which confirmatory factor analysis and 2PL IRT item-parameter estimation become stable (König, Spoden, and Frey 2020; Fujimoto and Neugebauer 2020), we use CTT point-biserial discrimination for item filtering and limit structural conclusions to exploratory factor analysis and PCA. We caution that the common $M\ge 30$ rule of thumb is a lower bound for these methods, not a guarantee of robustness; item-level confirmatory modeling here would likely need a substantially larger and more diverse panel than $M=30$. CTT discrimination at the item level assumes unidimensionality within each benchmark; the single-factor EFA evidence supports this, but confirmatory tests await larger panels. The repeated cross-fit, while addressing selection bias, reduces effective sample size per held-out fold; K=5 with 10 seeds was chosen to balance bias-variance tradeoffs and stabilize the per-category estimates at $M=27$. The three-pair aggregate in Stage 3 averages only three pairwise estimates per category, so its reported interval is a conservative span rather than a properly pooled bootstrap CI, and is consequently wide; the underlying pairwise significance pattern (14 of 15 cells, Table 3) is the more informative summary. The pooled three-benchmark EFA’s correlation matrix is rank-deficient ($p=15{,}705$ items $\gg M=27$ models), so KMO and Bartlett’s test are not reported for that analysis (Section 3.4); we rely instead on EFA convergence and the magnitude of variance explained. Finally, the two discriminant-validity controls are noisier than the ToM benchmarks: parse-failure rates exceed 2% for 20 of 27 models on MMLU-Pro and 14 of 27 on HellaSwag (Table 1). Because parse failures are scored as incorrect, the control scores mix ability with output-format compliance, and the partial correlations in Section 3.5 should be read with that in mind.

5. Conclusion

We conducted a convergent-validity and reliability audit of LLM Theory-of-Mind benchmarks, evaluating three instruments (ToMBench, ToMATO, and CogToM) that share the AToMS mental-state taxonomy under sharply different operationalizations. Five findings stand out. (1) Convergent validity is strong across all categories and all three benchmark pairs (Section 3.1). (2) Published item counts overstate effective measurement: roughly a quarter to a third of items carry no discriminative signal (Section 3.2). (3) An item-level cross-fit confirms genuine filtering benefit for the large majority of pairwise-category tests, with Belief on ToMBench$\leftrightarrow$CogToM the lone exception, supporting the interpretation that ToM-as-construct is more robust across operationalizations than raw benchmarks suggest (Section 3.3). (4) Factor analysis detects one dominant factor, not five, with a large unexplained residual unaligned with any AToMS category (Section 3.4). (5) Discriminant validity is partial but real: ToM performance correlates with general language ability but is not reducible to it (Section 3.5). Together, these results indicate that current LLM ToM benchmarks measure a coherent, largely unidimensional construct whose published item counts overstate the precision they deliver, and whose scores remain partly, but not entirely, confounded with general language ability.

Future work.

Two directions follow naturally from these findings. (a) Larger and more diverse model panels. Item-level factor analysis and CTT discrimination are exploratory at $M=27$; stable confirmatory factor analysis and 2PL IRT item-parameter estimation might require $M\ge 50$, with the panel spanning a broader range of architectures and training regimes, including closed-source frontier models currently excluded for inference reproducibility. Expanding the panel is the single most impactful methodological step. (b) Functional and interactive ToM evaluation. All three audited benchmarks are literal (static question answering); the psychometric infrastructure established here sets a baseline for auditing functional ToM tasks (Riemer et al. 2024), where a model must adapt to a partner in-context, and for testing whether the single-factor structure and language confound persist once interaction is required.

A. Model panel and per-benchmark accuracies

B. Core results tables

Tables 2–4 report the full per-category and per-benchmark statistics underlying Sections 3.1, 3.3, and 3.4 respectively.

C. AToMS construct mapping

Each of ToMBench’s fine-grained abilities is mapped to one of the five shared AToMS (Beaudoin et al. 2020) mental-state categories (Belief, Intention, Desire, Emotion, Knowledge); all cross-benchmark analyses (Stages 1 and 3) are restricted to these five categories. CogToM’s own seven-category taxonomy (Section 2) already includes the five AToMS categories by name, so for CogToM the mapping is the identity on those five categories and simply excludes its Percept and Non-Literal categories. Per-category item counts after mapping (ToMBench / ToMATO / CogToM) are: Belief 882 / 1,109 / 1,748; Intention 340 / 1,110 / 903; Desire 180 / 1,025 / 944; Emotion 420 / 996 / 1,306; Knowledge 290 / 1,161 / 846, for totals of 2,112 (ToMBench), 5,401 (ToMATO), and 5,747 (CogToM, out of 8,513 total items; the remaining 2,766 items are Percept/Non-Literal and are excluded from cross-benchmark analyses) analyzable items. Table 2 gives the point estimates; the 95% paired item-bootstrap CIs for the 15 pair$\times$category Stage-1 $\tau_b$ values are: Belief $[0.486,0.817]$ / $[0.438,0.814]$ / $[0.349,0.769]$, Intention $[0.528,0.836]$ / $[0.611,0.849]$ / $[0.479,0.804]$, Desire $[0.510,0.798]$ / $[0.529,0.826]$ / $[0.609,0.869]$, Emotion $[0.670,0.915]$ / $[0.681,0.917]$ / $[0.657,0.889]$, Knowledge $[0.498,0.773]$ / $[0.456,0.809]$ / $[0.542,0.871]$ (ordered ToMBench$\leftrightarrow$ToMATO / ToMBench$\leftrightarrow$CogToM / ToMATO$\leftrightarrow$CogToM throughout).

D. Per-category item retention after Stage-2 filter

Retention is more uniform on ToMATO and CogToM (70–84% across categories) than on ToMBench, where it ranges from 51.0% (Knowledge) to 76.3% (Belief). Within these five shared categories, 67.7% of ToMBench, 72.7% of ToMATO, and 75.9% of CogToM items are retained (the full-benchmark retentions are 66.4%, 72.7%, and 73.0% respectively; Figure 2). Notably, the ToMBench categories with the heaviest dropout (Knowledge, Emotion, Desire) are precisely those whose convergence is most rescued by filtering (Table 3), consistent with low-discrimination items being a primary source of pre-filter divergence. Belief fits the same pattern from the other side: it has the highest ToMBench retention (76.3%) and the smallest gain from filtering.

E. Response matrix integrity

The response matrices used throughout this audit are three model-by-item tables, one per benchmark:

• ToMBench (77,220 rows: 27 models $\times$ 2,860 items)

• ToMATO (145,827 rows: 27 models $\times$ 5,401 items)

• CogToM (229,851 rows: 27 models $\times$ 8,513 items)

Each matrix contains binary correctness scores (1/0) for all model-item pairs, with metadata columns for item categories, abilities, and benchmark source. Parse failures are coded as 0 (incorrect) in all accuracy calculations; parse-failure rates on the three ToM benchmarks are all below 3% (maximum 2.55% for Yi-1.5-34B on CogToM). Models with parse-failure rates exceeding 2% are marked with $\dagger$ in Table 1. All three matrices are reproducible deterministically from the sharded per-model JSONL outputs of the inference pipeline.

F. Stage-3 cross-fit sample loss

Because the Stage-3 cross-fit evaluates convergence only on held-out test items, each fold discards the test items that fall below the discrimination threshold when the mask is fit on the training items, and this is done independently within each of the three benchmark pairs (items from both benchmarks in a pair are pooled before folding, Section 2.3). Table 6 reports the resulting per-category sample loss, averaged within each pair over its 50 folds (5-fold $\times$ 10 seeds), together with the loss pooled across all three pairs (150 fold-results per category). Loss is broadly consistent across pairs and categories, ranging from 20.3% (Desire, ToMATO$\leftrightarrow$CogToM) to 33.4% (Emotion, ToMBench$\leftrightarrow$CogToM); pooled across all pairs and categories the overall mean is 27.3% (72,288 of 265,200 held-out item-evaluations removed). All 750 pair-category-folds ($3\text{ pairs}\times5\text{ categories}\times50\text{ folds}$) were valid; none were excluded for insufficient sample size or zero variance. Pooled sample loss does not track the size of the convergence gain as closely as ToMBench retention does (Appendix D; Table 3): Knowledge and Emotion combine high loss with large gains, but Desire has the lowest loss and the second-largest gain, and Belief has moderate loss and the smallest gain.

G. Discriminant validity: full pairwise statistics

Table 7 reports the within-ToM, within-general, and cross-construct Kendall $\tau_b$ correlations underlying Section 3.5; Table 8 reports the corresponding partial correlations for all three within-ToM pairs, controlling for MMLU-Pro, HellaSwag, and both jointly, with 95% CIs and bootstrap $p$-values for every entry.

References