Home / Tests & data

What the data actually says

A theory that cannot fail is not a theory. So the framework was turned into pre-registered falsifiers and run against real data: Reddit comment dumps (billions of rows), Wikipedia edit histories, and GitHub repositories. Thresholds were frozen before the data was harvested. The headline result is a split exactly in the shape the theory predicts: the structural gear seals as a pass, while three more ambitious claims honestly deflate. Below is every verdict, with its real numbers and figure.

How these tests are kept honest

The rules of engagement

Five rails, applied to every test on this page

  • Pre-registration with frozen thresholds. Every pass/fail rule is committed to a file before the data is harvested (the "FA-0 seal"). The decision is evaluated exactly once. We never move a threshold after seeing the numbers — that single rule is the program's whole integrity premise.
  • Nulls against the prosecutor's fallacy. A signal must beat a matched null: a block-label shuffle (does the effect depend on the specific communities, or any grouping?) and a matched calm window (does it happen more at a real event than in a quiet period on the same object?).
  • Look-ahead-free. Onsets are fixed from public event dates before any graph is built; filters run strictly causally; no future information enters a forecast.
  • Proxy-data humility. These run on mention-density and interaction-graph proxies, not a calibrated social state. They are illustrative of direction, magnitude and mechanism on real rosters — not a deployed classifier.
  • Honest negatives reported straight. When a claim fails, it is printed in full and kept. Three of the four powered second-wave tests deflate; that deflation is the most important empirical news here, because it is the exact pattern a bounded psychohistory would produce.
The falsifier scorecard

Every pre-registered bet, and where it stands

The paper names eight falsification tests plus a sharpening of the block test. Five now carry powered runs against real data; four are pending a live world-model training run (blocked on compute, not on more data).

TestThe betVerdictKey number
(ii′)Dynamic Neff collapse is community-specific before endogenous cascadesSEALED PASS9/12 fire, binomial p = 1.7×10⁻⁷, fresh roster
(iii)Early warning (critical slowing-down) precedes cascadesPARTIALbeats calm null p = 0.02; can't tell endo/exo (AUC 0.60)
(iii′)A substantive fraction of cascades are slow B-tippingREFUTEDB-fraction 0.33 < π_B = 0.60; mostly R-tipping shocks
(i)Attention / activity is conserved (zero-sum) at ecosystem scaleCONTRADICTEDfinance-subreddit basket ballooned ~14× in the mania
(ii)Blocks are independent in calm windowsWEAK SUPPORTmacro Neff 1.90 of 8, bottoms 0.47 at a real shock
(iv)Forecast skill in the smooth regime beats baselinesPENDINGawaits a live world-model training run
(v)Published fixed points are reliablePENDINGawaits a live world-model training run
(vi)Lucas invariance: drift stays in an absorbable bandPENDINGworld-model training run + multi-regime calibration
(vii)Regime occupancy (Soros): imitative < monotonePENDINGworld-model training run + live regime monitor

Cross-cutting checks also ran: the GameStop counterfactual (overdetermination), the operator-signal detector, the GitHub cross-domain replication, and the EnKF forward forecast.

The four pending tests are blocked on compute, not data

Tests (iv)–(vii) are the forward-forecasting falsifiers. They do not need more data; they need a live forecasting engine run forward: the modern instantiation (a trained world model plus an LLM/LRM ensemble). The binding constraint is a stronger world-model training run — a compute problem, not a data problem. This is independent research, and compute donations directly unblock these four tests. The paper is v0.5, in review; it reaches v1.0 when these turn green. To donate compute or collaborate, contact the author at wingston.sharon@gmail.com.

The headline result · test (ii′)

The criticality gear's prediction, sealed on fresh data

This is the load-bearing test: the whole criticality account turns on the claim that the effective number of independent blocks collapses across an onset. The honest story has a twist that took six runs to find, and it is a lesson in not testing the wrong quantity.

test (ii′)

Dynamic Neff collapse — community-specificity

SEALED PASS

The theory's real claim is specificity: before an endogenous cascade, the existing community's frozen block partition collapses harder than a random reshuffle of the same people. We pre-registered that as the primary endpoint, with a frozen binomial rule, on a fresh roster of 12 r/wallstreetbets cascades disjoint from every prior run (COVID crash, Archegos, Coinbase, the NVDA earnings prints, Credit-Suisse, the 2024 election, and more). It passed cleanly.

9 / 12
cascades collapse past their own block-label shuffle null
1.7×10⁻⁷
binomial p against the no-structure rate (p₀ = 0.10)
100th pct
median cascade beats all 300 reshuffles of its own nodes
×4
independent confirmations of community-specificity
Per-cascade N_eff collapse vs block-label shuffle, fresh WSB roster
The sealed primary endpoint (validation/neff_v4/). Left: each cascade's canonical-Neff collapse against its own block-label-shuffle 90th percentile (dots); green bars fire, red do not. Right: the observed collapse sits at the very top of its own 300-shuffle distribution for most cascades. The three non-firing cascades are the mechanical / exogenous events (a direct listing, a single Fed rate decision, a stock split) — exactly where a frozen-block Neff should be silent.

§ Why this is an honest pass, not a manufactured one. We did not relax the magnitude threshold an earlier run failed (that verdict stands, see below). We tested a different, theory-correct endpoint on fresh data. The September-2024 stimulus case clinches the logic: its raw collapse is only 0.065 — any magnitude bar would discard it — yet it beats every one of its 300 shuffles, because the signal lives in the community structure, not the magnitude.

The twist: why magnitude was the wrong yardstick

test (ii′) · sealing attempt #2

The raw-magnitude reading — non-discriminating

REPORTED, NOT GATING

Two earlier sealed runs tested whether the collapse magnitude exceeds a threshold derived from genuinely-quiet windows. They failed, and the failure is informative: it is why we switched to specificity.

~0.10
median Neff drop in genuinely-quiet WSB windows (tail to 0.43)
0.138 < 0.394
fresh-event median below the honest magnitude bar
9 / 10
but specificity fired anyway (third confirmation)
neff_v3: event collapse vs clean null distribution
Sealing attempt #2 (validation/neff_v3/). The decisive discovery was in the null itself: short high-volume onset windows compress Neff generically, so quiet windows already drop a median ~0.10. Magnitude therefore cannot tell an endogenous cascade from a busy-but-quiet week on a continuously high-volume forum. The specificity gate (does the real partition beat a shuffle?) discriminates where magnitude cannot.

§ The reconciliation. The dynamic collapse is a real structural signal that lives in the block partition (the shuffle test sees it four times) but is not a magnitude excursion beyond a quiet baseline — which the near-decomposability premise never required. Both halves are reported. The full six-pass arc and the synthesis are in validation/NEFF_COLLAPSE_SYNTHESIS.md.

The honest deflations

Three more ambitious claims, and how they fared

The structural gear seals. The dynamical, predictive and conservation claims do not — and that is the thesis, measured: the impersonal machinery is real but load-bearing only on the endogenous, reflexive minority of episodes, and most real cascades sit outside it.

test (iii)

Semantic early warning

POWERED PARTIAL

Does critical slowing-down (rising variance / autocorrelation) precede a cascade? With a semantic (embedding-based) observable rather than a scalar proxy, the detector beats a guard-banded calm null — but it cannot tell a genuine reflexive build from an exogenous shock.

p = 0.02
beats a guard-banded calm null; 5/5 endogenous above calm
AUC 0.60
endo-vs-exo separation — it detects "a build," not which kind
Early-warning ROC: endogenous vs exogenous separation
The powered early-warning battery (validation/early_warning_powered/). The signal is real against a calm baseline but the endogenous-vs-exogenous ROC sits near AUC 0.60 — a partial positive, not a discriminator.

§ This qualifies the paper's earlier headline negative: with a scalar proxy the signal washed out; with a vector observable, part of it survives. It does not overturn it — single embedding model, in-sample thresholds.

test (iii′)

Bifurcation-mix conjecture

REFUTED

Early-warning theory only works for slow bifurcation (B-) tipping. The conjecture — named in advance as the most likely to fail — was that a substantive fraction of real cascades are B-tipping. On a 24-cascade labelled roster, they are not.

0.33 < 0.60
B-tipping fraction vs the pre-registered π_B
24
labelled cascades; most are sudden R-tipping shocks

§ The bet failed honestly. Most real cascades arrive as sudden rate-induced shocks with no slow warning — which is exactly why early warning (iii) is only a partial positive. The two results are consistent.

test (i)

Conservation at ecosystem scale

CONTRADICTED

Is activity zero-sum across a basket of related communities? At the basket scale, no: a finance/meme-subreddit total ballooned roughly fourteen-fold through the GameStop mania. Attention conservation holds as a local, normalized-measure statement, not as a global head-count budget across a porous boundary.

~14×
9-subreddit finance/meme basket total through the mania
+1290%
incumbent-only growth; churn 40%
Ecosystem activity ballooning through the mania
The ecosystem conservation test (validation/conservation_ecosystem/). The basket is porous, so this cannot test the global claim — but at the scale measured, conservation is contradicted, matching the single-subreddit pilot.

§ Honest scope: a porous basket cannot falsify the global zero-sum claim. What it shows is that the conserved object is a normalized measure over a closed population, not raw activity across an open ecosystem.

Cross-cutting checks

Overdetermination, operators, replication, and a forward forecast

GameStop counterfactual

Structural overdetermination

SUPPORTED

Was GameStop a Seldon Crisis (structurally overdetermined) or a Mule (one individual moved it)? At the coarse scale it reads as overdetermined: attention rose structurally before the flagship spiked, and the whole meme basket fired in one window.

6 / 6
meme tickers peaked in the same week
×5.99
WSB attention build before GME spiked
×970
GME mention-density amplification at peak
GameStop attention build and multi-ticker co-peak
The counterfactual backtest. Honest caveat: the basket was selected on the outcome, and the co-peak is partly endogenous contagion — so this is suggestive structural priming, not a proof of any single operator's dispensability.
operator-signal detector

The major-player signature discriminates

DISCRIMINATES

An agent of non-negligible measure (an "operator") needs a first-class state. The detector separates a gradual internal buildup from a sudden external shock.

13 weeks
DFV / Roaring Kitty operator ramp into the squeeze
+3.54
operator-led score vs 2–3 weeks for macro shocks
Operator-signal detector: DFV buildup vs macro shocks
The operator-signal detector. Gradual internal buildup (operator) vs sudden external anticipation (macro shock) — the discriminator that does not wash out across the roster.
cross-domain replication

GitHub: 2 of 3 invariants travel

2 / 3 REPLICATE

Do the findings survive a different platform? On GitHub repositories, the structural-overdetermination and impersonal-early-warning-weakness results replicate, and operator concentration generalizes — but the gradual-buildup timing is platform-specific.

2 / 3
invariants reproduce cross-domain
88%
langchain founder commit share (concentration generalizes)
weeks
GitHub repos ignite faster — timing is platform-specific
GitHub cross-domain replication
The cross-domain replication. The key finding: structural and operator-concentration invariants are domain-general; timing is platform-specific.
forward forecast

EnKF walk-forward — an honest negative on the strong claim

CALIBRATED, NOT SKILLFUL

The first strictly-causal forward forecast on a real block. It beats climatology and is best-calibrated, but does not beat a persistence (last-value) baseline. The load-bearing positive is the self-diagnosing monitor firing on a real regime break.

0.181 vs 0.224
EnKF RMSE beats climatology; 95% coverage 0.98
0.181 vs 0.179
ties persistence — does NOT beat last-value
z = −3.33
monitor flagged the Apr-2025 regime break in real time
EnKF walk-forward and misspecification monitor
The EnKF forward test. The forward-skill claim survives only in the weaker form (better than climatology, calibrated, self-diagnosing). The monitor firing on the out-of-model event is the "Mule"-detection capability.
Coverage dashboard · r/AskEconomics top 100

Where the questions actually live

A different kind of empirical check: route the 100 most-discussed r/AskEconomics questions through the engine and see which layers fire. The distribution is the test of the thesis — and it confirms it by measurement. The dramatic machinery (criticality, reflexivity) is a rare special regime; the quiet slow-stock core does almost all the work.

Scope verdicts

Layer activation frequency

The dramatic machinery (criticality ≈ 21, reflexivity ≈ 34) is a special regime. Most questions are slow-stock accounting (L1 ≈ 93) — exactly what the thesis predicts: the quiet core dominates, the fat tails are rare.

All 100 readings

#TitleLayersScopeScore
Limits & provenance

What this does not claim

Read it whole in the paper (PDF), or revisit the mathematics behind each test.