Methods and further results

What the data covers

Oyez supplies case records, argument audio, speaker-labeled transcripts, and timestamps. The saved argument collection contains 1,408 arguments from October Terms 2005 to 2025. The article reports the results of that collection; it doesn’t claim that the latest term’s outcome records are complete.

The Supreme Court Database supplies ideological direction, majority membership, issue area, lower-court direction, and the winning side. Release 2025 Release 01 ends at October Term 2024. Its descriptive direction ledger contains 7,956 coded votes in divided cases. This is a larger sample than the cases that can be matched to usable argument measurements.

Petitioner votes are reconstructed from the coded winning side and majority or dissent membership. This binary representation simplifies mixed and multi-issue decisions. Ideological labels follow the Database’s codebook; they aren’t measures of motive. The argument model uses counts and timing, not the provisional legal-reasoning annotations in the explorer below.

Argument ledger, direction ledger, Oyez, and Supreme Court Database.

The question-counting comparison uses one common sample

The headline adaptation of Roberts’s rule counts substantive justice words addressed to each side. It predicts that the side receiving fewer words wins. A speaking turn is an uninterrupted contribution, so neither words nor turns are a literal count of individual questions. Rearguments are consolidated to one case outcome.

The common sample uses Oyez where its winner can be resolved and the Supreme Court Database where it can’t, within OT2005 to OT2024. Of 1,112 resolved cases, two have tied word counts and are omitted from both rules in the paired comparison. That leaves 1,110 cases:60.5% for word counting and 66.4% for always choosing the petitioner.

The paired 95% interval for the word-counting rule’s accuracy minus the petitioner rule’s is -9.9 to -1.9 percentage points. A saved sensitivity check that prefers SCDB where both sources resolve the case yields 60.5% for word counting. The headline retains its original, explicitly named source order.

Each alternative has its own usable case set. The unanimous and divided subsets, for example, can have different petitioner win rates. Their accuracies shouldn’t be compared with the full-sample baseline as though they were on one denominator.

The earlier 77% figure came from the Oyez-resolved subset. Cases dropped by that resolver had a lower petitioner win rate when matched to SCDB. The included, excluded, and combined samples, plus the term-level breakdown, remain in outcomeTestSelection. Missingness changed the apparent size of the benchmark advantage.

Correction: the ideological prediction test used an outcome-derived input

An earlier version reported that argument measurements predicted Gorsuch’s ideological votes more accurately than his voting history. That predictive interpretation is withdrawn. The model oriented its word imbalance using SCDB’s lower-court direction field. The codebook explains that this field is often constructed from the Supreme Court’s own ideological direction and whether it affirmed, reversed, or remanded.

The lower court’s actual ruling existed before argument. That doesn’t make a later field derived using the Supreme Court’s answer a valid pre-argument input. Training on earlier years can’t fix information about the test outcome entering a test feature. The earlier direction comparison, including its per-justice intervals, therefore can’t establish predictive performance before decision. A correction for testing multiple justices doesn’t address that problem either.

The original direction results remain archived. Their historical predictions were reproduced for the audit, including Gorsuch’s 185 correct votes out of 245. They aren’t evidence for the revised essay’s predictive claims. The descriptive vote map uses final vote direction as a description of outcomes, which is a separate use. The petitioner-vote models below exclude lower-court direction entirely.

The revised test adds argument to the same background model

All models are regularized logistic regressions. They produce a probability; at least one-half becomes a petitioner prediction for the accuracy score. The primary test covers 8,878 measured votes in 1,136 cases, OT2007 to OT2024. These cases represent 1,130 distinct SCDB decisions because some decisions cover related dockets.

The background model uses the justice’s previous petitioner-vote share, previous conservative-vote share, appointment party, source court, jurisdiction, and whether the United States is named as petitioner or respondent. It includes party-by-United-States interactions, previous petitioner-vote share for the justice within the source court, and historical petitioner case-win rates overall and by source court. Justice histories shrink toward the historical overall rate with ten prior observations; source-specific histories use twenty. Historical ideological share uses a symmetric ten-observation prior.

The United States indicator matches the normalized names “United States” or “United States of America.” It excludes separately named agencies, officials, and amicus appearances. Source court and jurisdiction are categorical, with categories learned only from training rows. Unseen categories receive zero on the learned category indicators. Missing party names have a separate indicator.

The argument inputs are word imbalance between sides, whole-bench imbalance, the difference between those imbalances, speaking share, speaking share relative to the justice’s earlier history, words per turn, and words per second. The combined model adds exactly these inputs to the background model. It doesn’t classify legal reasoning, tone, or persuasiveness.

The first training period is OT2005 to OT2006. Each later term is held out in sequence. Training outcomes must be dated before October 1 of the test year; every row’s history features use strictly earlier terms and decisions already available by that row’s term cutoff. Current-case ideological direction, lower-court direction, outcome, vote margin, and majority membership never enter the predictive features. Standardization uses training rows only.

The fixed primary L2 penalty is 10 on summed logistic loss, with an unpenalized intercept. Penalties 1 and 100 are reported sensitivities; the test results weren’t used to pick a winning penalty. All 360 fits converged. The saved model records include coefficients, feature names, and training means and scales.

These are retrospective records, not a timestamped reconstruction of every input as it existed before argument. The model also lacks briefs and independently coded ideological positions for both sides. It is a more detailed background comparison, not a comprehensive political or legal benchmark. Its accuracy is actually below the constant petitioner prediction, which is reported alongside it.

Probability scores and uncertainty

Log loss penalizes a wrong prediction more severely as confidence increases. Brier score is the average squared error of the predicted probability. Lower values are better for both. Adding argument to the background vote model improves log loss by 0.0269, with a paired 95% interval from 0.0151 to 0.0379. The interval remains positive when entire terms are resampled.

Argument alone also improves log loss over the historical constant vote rate:0.0317, interval 0.0216 to 0.0415. For the combined model against that constant, the log-loss interval crosses zero. The combined model’s accuracy and Brier improvements over the constant remain positive. These distinctions prevent the weak background model from being the sole benchmark.

The primary intervals use 4,000 paired bootstrap draws, seed 20261002. Each draw resamples whole SCDB decisions and retains all their dockets and justice votes. Point estimates come from the original sample. The time sensitivity resamples 18 entire terms. It preserves dependence inside each term, but doesn’t model longer serial dependence across terms.

These intervals condition on the fitted historical predictions. Models aren’t refitted inside the bootstrap, so the intervals don’t represent every source of model-development uncertainty. The analysis specifies its primary comparisons and reports the sensitivities together. Per-justice and subset estimates are exploratory and aren’t a new search for significant winners.

Two ways to predict the Court

The aggregated model computes the probability of a strict majority of the scored justices supporting the petitioner, assuming their votes are independent conditional on the measured features. It uses the available votes, which often exclude a silent justice. It isn’t a full-bench forecast. In even-sized sets a tie isn’t a petitioner majority. The accounting in Exhibit 4 compares the background and combined models on this same procedure.

The direct case models use the source court, jurisdiction, United States indicators, and earlier case-win rates, plus the means and standard deviations of the scored justices’ background features. The version with argument adds the means of the seven argument measurements and the standard deviation and median of individual word imbalance. It is trained directly on the case winner. No predicted vote probabilities enter this direct model, and no realized vote margins enter either model.

Direct case prediction changes the estimation target as well as the aggregation method. Its result is a robustness check, not an experiment identifying whether dependence among justices caused the earlier failure. Adding argument yields65.8% accuracy against 66.5% for the direct background model. The paired 95% interval for its accuracy advantage is -2.6 to 0.9 percentage points.

The direct model’s log-loss improvement over its background counterpart is0.0051, interval -0.0117 to 0.0215. Against a probability set to the petitioner win rate in earlier training cases, its log loss is worse. The deterministic “always petitioner” rule is compared on accuracy only; it isn’t treated as a claim of 100% probability.

Stability, coverage, and source quality

The vote-level gain persists in both periods and both penalty sensitivities. In OT2016 to OT2024, the direct case model also improves the probability score over its background counterpart. That period doesn’t establish a winner-accuracy advantage. The full-period case probability comparison remains inconclusive.

The issue-area sensitivity adds SCDB’s issue classification to both models. The codebook identifies issues using the Court’s own account of the decision. It is a retrospective adjustment, not verified pre-argument information, and is excluded from the primary test.

Full recorded coverage means the number of distinct scored justices equals the reported majority plus minority count. It holds in 247 test cases. The broader sample leaves out votes without side-attributed argument measurements; missing measurements aren’t imputed as silence. The transcript sensitivity requires positive word totals for both sides and no section with an unresolved side. It checks saved predictions on that subset rather than retraining the models. Unanimity and close margins are known only after decision and serve as diagnostics, never prediction inputs.

The input audit found one Oyez argument joined to two separate Medellín decisions. The new test excludes that ambiguous argument. Identical consolidated-docket records are deduplicated to one row per justice and argument; conflicting records would exclude the argument. The input audit collapsed 8 duplicate rows and found no remaining conflicting vote records. Related dockets share a decision-level resampling cluster.

Oyez-derived and SCDB-derived labels agreed on 96.0% of 9,771 overlapping historical votes in the earlier source comparison. Disagreement is a quality concern, not a ceiling on accuracy. The new vote models use SCDB outcomes throughout. The word-counting comparison retains its separately documented source order. A full independent review of every outcome hasn’t been completed.

Reproducing the revision

The research design was written before fitting the new models and amended after the source audit to remove lower-court direction. The read-only export produces a compressed, hashed snapshot of public inputs. The analysis runs locally from that snapshot using Python, NumPy, and SciPy; it needs no live database connection.

• Read-only export and input snapshot.
• Analysis, results ledger, and row-level predictions and fitted models.
• Numerical and temporal validation.
• Historical replay and reconciliation.

The earlier test contained 9,385 votes in 1,201 cases, OT2006 to OT2024. Its 119 changed case calls, 55 corrections, and 64 spoiled calls are reproducible under that specification. They aren’t the revised model’s results. The new sample begins one term later and removes ambiguous and duplicate joins. The historical results remain archived rather than being silently overwritten.

Changes in membership matter to the party-label trend

The descriptive measure asks whether a Republican appointee cast a vote coded conservative, or a Democratic appointee cast one coded liberal. Removing Stevens and Souter largely eliminates the increase across this period. That demonstrates sensitivity to membership. It doesn’t establish that no continuing justice changed, or that the Court’s broader polarization stayed constant.

The individual rates explain why those two names matter. Their appointing party was usually a poor guide to their ideological votes during the years in this sample. Historical voting records are a more informative baseline for them.

The usefulness of an appointment label varies by issue

The same descriptive rule has different accuracy across SCDB issue areas. This comparison covers divided cases since October Term 2015. It describes these samples; it doesn’t establish that the gap would persist on future cases.

Questioning similarity and voting agreement became less closely related

The relationship between how similarly justice pairs questioned and how often they voted together weakened across the measured eras. A comparison restricted to the 6 justices present in both periods also weakens. This is a descriptive association between pairwise measures. It isn’t a direct test of whether the vote-prediction model became less accurate over time.

The move to telephone arguments in 2020 and the later use of ordered questioning could affect these measures, but the analysis doesn’t isolate a cause. The validation ledger reports the alternative samples. The argument ledger also contains the separate seniority-order counts by era and term.

Correction: missing outcomes were presented as undecided cases

An earlier version of the essay presented 22 model outputs as forecasts published before the answers. That claim is withdrawn. The saved file was generated on July 1, 2026, and selected cases without a resolved outcome in its source. That condition doesn’t establish that a case was still undecided.

Two examples establish the problem. Ellingburg v. United States was decided on January 20, 2026. Urias-Orellana v. Bondi was decided on March 4, 2026. Both appear in the July file as awaiting a public outcome. These are verified examples, not an exhaustive audit of the file.

The original file is preserved as a record of the exploratory model outputs, including its incorrect status fields. It shouldn’t be used as evidence of performance on predictions made before decisions. The displayed forecast table and its early-publication claims have been removed. The historical tests described above are separate evaluations.

A future prospective test needs verified decision status, complete accounting for participating justices, a fixed protocol, and a timestamped prediction published before the decision. It must then be scored against independently checked outcomes. No such prospective score is claimed here.

Read the source turns

This explorer holds 115 turns from the two arguments quoted in the essay, covering three justices. It lets you inspect the text behind the examples. Its phrase-rule categories are provisional navigation aids and aren’t inputs to the predictive models.

A blinded second pass by another AI model agreed with the provisional categories on 1 complete label pair out of 30 sampled turns. No human validation is recorded. Neither model’s categories can support a claim about a justice’s legal reasoning on that evidence.