Live boardHow it works ICML 2026 resultsFor judgesEarlier runsGitHub

PaperCI on ICML 2026 papers

The answer key is Hugging Face's ICML 2026 Open Reproductions: community teams reproduced papers claim by claim, and a GLM-5.2 judge graded their logbooks. We kept only papers where at least two medium- or high-quality logbooks agreed (75% or more) that the paper was verified or falsified. PaperCI runs each one blind: no agent ever sees the community verdict. Builder model: factory:deepseek-v4.1-flash; verifier model: factory:glm-5.3-flash.

Automated result: PaperCI ran on a 2-vCPU CPU-only machine without internet access during runs, so it may fail or disagree for reasons unrelated to the paper. It is not a judgement of the authors' work. The community labels come from an LLM judge, so they can be wrong too. "No decision" means PaperCI could not build or measure the paper within its limits. "Reduced scale only" means the code ran and was measured on a smaller setting (fewer steps, seeds or data) that fits a CPU; it is evidence that the code works, not a verdict on the paper's full-scale numbers.

18papers run blind
11built and run offline
0%reached a full-scale verdict
-agreement when it did
9 · 8/22reduced-scale runs · claims matched

Pass 2 reruns the same blind papers with the agent after its overnight self-improvement (it never learned from these papers). Pass 1 for comparison: 18 run, 11 built and run offline, 7 reduced-scale runs, 8/16 reduced-scale claims matched, full-scale verdicts 0.

A Unifying View of Variational Generative Wasserstein Flows arXiv 2605.31369 · Deep Learning

Hugging Face community: did not reproduce (100% of 4 independent logbooks; 1 2 3 4) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status max_turns: no submit after 18 turns; harness auto-submit failed the offline dry run: REJECTED: the offline dry run failed (network is disabled during the real run). exit=124 (720.3s) · build max_turns, 18 agent turns
Claims and evidence from the community reproductions
  • The Generative Wasserstein Flows (GWF) framework shows that Variational Wasserstein Gradient Flow (VWGF) and Scalable JKO (S-JKO) objectives, despite different formulations, are mathematically equivalent (Proposition 3.1). verified x4
    [verified] A proof certificate showing minimax ordering forces A=D via nonnegative gaps summing to zero, plus continuous 2-D anisotropic Gaussian checks matching VWGF and S-JKO objective values to absolute errors of 2.09e-14, 3.55e-15, and 4.44e-14 across three tau values. logbook
  • The paper shows explicit Wasserstein gradient descent with a regularized f-divergence is equivalent to JKO steps for the corresponding unregularized f-divergence under specific parameter relationships (Proposition 3.2). verified x4
    [verified] SymPy finds the unique coefficient match gamma=1/2 and epsilon=2tau with zero residual, and continuous 2-D Gaussian Moreau envelope checks for KL, chi-squared, and Jensen-Shannon show errors of 1.08e-8, 1.25e-8, and 1.02e-8 at gamma=1/2 versus 0.02-0.09 at gamma=0.35. logbook
  • The paper introduces a Donsker-Varadhan formulation for the KL divergence within the JKO scheme that yields tighter lower bounds than classical variational representations while preserving equivalence at optimality (Section 3). verified x4
    [verified] SymPy certifies the scalar inequality exp(s-1)-s >= 0 exactly (minimum 0 at s=1, positive second derivative, infinite boundary limits), and 60 continuous off-optimum Gaussian cases all show nonnegative DV-classical gaps with smallest gap 0.0178 and equality error 8.33e-17 at optimal critics. logbook
  • The GWF framework is extended beyond f-divergences to integral probability metrics, including Wasserstein-1 and squared Maximum Mean Discrepancy (MMD²), introducing a JKO-regularized version of MMD-GAN that does not require explicit discriminator optimization falsified x4
    [falsified] The paper's own Equation 18 is max_phi min_T 1/2||Id-T||^2 + MMD^2_{k_phi}(T#mu_l, nu), which explicitly optimizes over phi (discriminator); Algorithm 2 performs gradient ascent on phi and the authors' code calls optimizerD.step(). A point-mass counterexample with PSD kernel k_phi(x,y)=phi^2*xy conf logbook
  • Experiments on MNIST and CIFAR-10 measured by Fréchet Inception Distance (FID) show JKO regularization gives moderate but consistent improvements for f-divergences (KL, Jensen-Shannon, χ²) and more pronounced gains for MMD, up to 20 FID points (Section 6/7 exp falsified x4
    [falsified] The paper's own reported CIFAR-10 table shows Jensen-Shannon no-JKO FID of 14.60 beating all JKO step sizes (best JKO 15.23), a worsening of 0.63 FID, and a different-seed table repeats the counterexample (no-JKO 16.84 vs best-JKO 16.87), directly contradicting 'consistent improvements for f-diverge logbook
  • Parametric JKO updates are shown to approximate preconditioned Wasserstein gradient flows in the small-step-size regime (τ→0) (Proposition 6.3). verified x4
    [verified] A general Frechet-remainder certificate derives G(theta_plus)*Delta/tau = -grad energy + o(1), and a 2-D nonlinear parametrization F_(a,b)(z)=z+(a+0.35ab, 0.45a+b+0.15a^2) with non-trivial metric tensor (min eigenvalue 0.2468, off-diagonal >= 0.7283) shows theorem residuals converging at log-log slo logbook
What PaperCI measured
  • no metric was measured

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks arXiv 2511.04689 · General Machine Learning

Hugging Face community: did not reproduce (83% of 6 independent logbooks; 1 2 3 4 5) · PaperCI: inconclusive · no decision · full PaperCI review
PaperCI's reason: integrity flags: no_repo_code_path · build built, 15 agent turns
Claims and evidence from the community reproductions
  • ATLAS reduces the number of required benchmark items by up to 90% relative to full static benchmarks while maintaining measurement precision of model ability estimates (Section 1, abstract claim substantiated in-text). verified x6
    [verified] Independent catR run on HellaSwag (386 models, 5600 items) achieved 99.26% item reduction with MAE 0.163 vs paper's 0.154, supporting the ≥90% reduction claim with maintained precision on a non-toy setup. logbook
  • On HellaSwag (5,608 total items), ATLAS matches whole-bank ability estimates using only 42 adaptively selected items, achieving 0.154 MAE against the full-bank estimate (Table 1). verified x2, falsified x4
    [verified] catR run on HellaSwag used mean 41.17 adaptive items (≈42 claimed) with MAE 0.163 vs paper's 0.154 (gap 0.009), closely matching the claimed operating point on real response data. logbook
  • Across the benchmark suite in Table 1, MAE varies by task: 0.067 for TruthfulQA, 0.099 for ARC, 0.149 for WinoGrande, 0.154 for HellaSwag, and 0.159 for GSM8K (Table 1). verified x6
    [verified] All five benchmarks reproduced within ±0.05 of paper Table 1 MAE values: TruthfulQA 0.0645 vs 0.067, ARC 0.0867 vs 0.099, WinoGrande 0.142 vs 0.149, HellaSwag 0.163 vs 0.154, GSM8K 0.150 vs 0.159, with matching ordering across the suite. logbook
  • Among more than 4,000 evaluated models, 23-31% shift by more than 10 rank positions when models are ranked by IRT-based ability estimates versus by raw accuracy (Section 4.4, Figure 2). toy x1, falsified x5
    [toy] Rank-shift computed on only 386-521 models per benchmark (vs claimed >4,000), finding 66-84% shift rates (vs claimed 23-31%); direction confirmed but specific magnitude and population scale not reproduced. logbook
  • ATLAS keeps item exposure rates below 10% and test overlap between model evaluations to 16-27%, compared to the 100% item exposure inherent to static benchmarks, while Section 5.1 finds 3.2-5.7% of items exhibit negative discrimination indicating annotation er toy x1, falsified x5
    [toy] Exposure/overlap computed on HellaSwag only (one of five benchmarks): mean exposure 3.85% supports <10%, but max exposure 40.2% exceeds it; overlap 11.1% is below claimed 16-27% band; negative discrimination 9.93% exceeds claimed 3.2-5.7%. logbook
What PaperCI measured
  • ability estimation MAE: paper 0.157, PaperCI measured 0.168689 (n=3, within tolerance: True)
  • ability estimation MAE: paper None, PaperCI measured None (n=None, within tolerance: None)
  • accuracy reconstruction MAE: paper None, PaperCI measured None (n=None, within tolerance: None)
  • split-half ranking stability (Spearman correlation, ability-based): paper None, PaperCI measured None (n=None, within tolerance: None)
  • end-to-end runtime per model: paper None, PaperCI measured None (n=None, within tolerance: None)

Are Two Datasets Close Enough With Statistical Significance? A Kernel Distributional Closeness Testing Approach arXiv 2507.12843 · General Machine Learning

Hugging Face community: did not reproduce (80% of 10 independent logbooks; 1 2 3 4 5) · PaperCI: toy-scale · reduced scale only (3/3 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (Used the repository's own self-contained synthetic Gaussian-mixture DCT demo (demos/mixture_gaussian_dct.py) instead of the full Table 2 benchmark sweeps over b); 3 metric(s) measured · build built, 10 agent turns
Claims and evidence from the community reproductions
  • NAMMD is defined by normalizing squared MMD by the RKHS norms of the two compared distributions, producing a bounded value in [0,1] that increases as the underlying distributions become more concentrated (Section defining NAMMD). verified x9, toy x1
    [verified] The implementation computes NAMMD = MMD²/(4K−||μP||²−||μQ||²), confirms all five population controls lie in [0,1], and demonstrates the concentration effect by holding MMD² fixed at 0.12 while decreasing Gaussian std from 1.5 to 0.25, yielding a monotonic NAMMD increase from 0.038129 to 0.056754. logbook
  • Theorem 4 proves that the NAMMD-based distribution closeness test maintains type-I error bounded by the nominal level alpha (Theorem 4). verified x8, falsified x1, toy x1
    [verified] Across 800 independent P=Q trials with 45 observations per group and 199 fresh permutations each, NAMMD rejects 39 times (4.8750%) at α=0.05, with a two-standard-error interval of [0.0335, 0.0640] that is consistent with type-I error control. logbook
  • Theorem 7 proves that whenever an MMD-based distributional closeness test correctly rejects the null hypothesis, the NAMMD-based test also rejects with high probability, and gives examples where NAMMD rejects while MMD-based DCT fails (Theorem 7). verified x8, falsified x1, toy x1
    [verified] In 600 trials satisfying the theorem's norm condition (0.4640 < 0.8274), all 539 MMD rejections are also NAMMD rejections (0 MMD-only), and NAMMD additionally rejects 45 cases (power 0.973 vs. 0.898), directly demonstrating both inheritance and NAMMD-only rejection. logbook
  • Theorem 5 shows the sample complexity of the NAMMD-based test scales as 1/(NAMMD(P,Q;kappa) - epsilon)^2 (Theorem 5). verified x9, toy x1
    [verified] The exact theorem bound was evaluated at gaps [0.05, 0.1, 0.2], yielding required sample sizes [28778.832, 7194.708, 1798.677]; the product gap² × bound is constant at 71.947, confirming inverse-square scaling. logbook
  • Across five benchmark datasets (blob, higgs, hdgm, mnist, cifar10), NAMMD attains higher average test power than MMD-based DCT, e.g., 0.976 vs. 0.973 at epsilon=0.1 and 0.989 vs. 0.970 at epsilon=0.7 (Experimental results section). falsified x7, toy x2, inconclusive x1
    [falsified] The ε=0.1 numbers match (NAMMD 0.976 vs. MMD 0.973), but at ε=0.7 the paper's own table TAB:same_mmd reports NAMMD as 0.984, not the claimed 0.989, contradicting the specific numeric assertion. logbook
  • The method is applied without ground-truth labels to three case studies: ImageNet distribution-shift variant classification, confidence-margin detection, and adversarial perturbation quantification on CIFAR-10 (Case studies section). verified x4, falsified x2, inconclusive x2, toy x2
    [verified] All three case studies were executed on an NVIDIA L4 with real public datasets: ImageNet shift variants (5×500 images, NAMMD values from 0.012 to 0.033), confidence-margin detection (ImageNet vs. ImageNet-v2 split into thirds with NAMMD per third), and CIFAR-10 adversarial quantification (1,200 imag logbook
What PaperCI measured
  • test power: paper 1.0, PaperCI measured 1.0 (n=3, within tolerance: True)
  • test power: paper 1.0, PaperCI measured 1.0 (n=3, within tolerance: True)
  • test power: paper 1.0, PaperCI measured 1.0 (n=3, within tolerance: True)
  • NAMMD value: paper None, PaperCI measured None (n=None, within tolerance: None)
  • runtime: paper None, PaperCI measured None (n=None, within tolerance: None)

CodeTaste: Can LLMs Generate Human-Level Code Refactorings? arXiv 2603.04177 · Applications

Hugging Face community: did not reproduce (100% of 3 independent logbooks; 1 2 3) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status gave_up: needs_credentials · build gave_up, 4 agent turns
Claims and evidence from the community reproductions
  • CodeTaste comprises 100 refactoring instances mined from 87 GitHub repositories across 6 programming languages, with an average of 91.52 files and 2,605.39 lines changed per task (Section 3.5, Table 1). verified x3
    [verified] The independent audit script parsed all 100 golden.diff files from the released v1.0.0 artifact, yielding instances=100, unique_repositories=87, languages=6, and golden means of files_changed=91.52 and lines_edited=2605.39, all matching the claim exactly. logbook
  • On the Instructed Track, GPT-5.2 achieves a 69.6% alignment score and 76.0% functional-correctness (Pass) rate, versus 32.4% alignment and 47.0% Pass for Claude Sonnet 4.5, and 11.8% alignment for Qwen3 (Section 4.2, Table 3). falsified x3
    [falsified] Recomputed metrics from 2000 released evaluation results show GPT-5.2 at 69.6% alignment and 76.0% PASS (matching), but Claude Sonnet 4.5 at 32.1% alignment and 43.0% PASS (claim says 32.4% and 47.0%) and Qwen3 at 10.7% alignment (claim says 11.8%), contradicting the specific values in the claim. logbook
  • On the Open Track, where agents see only a high-level focus area instead of a detailed specification, the best alignment score collapses to 7.7% (GPT-5.2, Direct mode) (Section 4.3). falsified x3
    [falsified] GPT-5.2 Open Direct alignment from released outputs is 7.93% (rounded to 7.9%), not the claimed 7.7%; while the qualitative collapse below 8% is supported, the specific numerical value is contradicted. logbook
  • A propose-then-implement (Plan mode) decomposition raises GPT-5.2's Open Track alignment from 7.7% to 14.1%, and Oracle multiplan selection raises it further to 19.4% (Section 4.3). falsified x3
    [falsified] Released outputs show GPT-5.2 Open Track alignment of 7.9% (direct) → 15.1% (plan) → 20.6% (multiplan), whereas the claim asserts 7.7% → 14.1% → 19.4%; the directional improvement is confirmed but all three specific values are contradicted. logbook
  • Static-rule evaluation uses an average of 93.07 additive/reductive OpenGrep rules per task (30 additive, 63 reductive) to measure Instruction Following Rate alongside test-suite Pass rate (Section 3.4, Table 1). falsified x3
    [falsified] Counting YAML rule objects across all 100 instances gives 17.94 additive, 63.41 reductive, and 81.35 total rules per task on average, contradicting the claimed 30 additive, 63 reductive, and 93.07 total; the reductive count is close but additive and total are substantially off. logbook
  • GPT-5.2's superior alignment comes at roughly $5.17 per task in agent inference cost, compared to $3.46 for Claude Sonnet 4.5 and $0.59 for GPT-5.1 Codex Mini (Section 4.4). verified x3
    [verified] Released Instructed Direct metadata gives cost_mean of $5.1747 for GPT-5.2, $3.4634 for Claude Sonnet 4.5, and $0.5943 for GPT-5.1 Codex Mini, matching the claimed $5.17, $3.46, and $0.59 per task. logbook
What PaperCI measured
  • no metric was measured

Cognitive Fatigue in Autoregressive Transformers: Formalization and Measurement arXiv 2605.30981 · Deep Learning

Hugging Face community: did not reproduce (100% of 2 independent logbooks; 1 2) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status max_turns: no submit after 18 turns · build max_turns, 18 agent turns
Claims and evidence from the community reproductions
  • The Fatigue Index is defined as a weighted combination FI_t = 0.40·φ_A(A_t) + 0.35·φ_E(E_t) + 0.25·φ_D(D_t) of prompt-attention decay, embedding drift, and entropy-collapse signals computed during autoregressive decoding (Section 3, Section 4). verified x2
    [verified] 175 test points confirm boundedness (FI ∈ [0,1]), monotonicity in all three components (e.g., FI(0.2)=0.37 ≥ FI(0.8)=0.13 for attention), weights sum to 1.0 (0.40+0.35+0.25), and the formula computes correctly (FI(0.5,4.0,20.0)=0.30). logbook
  • The Fatigue Index's predictive validity for downstream task degradation is measured at ρ = 0.848 on HotpotQA, ρ = 0.820 on TriviaQA, and ρ = 0.856 on SQuAD (Table 3). falsified x2
    [falsified] On OPT-2.7B (the paper's model) with 1000 real HotpotQA and 500 real SQuAD samples, ρ = -0.340 and ρ = 0.201 respectively, dramatically below the claimed 0.848 and 0.856, with HotpotQA showing a sign flip; only TriviaQA (ρ=0.755) is close to the claimed 0.820. logbook
  • The Fatigue Index achieves an AUROC of 0.976 for detecting repetition and severe degeneration on HotpotQA (Table 5). falsified x2
    [falsified] On OPT-2.7B with 1000 real HotpotQA samples, FI AUROC = 0.247 vs. the claimed 0.976, and the component ordering is inverted (entropy 0.386 and attention 0.382 outperform the aggregated FI, contradicting the paper's key finding that FI outperforms all components). logbook
  • Instruction-tuned models below 3B parameters (e.g., TinyLlama-1.1B, OPT-1.3B) exhibit faster fatigue-driven collapse than base models, with this trend reversing at the 7B scale (Falcon-7B), as shown by non-monotonic scaling in Figure 6 (Section 7.5). falsified x2
    [falsified] A 5-model sweep on 120 real HotpotQA samples each confirms the sub-3B effect (TinyLlama-1.1B-Chat mean entropy 0.748 vs OPT-1.3B 1.188) but contradicts the 7B reversal: Mistral-7B-instruct (0.618) collapses more than Mistral-7B base (1.266), showing no reversal at 7B scale. logbook
  • Hysteresis-based alerting (activation threshold θ=0.50, deactivation θ_low=0.40) reduces false-positive alert rates by 91-93% compared to a static-threshold alerting scheme (Table 4). falsified x2
    [falsified] On OPT-2.7B with 2000 real generations, hysteresis reduces flips by 48.8% (1.982→1.015 flips/gen), well below the claimed 91-93%; the mechanism works directionally but the specific quantitative claim is contradicted. logbook
  • Stress tests show front-positioned evidence in the prompt receives 5-10x higher mean attention than middle- or end-positioned evidence, and 4-bit quantization increases entropy-collapse depth relative to FP16 (Figure 3, Figure 4, Section 5.2). verified x1, falsified x1
    [verified] Front-positioned evidence receives 6.9x (front/middle) and 8.5x (front/end) higher mean attention, within the claimed 5-10x range; 4-bit NF4 quantization yields lower mean entropy (4.09 vs 4.73 FP16), confirming increased entropy-collapse depth. logbook
What PaperCI measured
  • no metric was measured

Differentially Private Range Subgraph Counting arXiv 2606.08179 · Social Aspects

Hugging Face community: did not reproduce (100% of 3 independent logbooks; 1 2 3) · PaperCI: toy-scale · reduced scale only (0/2 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (); 2 metric(s) measured · build built, 18 agent turns
Claims and evidence from the community reproductions
  • Theorem 1.3 gives the first efficient approximate-differential-privacy algorithm for range subgraph counting, with additive error O(HS̃_fH(G) · sqrt((ε+log(1/δ))·log(n|Q|)) · log^(2d)n / ε) (Theorem 1.3). verified x1, inconclusive x1, falsified x1
    [verified] Axiom-free Lean theorems compile for arbitrary-parameter utility and privacy composition matching the Theorem 1.3 error bound, and 12 executed ADP_RSC cells on CA-Netscience (143,641 queries) show finite positive local-sensitivity scales with ADP_RSC beating ADP_Comp by at least 8.1×. logbook
  • Theorem 1.4 proves a lower bound showing additive error must scale as Ω(log^(d-1)n · GS_fH) for constant dimension d, growing to n^Ω(1) · GS_fH when d = Ω(log n), i.e. exponential in the dimension (Theorem 1.4). verified x1, inconclusive x1, falsified x1
    [verified] Axiom-free Lean theorems compile for all three lower-bound regimes (constant d, d=O(log n), d=Ω(log n)) with frozen source lineage through the discrepancy lemma, reduction, and reconstruction contradiction; reversed transfer directions fail compilation, confirming the proof structure. logbook
  • Algorithm 3 (PDP_RSC) achieves pure differential privacy for range subgraph counting with additive error O(GS_fH · sqrt(log(n|Q|)) · log^(3d)n / ε), built on a range-tree construction (Algorithm 2, TreeConst) and a subgraph-occurrence projection (Algorithm 1, verified x3
    [verified] Lean proves the projection-count identity for arbitrary occurrence lists, 480/480 direct-count audits match projection answers, and 12 PDP_RSC cells verify pure-DP sensitivity bounds with destructive mutations (halving scale, removing Proj, zero noise) all caught. logbook
  • Algorithm 5 (ADP_RSC) attains the approximate-DP guarantee of Theorem 1.3 by combining private higher-order local sensitivity estimation (Algorithm 4, EstimateHS) with the range-tree mechanism (Section 3, Algorithm 5). verified x1, falsified x2
    [verified] Axiom-free Lean theorems compile for EstimateHS-to-range-tree utility and (m_H+1)ε′=ε privacy composition; 12 executed cells show the triangle scale at ε=2 is 179138.436 (distinct from PDP global-sensitivity scale), and substituting global sensitivity is caught. logbook
  • On the Wiki-Squirrel (5,201 nodes, 198,353 edges), WormNet-v3 (16,347 nodes, 762,822 edges), and CA-Netscience (379 nodes, 914 edges) graphs, the proposed PDP_RSC and ADP_RSC algorithms are reported to outperform PDP_Comp and ADP_Comp baselines by 3 to 4 order falsified x3
    [falsified] Exhaustive evaluation of all 294,418,451 distinct n² queries across three graphs shows ADP accuracy gaps of only 0.96, 1.77, and 2.18 orders (all below 3), and PDP gaps of 2.54, 4.34, and 5.22 orders (not uniformly 3–4); the paper's literal '3 to 4 orders' phrase is in its runtime paragraph, not acc logbook
What PaperCI measured
  • query-time speedup vs baselines: paper 3.0, PaperCI measured 2343.333333 (n=3, within tolerance: False)
  • total-time speedup vs baselines: paper 4.0, PaperCI measured 18.846667 (n=3, within tolerance: False)

Efficient Inference for Noisy LLM-as-a-Judge Evaluation arXiv 2601.05420 · General Machine Learning

Hugging Face community: did not reproduce (100% of 2 independent logbooks; 1 2) · PaperCI: inconclusive · no decision · full PaperCI review
PaperCI's reason: integrity flags: no_repo_code_path · build built, 16 agent turns
Claims and evidence from the community reproductions
  • The efficient influence function (EIF) estimator, the maximum likelihood estimator (MLE), and optimally-tuned PPI++ all attain the same asymptotic variance, which is strictly smaller than that of vanilla PPI, which in turn is strictly smaller than that of the verified x2
    [verified] Independent SymPy symbolic audit establishes exact identities V_MLE−V_EIF=0, V_PPI++−V_EIF=0, V_PPI−V_PPI++ as a strictly positive square, and V_RG−V_PPI with positive-form proof, confirmed by a 10,000-point numerical audit with zero violations. logbook
  • For binary outcomes, optimally tuned PPI++ matches the efficiency of the EIF estimator because the regression function E[Y|Ŷ] is necessarily affine, so no further efficiency gain is possible from nonparametric calibration (Section on efficiency comparison). verified x2
    [verified] For binary Ŷ, μ(Ŷ)=E[Y|Ŷ] is determined by two values and hence affine; independent symbolic calculation yields the exact identity V_PPI++−V_EIF=0, confirming equal asymptotic variance. logbook
  • In simulations with N=2,000 and 1,000 replicates on binary outcomes, when judge quality is low (q0+q1=1.2), Rogan-Gladen confidence intervals are roughly 10x wider than EIF/PPI++ intervals, and EIF/PPI++ intervals are 35-55% narrower than vanilla PPI intervals falsified x2
    [falsified] Four independent full-scale seed panels (144,000 attempted splits, 9 θ values × 4 label ratios × 1000 replicates each) show median RG/EIF width ratio of ~5.9× (not ~10×) and median PPI++ reduction of ~31.5% (below the claimed 35–55%), with only 46/144 cells falling in the claimed reduction band. logbook
  • On real Chatbot Arena preference data across three model pairs (414-494 samples each) using 10% calibration splits, PPI++, EIF, and MLE achieve near-nominal 90% coverage with mean interval widths of 0.27-0.30, while Rogan-Gladen achieves 0.76-0.96 width with 1 falsified x2
    [falsified] Released data contains 325/276/275 comparisons (not claimed 414–494); across 1,000 deterministic 10% calibration splits, joint MLE returns intervals on only 331–981/1000 splits with fail-closed coverage of 30.6–86.9%, contradicting the compound claim that MLE achieves near-nominal 90% coverage. logbook
  • The efficient influence function for the target parameter θ=E[Y] is φ*(R,Y,Ŷ;P) = μ(Ŷ) − θ + [R/P(R=1)]{Y − μ(Ŷ)}, where μ(ŷ)=E[Y|Ŷ=ŷ] is the optimal translation of the surrogate label (Proposition 5). verified x2
    [verified] Explicit derivation confirms E[φ(O)]=0 via iterated expectation and MCAR (E[R/π|Y,Ŷ]=1), with reduction to Y−θ when π=1, matching the stated formula under the paper's MCAR labeling design. logbook
What PaperCI measured
  • judge sensitivity: paper 0.82, PaperCI measured 0.052 (n=3, within tolerance: False)
  • judge sensitivity: paper 0.69, PaperCI measured 0.04 (n=3, within tolerance: False)
  • judge specificity: paper 0.64, PaperCI measured 0.052 (n=3, within tolerance: False)
  • true win rate: paper 0.523, PaperCI measured 0.025 (n=3, within tolerance: False)

Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing arXiv 2602.02157 · Deep Learning

Hugging Face community: did not reproduce (100% of 3 independent logbooks; 1 2 3) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status max_turns: no submit after 18 turns · build max_turns, 18 agent turns
Claims and evidence from the community reproductions
  • On CharacterTrajectories, MVC-CDE with GP smoothing requires 95.17 ± 7.41 function evaluations (NFE), versus 633.11 ± 28.31 for linear-spline and 595.91 ± 9.99 for cubic-spline neural CDEs (Table 1, Section C.1). falsified x1, verified x2
    [falsified] Five-seed runs on the authors' own code measure MVC-GP NFE at 37.8±2.7, cubic at 224.2±4.8, and linear at 309.4±20.8, all roughly 40–50% of the claimed 95.17, 595.91, and 633.11 respectively—discrepancies far exceeding the reported standard deviations, though the ordering (GP < cubic < linear) is pr logbook
  • On SpokenArabicDigits, MVC-CDE with GP smoothing requires 95.40 ± 0.89 NFE versus 2237.40 ± 20.02 for linear spline and 1003.80 ± 24.19 for cubic spline, corresponding to total training times of 7.32 ± 0.15s versus 106.15 ± 3.76s and 57.90 ± 1.73s respectively falsified x1, verified x2
    [falsified] On SpokenArabicDigits, measured clean-test NFE is 46.2±2.68 for MVC-GP, 341.4±6.84 for cubic, and 1113.0±70.61 for linear, versus claimed 95.40, 1003.80, and 2237.40; the NFE values (hardware-independent) are substantially lower than claimed across all arms, contradicting the specific numerical clai logbook
  • On UWaveGestureLibrary, MVC-CDE with GP achieves 218.60 ± 10.04 NFE and 12.29 ± 0.27s training time, versus 4340.60 ± 216.07 NFE / 176.19 ± 5.13s for linear spline and 1076.60 ± 33.57 NFE / 53.22 ± 2.25s for cubic spline (Table 1, Section C.1). falsified x3
    [falsified] Five-seed UWaveGestureLibrary runs measure MVC-GP NFE at 168.6±3.3 (vs 218.60), cubic at 1011.0±90.3 (vs 1076.60, close), and linear at 2001.0±133.3 (vs 4340.60, far off); the linear-spline NFE and MVC-GP NFE deviate well beyond their standard deviations from the claimed values. logbook
  • MVC-GP demonstrates a speedup ranging from 4.3× to 14.5× compared to standard spline-based neural CDE methods across the evaluated datasets (Section 4.2). falsified x3
    [falsified] Measured training-time speedups span 4.15× to 17.01× across three datasets, versus the claimed 4.3×–14.5×; while 4 of 6 individual comparisons fall within the claimed band, the measured range is wider at both ends, with the upper bound (17.01×) substantially exceeding 14.5×. logbook
  • Theorem 3.1 shows that NFE is strictly determined by the L_{1/(p+1)}-quasi-norm of the (p+1)-th derivative of the control path, and Corollary 3.3 shows NFE scales as h^{-1} for smoothing-based paths with lengthscale h (Section 3.2, Section 3.4). verified x3
    [verified] A standard LTE derivation independently recovers the L_{1/(p+1)} quasi-norm functional for NFE, and a 504-cell closed-form audit using X_h(t)=a·sin(t/h) with 60-digit arithmetic confirms h^{-1} scaling (max ratio error <10^{-59}) and δ^{-1/(p+1)} tolerance dependence across 7 solver orders, 3 tolera logbook
  • Figure 4 shows that under additive noise, smoothing-based methods (kernel/GP) maintain almost constant NFE, while interpolation-based (linear/cubic spline) methods show a drastic increase in both NFE and error rate (Section 4.3). verified x3
    [verified] A 120-cell noise sweep (3 methods × 5 seeds × 8 noise levels) on the authors' code shows MVC-GP NFE changes by 0.0 with only 2.40 accuracy points lost at noise 0.7, while cubic adds 183.6 NFE and loses 50.38 points and linear adds 2116.8 NFE and loses 39.93 points, directly supporting the claim that logbook
What PaperCI measured
  • no metric was measured

Frictional Q-Learning arXiv 2509.19771 · Reinforcement Learning

Hugging Face community: did not reproduce (100% of 2 independent logbooks; 1 2) · PaperCI: toy-scale · reduced scale only (0/2 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (Reduced from the paper's 5,000,000 timesteps per env to 3,000 timesteps on a single env (HalfCheetah-v4), learning_starts reduced 25,000 -> 500, single seed (SE); 2 metric(s) measured · build built, 16 agent turns
Claims and evidence from the community reproductions
  • Frictional Q-Learning uses a contrastive variational autoencoder (cVAE) to generate candidate actions aligned with replay-buffer-supported actions while separating salient features from irrelevant variation (Section on Architecture Components). verified x1, toy x1
    [verified] The logbook trains a neural cVAE with PyTorch on real Gymnasium MuJoCo replay trajectories (HalfCheetah, Hopper, Ant, Walker2d) across 12 cells with 3 seeds each, achieving target reconstruction MSE ≤0.0027 and selected-candidate cosine ≥0.9975 with replay actions, demonstrating the candidate-genera logbook
  • Theorems 1-4 establish convergence guarantees for BCQ-style constraints under coherent replay buffers in deterministic MDPs, while Theorem 5 addresses error convergence when orthogonal actions diverge from true dynamics (Theorem 1-4, Theorem 5). falsified x2
    [falsified] An exact two-state counterexample (code/exact_theorem5_zero_reward_counterexample.py) with transition mismatch p_0=(0,1) vs p_theta=(1,0) and zero reward yields Bellman fixed point Q=(0,0) under both kernels, giving extrapolation error exactly (0,0) despite the mismatch—contradicting Theorem 5's cla logbook
  • Lemma 2 shows that orthonormal vectors of rank k>=2 always exist in continuous action spaces, underpinning the method's use of orthogonal 'anti-support' directions (Lemma 2). verified x2
    [verified] Exact rational Householder constructions (code/orthogonal_scope_exact.py and code/exact_fql_theorem_certificate.py) verify orthonormal complement vectors across dimensions 2–4096 and ranks 2–64, with 220+960+3840+11520 rows all showing exactly zero pairwise and action-complement dot products in Frac logbook
  • On Humanoid-v4, FQL achieves a return of 6,713.75 +/- 103.89, outperforming SAC (6,463.46) and TD3 (6,524.20) with narrower standard deviation (Table 3). verified x2
    [verified] Independent reaggregation from released TensorBoard event streams using the paper's 'Seed' metric (mean of per-seed max Test/return) produces FQL 6713.753±103.893, SAC 6463.458±322.067, TD3 6524.199±401.628, matching Table 3 exactly; a destructive final-checkpoint control fails to reproduce, confirm logbook
  • An ablation over the friction coefficient beta shows task-dependent optimal values, with HalfCheetah-v4 exhibiting instability at suboptimal beta settings (Figure 3). verified x2
    [verified] Reaggregation of released raw runs across β∈{0.0,1.0,2.0,5.0} and 5 MuJoCo tasks (5 seeds each) shows different optimal β per task (HalfCheetah β=2.0, Hopper β=5.0, Ant β=0.0, etc.), and HalfCheetah suboptimal settings exhibit larger mean peak-to-later drawdowns (8190.88 at β=0.0 vs 3125.99 at β=2.0 logbook
  • A comparison of incorporating orthogonal actions via the critic versus via data augmentation shows the critic-based approach outperforms augmentation across most MuJoCo environments (Figure 4). verified x2
    [verified] Matched-beta comparison from released raw runs across 5 environments (5 seeds per route) shows critic-based orthogonal action incorporation has higher final mean return in 4/5 environments (Hopper, Walker2d, Ant, Humanoid), with augmentation higher only in HalfCheetah-v4, consistent with 'across mos logbook
What PaperCI measured
  • average return (final evaluation): paper 15969.59, PaperCI measured -603.929 (n=1, within tolerance: False)
  • average return (final evaluation): paper None, PaperCI measured None (n=None, within tolerance: None)
  • normalized return (offline RL): paper 50.66, PaperCI measured -603.929 (n=1, within tolerance: False)
  • normalized return (offline RL): paper None, PaperCI measured None (n=None, within tolerance: None)
  • average return (final evaluation): paper None, PaperCI measured None (n=None, within tolerance: None)

Identifying and Correcting Label Noise for Robust GNNs via Influence Contradiction arXiv 2601.17469 · Deep Learning

Hugging Face community: did not reproduce (100% of 2 independent logbooks; 1 2) · PaperCI: toy-scale · reduced scale only (2/2 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (); 2 metric(s) measured · build built, 18 agent turns
Claims and evidence from the community reproductions
  • ICGNN computes an Influence Contradiction Score (ICS) combining structure-level influence from personalized PageRank graph diffusion and attribute-level influence from a KNN affinity graph over node representations, then fits a Gaussian Mixture Model via EM on falsified x2
    [falsified] Across 40 detection runs on Cora and Citeseer, the released ICS is rank-equivalent to its attribute half alone (Spearman ≥ 0.9989, AUC gap never exceeds 0.0085), because the diffusion matrix is never column-normalized (Cora column sums 0.625–4.584) creating a 64–218× scale gap. Even re-implemented p logbook
  • Under 20% uniform label noise, ICGNN achieves 87.4% accuracy on Coauthor CS versus 86.2% for DND-NET, and 80.9% on Cora versus 76.8% for CGNN, with Wilcoxon rank-sum tests confirming statistical significance at the 0.1 level in most comparisons (Table 1). inconclusive x2
    [inconclusive] ICGNN's own accuracy reproduces well across all six datasets (Coauthor CS 86.79 vs 87.4, Cora 80.09 vs 80.9, all within −1.70 to +0.42 points), but the comparative margins against DND-NET and CGNN were never tested—DND-NET has no public code and CGNN (available in NoisyGL) was not run—so the signifi logbook
  • Under 20% pair (asymmetric) label noise, ICGNN reaches 85.9% accuracy on Coauthor CS and 80.6% on Pubmed, outperforming baselines NRGNN, RTGNN, CGNN, and DND-NET (Table 1). inconclusive x2
    [inconclusive] Accuracy reproduces across all six datasets (CS 84.77 vs 85.9, Pubmed 79.52 vs 80.6, all within −1.13 to −0.13), but the head-to-head comparison against NRGNN on identical splits shows ICGNN clearly ahead in only 2 of 8 cells (both Citeseer, by more than claimed), indistinguishable in 4, and behind logbook
  • Ablation results show removing the noise-cleaning (soft label correction) component causes the largest accuracy drop (~2.2%), removing structure-level ICS causes a ~1.5% drop, and removing attribute-level ICS causes a ~1% drop, while replacing the graph diffus inconclusive x1, falsified x1
    [inconclusive] On Cora under the paper's specification, s-ICS removal costs −1.68 (uniform, matching ~1.5%) and a-ICS removal costs −1.09/−0.98 (matching the paraphrased ~1%), but the headline assertion fails: noise-cleaning removal costs only −0.55/−1.08 (not ~2.2%, never the largest drop), and adjacency substitu logbook
  • ICGNN maintains its overall time complexity of O(EN + L^3 + CN), scaling linearly with sample size, and remains robust as noise rates increase up to 40% and as label rates shrink to 0.5%-2% (Complexity and Robustness Testing sections). verified x2
    [verified] Noise robustness is reproduced: DBLP and Pubmed degrade gracefully from 10% to 40% noise (e.g., DBLP/uniform 81.51→76.84, pair noise consistently worse), matching Figure 2's shape. Label-rate robustness is reproduced across all 16 cells (DBLP and Pubmed, 0.5%–2%, both noise models), with worst-case logbook
What PaperCI measured
  • test accuracy (node classification): paper None, PaperCI measured None (n=None, within tolerance: None)
  • test accuracy (node classification): paper 80.9, PaperCI measured 78.453333 (n=3, within tolerance: True)
  • test accuracy (node classification): paper 80.3, PaperCI measured 78.454067 (n=3, within tolerance: True)
  • test accuracy (node classification): paper None, PaperCI measured None (n=None, within tolerance: None)
  • test accuracy (node classification): paper None, PaperCI measured None (n=None, within tolerance: None)

Prototype-Grounded Concept Models for Verifiable Concept Alignment arXiv 2604.16076 · Social Aspects

Hugging Face community: did not reproduce (100% of 3 independent logbooks; 1 2 3) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status max_turns: no submit after 18 turns · build max_turns, 18 agent turns
Claims and evidence from the community reproductions
  • Prototype-Grounded Concept Models (PGCMs) ground concept predictions in a fixed number of learned visual prototypes (m=30 for ColorMNIST+, m=120 for CelebA, m=100 for CLEVR-Hans) via a prototype selector, enabling inspection through concept alignment tables (A verified x3
    [verified] The logbook executes the PGCM selector on a real full-test ColorMNIST batch and confirms shape [512, 2, 30] with 30 learned prototypes and probability-sum error 1.192e-07 (rows sum to one). Released configuration files independently pin m=30/120/100 for ColorMNIST+/CelebA/CLEVR-Hans, and a destructi logbook
  • On ColorMNIST+, PGCM matches CBM performance with concept accuracy 99.2±0.0 vs 99.2±0.1 and task accuracy 99.7±0.0 vs 99.6±0.1, while on CelebA CBM's task accuracy (84.0±0.3) modestly exceeds PGCM's (83.0±0.0) (Table 4). verified x3
    [verified] Fresh full-test ColorMNIST+ runs give PGCM task/concept accuracies of 0.9938/0.9881 and CBM 0.9960/0.9913, approximately matching the paper's claimed values (99.7/99.2 vs 99.6/99.2) in direction and magnitude, though not exactly. CelebA cells are source-pinned rather than freshly executed, which wea logbook
  • PGCM training uses an ELBO-style objective (Equation 2) combining a KL regularization term over prototype selection, task loss, concept loss, and image reconstruction loss (Equation 2). verified x3
    [verified] Direct training_step execution logs finite nonzero values for all four claimed objective components: task=0.013922, concept=0.046831, reconstruction=0.031371, and KL-to-uniform=3.367012. Destructive controls confirm KL endpoints 0 and log(30) for uniform and one-hot selectors, and removing any term logbook
  • On ColorMNIST+ with noisy labels, targeted prototype-level interventions improve accuracy from 92.9% to 96.9% by removing prototypes and from 92.8% to 97.8% by editing prototypes (Table 3). falsified x3
    [falsified] After training with the specified label corruption and independently identifying 6 corrupted prototypes, full-test concept accuracy changes from 0.9761 to 0.9759 (removal) and 0.9756 (editing), with affected-concept changes of +0.0000/-0.0007. This contradicts the paper's claimed large improvements logbook
  • A prototype-swapping procedure applied halfway through training replaces learned prototype embeddings with their nearest training instances to improve interpretability (Interpretability Optimizations section). verified x3
    [verified] All 30/30 post-swap checkpoint prototypes are verified to bind to specific training instances from the complete 54,000-object training split via byte-exact masked-object hashing or exact source-image-support provenance. A destructive control adding one non-background pixel to a prototype is correctl logbook
  • PGCM shows improved responsiveness to concept interventions compared to CBM, CRM, and CMR baselines, particularly when exploiting inter-concept dependencies in the PGCM* variant on ColorMNIST+ (Figure 4). falsified x3
    [falsified] Across three fixed intervention orders on all 5,000 test examples, PGCM* concept AUC is 0.995330, which is lower than CBM's 0.995643, contradicting the claim that PGCM shows improved responsiveness over all baselines including CBM. PGCM* does exceed CRM (0.994621) and CMR (0.994775), but not CBM. logbook
What PaperCI measured
  • no metric was measured

Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens arXiv 2605.30960 · Optimization

Hugging Face community: did not reproduce (82% of 11 independent logbooks; 1 2 3 4 5) · PaperCI: toy-scale · reduced scale only (0/2 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (); 2 metric(s) measured · build built, 18 agent turns
Claims and evidence from the community reproductions
  • ZoVH is introduced as a variance-reduced zeroth-order Hessian estimator that unifies One-Point, Two-Point, and Three-Point Stein-type estimators as special cases of a Gaussian sampling policy with different baseline choices (Proposition 3.2, Section 3). verified x11
    [verified] On a d=12 quadratic with K=400 directions, the general Gaussian-policy estimator with baselines b=0, b=f(θ), and antithetic+b=f(θ) recovers the One-Point, Two-Point, and Three-Point Stein estimators to Frobenius distances of 0.0, 0.0, and 8.1e-13 respectively (on a Hessian norm scale of ~42), direct logbook
  • Theorem 4.6 proves the ZoVH estimator is unbiased for the Hessian of the smoothed objective, E[Ĥ(θ)] = ∇²Fμ(θ), establishing the theoretical equivalence between the ZO Hessian estimator and the smoothed single-step Policy Optimization objective's Hessian (Theo verified x10, toy x1
    [verified] On a d=10 quadratic with R=4000 Monte Carlo draws, the ZoVH averaged-baseline residual decays with log-log slope −0.518 (matching the 1/√R unbiasedness signature), while the naive zero-baseline estimator shows a persistent bias of 752,847 matching the closed-form F(θ)/μ²·√d = 753,072 to 0.03%. logbook
  • Theorem 4.7 shows the averaged-baseline estimator is variance-optimal, reducing in high dimensions to b* = E_u[F(θ+μu)], and Theorem 4.8 bounds the estimator's expected Frobenius-norm error as a sum of a K,d,μ-dependent variance term and an L2²μ²d bias term (T verified x9, toy x1, inconclusive x1
    [verified] On a d=8 quadratic, the empirical arg-min of the variance sweep over baseline b lands exactly on F_μ(θ)=85.1399 (matching the analytic value), and the MSE-vs-K log-log slope is −1.036 (theory: −1) while the smoothing-bias-vs-μ slope is +2.000 (theory: +2), confirming both theorems' key predictions. logbook
  • On synthetic Quadratic and Rosenbrock functions, ZoVH with query-reuse parameter N=6 achieves an 8× accuracy improvement over central-difference Hessian estimators, and a 3.4× improvement on the Styblinski-Tang function (Figure 2, Section 6.1). toy x3, verified x3, falsified x5
    [toy] Experiments run at d=50 (vs paper's d=5000, a 100× reduction); while ZoVH beats central-difference on Quadratic (7.68×) and Rosenbrock (8.29×) close to the paper's ~8×, the Styblinski-Tang ratio is 7.95× versus the paper's 3.4× — a major discrepancy inconsistent with the logbook's own √d scaling arg logbook
  • On a black-box MNIST adversarial attack task, ZoVH with N=4 achieves a 4.04× speedup over the baseline, requiring 403±76 iterations versus 1629±610 iterations (Table 1, Section 6.2). toy x1, falsified x9, verified x1
    [toy] The MNIST CNN adversarial attack is replaced with a d=15 ill-conditioned quadratic surrogate; the ordering Vanilla > ZoVH(N=1) > ZoVH(N=4) is reproduced but the speedup is 1.72× versus the claimed 4.04×, and the specific iteration counts (403±76 vs 1629±610) are not reproduced at all. logbook
  • Across four synthetic convergence benchmarks (Quadratic, Levy, Rosenbrock, Ackley), ZoVH with N=4 achieves a 22× speedup versus vanilla zeroth-order optimization in reaching target accuracy (Section 6.2). inconclusive x1, toy x2, verified x3, falsified x5
    [inconclusive] At d=8 (vs paper's d=10000), ZoVH shows no speedup (mean 0.74×, with Ackley at 0.25× and Levy/Rosenbrock hitting iteration caps); the 1250× dimension reduction is too large to either confirm or refute the 22× claim at the paper's scale, and the logbook's mechanistic explanation for the gap is specul logbook
What PaperCI measured
  • Hessian estimation accuracy improvement over central-difference estimator: paper 81.52, PaperCI measured -63.306667 (n=3, within tolerance: False)
  • Hessian estimation accuracy improvement over central-difference estimator: paper 63.71, PaperCI measured -109.86 (n=3, within tolerance: False)
  • Average speedup over Vanilla ZOO: paper None, PaperCI measured None (n=None, within tolerance: None)

Testing For Distribution Shifts with Conditional Conformal Test Martingales arXiv 2602.13848 · Probabilistic Methods

Hugging Face community: did not reproduce (100% of 4 independent logbooks; 1 2 3 4) · PaperCI: toy-scale · reduced scale only (0/1 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (Used the paper's default synthetic change-point configuration (calibration D0 = 2000 N(0,1) samples, 1000 streamed test samples, location shift +1.0, alpha=0.05); 1 metric(s) measured · build built, 12 agent turns
Claims and evidence from the community reproductions
  • The conditional conformal test martingale (CTM) uses a fixed reference dataset D0 rather than an ever-growing reference set, avoiding test-time contamination that dilutes evidence of distribution shift in standard CTMs (Section 3). verified x4
    [verified] Code inspection of the released implementation confirms a fixed empirical CDF built from cal_samples at initialization with no test-time insertion, and 100-trial experiments show conditional CTM detecting an immediate N(0,1)→N(1,1) shift at median step 21 vs 29 for the growing-reference baseline. logbook
  • Theorem 3.1 establishes finite-sample, anytime-valid type-I error control, bounding the probability of falsely rejecting the null by alpha with probability at least 1-delta over the randomness of the reference set (Theorem 3.1). verified x1, falsified x3
    [verified] The theorem statement was audited for both alpha and delta randomness layers, and 200 null runs confirmed the prediction: DKW-corrected method rejected 0/200 times at alpha=0.05, while disabling the correction yielded 15/200 (7.5%) rejections. logbook
  • Theorem 3.3 shows the method achieves asymptotic power one whenever the effective signal strength Delta_k is positive, with expected stopping time bounded by O(1/Delta_0^2 * ln(1/(alpha*Delta_0)) + 1/Delta_0^2) (Theorem 3.3). verified x3, inconclusive x1
    [verified] A 3,500-trial campaign across 7 shift magnitudes (500 reference sets each, n=1000, horizon 10,000) found 100% detection whenever Δ₀>0, with stopping-time scaling exponent −1.831 (close to the theorem's −2) and rate-proxy regression slope 0.847. logbook
  • The betting function (Eq. 8) subtracts a Dvoretzky-Kiefer-Wolfowitz-based estimation-error correction term from the standard betting score to remove spurious wealth gains from estimation uncertainty (Section 3, Equation 8). verified x4
    [verified] Code inspection confirms the released betting_function subtracts sign(eta)×interval_eps before scaling; ablation over 200 null runs shows 0% rejection with correction vs 7.5% without, demonstrating the DKW term is operationally active. logbook
  • On synthetic experiments with immediate, delayed, and gradual distribution shifts, the conditional CTM detects shifts faster than the standard (invalid) CTM while maintaining type-I error control at alpha=0.05 (Figure 2, Figure 3). falsified x3, verified x1
    [falsified] 100-trial synthetic experiments show conditional CTM is faster only for immediate shifts (median 21 vs 29); for delayed shifts it is slower (362 vs 359) and for gradual drift it is also slower (237 vs 199), contradicting the blanket speed-advantage claim across all three shift types. logbook
  • On ImageNet-C corruption benchmarks, detection performance improves with reference set size n >= 500, with detection ratio exceeding 1 for corruption groups involving larger distributional shifts (Figure 4). toy x2, inconclusive x2
    [toy] Real ImageNet-C severity-5 data with ViT-Base and matching reference sizes n∈{100,500,4000} shows conditional power rising 0.800→0.847→0.940 and all family ratios >1, but the corrupted stream was shortened from 37,500 to 500 images (75× reduction), limiting scope. logbook
What PaperCI measured
  • mean conformal p-value under the null (in-distribution level): paper 0.5, PaperCI measured 0.671933 (n=3, within tolerance: False)

Time series saliency maps: Explaining models across multiple domains arXiv 2505.13100 · Applications

Hugging Face community: reproduced (75% of 4 independent logbooks; 1 2 3 4) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status max_turns: no submit after 18 turns · build max_turns, 18 agent turns
Claims and evidence from the community reproductions
  • Cross-domain Integrated Gradients generalizes Integrated Gradients to any invertible, differentiable transformation, including complex-valued domains such as the Fourier transform, while provably preserving path independence and completeness guarantees (Defini verified x3, toy x1
    [verified] Numerical checks on Fourier (completeness residual 4.17e-07, path gap 5.68e-10), ICA-style (residual 2.38e-07), and STL-style (residual 2.22e-15, path gap 1.78e-15) domains confirm path independence and completeness; a rank-deficient negative control shows completeness residual 3.0 when invertibilit logbook
  • The method is applied to frequency-domain attribution for heart rate extraction on the PPGDalia photoplethysmography dataset (Section 5, Figures 1-3). verified x3, toy x1
    [verified] Full 15-subject PPG-DaLiA run with 64,682 windows and 300 IG steps shows frequency IG outperforms time IG in the paper's direction in 6/6 deletion/insertion comparisons with positive bootstrap CIs in 5/6, though checkpoint provenance is mixed (only 2 paper-released weights). logbook
  • The method is applied to ICA-domain attribution for seizure detection on the Physionet Siena Scalp EEG Database v1.0.0 (Section 5). verified x3, toy x1
    [verified] All 41 Siena EEG records processed at 256 Hz with 19-channel FastICA and 300 IG steps; ICA deletion/insertion 0.175470/0.088149 vs random 0.006008/0.461945 closely match paper Table 5 values (0.1776/0.0696 vs 0.0083/0.4396), largest difference 0.022. logbook
  • The method is applied to seasonal-trend (STL) decomposition attribution for forecasting on synthetic time series combining exponential trend and seasonal components (Section 5, Figures 4-5). verified x3, toy x1
    [verified] TimesFM run on 11 synthetic series at 2 horizons with 300 IG steps shows trend as dominant absolute component in 22/22 comparisons (mean trend IG 4.97 at horizon 0, 5.61 at horizon 97), with batch-equivalence control max diff 0.0. logbook
  • Proposition 1 establishes completeness, guaranteeing that summed cross-domain attributions equal the model output difference f(x) - f(x_hat) (Proposition 1). verified x3, toy x1
    [verified] Completeness residuals are negligible across Fourier (4.17e-07), ICA (2.38e-07), and STL (2.22e-15) domains, confirming summed attributions equal f(x)-f(x_hat); the non-invertible control produces residual 3.0, confirming the theorem's conditions matter. logbook
  • The framework is implemented as an open-source Python library compatible with both TensorFlow and PyTorch, with core algorithms detailed as Algorithms 1-3 in Appendix A (Appendix A). verified x3, toy x1
    [verified] All 8 documented modules import successfully for both torch_ig and tensorflow_ig; PyTorch tests pass 26/26 and TensorFlow tests pass 19/19; all 4 example notebooks exist and CPU smoke tests produce finite attributions in both frameworks. logbook
What PaperCI measured
  • no metric was measured

Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization arXiv 2605.14373 · Optimization

Hugging Face community: did not reproduce (100% of 3 independent logbooks; 1 2 3) · PaperCI: toy-scale · reduced scale only (0/2 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (Repository ships only the CoCD optimizer (cocd.py, zeroth_order_optim.py); no model, training loop, or benchmark code. /work/paperci_run.sh embeds a driver that); 2 metric(s) measured · build built, 17 agent turns
Claims and evidence from the community reproductions
  • CoCD (Coherent Coordinate Descent) is a deterministic zeroth-order optimizer that achieves O(1) query complexity per step while provably maintaining a global descent direction (Abstract, Theorem 5.3). verified x2, toy x1
    [verified] Objective evaluations per step are exactly 2B with zero deviation across 15 (B,n) settings spanning n=102 to 1,000,002; determinism is confirmed by bit-identical trajectories across two RNG seeds (max|Δx|=0.0) vs 3.71e-3 for a randomized control; update support reaches 99.97% of 13,145 coordinates a logbook
  • Theorem 5.10 proves that under Polyak-Łojasiewicz conditions, CoCD achieves linear convergence at rate (1-2μC1/C2)^t despite using stale gradient estimates, with staleness factor τ=n/B-1. verified x2, inconclusive x1
    [verified] On a 64-dimensional quadratic with exact μ=1.0, L=20.0 satisfying PŁ, the theorem's bound f_t−f* ≤ ρ^t(f_0−f*) held at all 36,213 recorded iterates across four staleness settings; measured max buffer age exactly matched τ=n/B−1={0,1,3,7}; convergence was geometric (R²=0.999988, 15.8 decades of decay logbook
  • A momentum parameter γ controls memory of past stale gradients, ranging from γ=1.0 (perfect memory) to γ=0.0, which recovers standard BCCD with no historical gradient reuse (Section 5). verified x3
    [verified] γ=0 produces bit-identical trajectories to an independently written BCCD over 200 steps on the 13,145-parameter MLP (max|Δx|=0.0); γ=1 retains stale entries with ratio exactly 1.0 for 60 steps and reconstructs the full gradient to 7.64e-04 relative L2; intermediate γ values decay as exactly γ^k (max logbook
  • On Table 2, CoCD substantially outperforms BCCD in sample efficiency: SARCOS regression loss of 31.18 vs. 188.73, MNIST accuracy of 95.48% vs. 27.03%, and CIFAR-10 accuracy of 45.08% vs. 10.13%. falsified x3
    [falsified] Under the authors' released optimizer, the paper's own Table-1 SARCOS hyperparameters (lr=0.001, γ=1.0, ε=1.0, B=64, 50 epochs), and the full 44,484-row dataset, CoCD diverges to 4.77e+22 while BCCD converges to 57.91—reversing the claimed ordering (31.18 vs 188.73) by over 20 orders of magnitude; t logbook
  • CoCD achieves roughly 2x wall-clock speedup over ZO-SGD on regression tasks (8.1s vs. 15.7s per episode), and Appendix E scales the method to a ResNet-20 (~270k parameters) on CIFAR-10 with a query budget B=2048 (Figure 4, Appendix E). verified x1, toy x1, inconclusive x1
    [verified] Matched-budget timing on full-scale SARCOS (verified 128 objective evaluations per step for all methods) gives CoCD 9.63 s/epoch vs ZO-SGD 18.00 s/epoch = 1.868× speedup, within 3.6% of the claimed 1.938×, with 26.9% of ZO-SGD's time in Gaussian sampling vs 0.0% for CoCD; a from-scratch ResNet-20 wi logbook
What PaperCI measured
  • final test accuracy: paper 45.08, PaperCI measured 97.366667 (n=3, within tolerance: False)
  • wall-clock runtime per episode (speedup vs ZO-SGD): paper 8.1, PaperCI measured 24.356667 (n=3, within tolerance: False)
  • speedup factor over ZO-SGD (wall clock): paper None, PaperCI measured None (n=None, within tolerance: None)

Two-Layer Linear Auto-Regressive Models Estimate Latent States arXiv 2606.12691 · General Machine Learning

Hugging Face community: did not reproduce (91% of 11 independent logbooks; 1 2 3 4 5) · PaperCI: toy-scale · reduced scale only (3/4 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (Reduced vs paper: T=2000 (paper 10000) and epochs=30 (paper 150); AR window L=8 and horizon H=1; batch_size=64. Hidden width set to state dim n=4. System n=4,p=); 4 metric(s) measured · build built, 13 agent turns
Claims and evidence from the community reproductions
  • The informal main result states that the hidden representation learned by a two-layer linear autoregressive model trained on a partially observed linear dynamical system coincides, up to a similarity transformation S, with the state estimates of the optimal Ka verified x11
    [verified] At native scale (n=4, p=2, m=3, T_train=10^4, 5 seeds), H=5 mean R²=0.9951 vs paper's 0.998, with time-shuffled control collapsing to R²≈−0.05 confirming genuine per-timestep correspondence. The untrained random-init control reaching R²=0.962 is a notable caveat about the metric's diagnostic power, logbook
  • Theorem 4 (Section 4.4) formalizes this latent-state-recovery result, giving explicit finite-sample error bounds between the learned representation and the Kalman filter estimate up to the similarity transformation. verified x9, inconclusive x1, falsified x1
    [verified] The similarity transform S exists exactly: ‖SĜ₁−𝒞‖=4.16e-15 over 60/60 systems, and finite-sample error decays with T-exponent −0.822 (fit R²=0.998) vs predicted −1, with the shortfall explained by a fixed T_test floor for fitting S. The h=3<n=4 destructive control yields 4.61× larger recovery error logbook
  • Theorem 2 (Section 4.3) bounds the in-sample prediction error at global optima of the empirical risk at a rate of O(H/T), where H is history length and T is trajectory length. verified x9, toy x1, falsified x1
    [verified] Across 6,300 cells at native scale, the pooled T-exponent is −1.019 (fit R²≥0.997) in the stable regime ρ(A)=0.9, with H-exponent 1.756 falling within the bound's predicted [1,2] range. The L-hypothesis is shown load-bearing: at ρ(A)=1 with fixed L=10 the exponent degrades to −0.631, but growing L a logbook
  • Theorem 3 (Section 4.3) gives a finite-sample bound on parameter estimation error in Frobenius norm for the learned autoregressive weights. verified x10, toy x1
    [verified] The sandwich inequality underlying Theorem 3 holds with 0/350 violations on both upper and lower bounds. The Frobenius-norm parameter error decays at T-exponent −1.042 (H=1) and −1.040 (H=5), both close to the predicted −1. Persistence of excitation is confirmed (λ_min flat at ~0.035 across 64× T ra logbook
  • Proposition 1 (Section 4.1) shows the Kalman filter is well-approximated by a finite-history autoregressive model, with truncation error decaying exponentially in history length L. verified x11
    [verified] The AR-KF identity x̃_t = 𝒞z̄_t + Ā^L x̃_{t−L} is exact to 6.23e-13 over 1,200 cells (40 systems × L=1..30). Measured exponential decay base matches ρ(Ā) at median ratio 0.9987 (range [0.903, 1.004], all 40/40 systems ≤1.02·ρ(Ā)). The stated high-probability bound has 0/3,562,800 violations, and the logbook
  • In numerical experiments, a grid search over hidden dimension h identifies h=4 as optimal, matching the true latent state dimension n=4 (Section 5.2, Table 1), and learned activations achieve mean R-squared >= 0.979 alignment with Kalman filter state estimates falsified x9, verified x2
    [falsified] The grid search argmin is h=10/8/10 across three Adam-trained system replicates (not h=4), and at the exact global optimum the training loss is provably monotone non-increasing in h (0/300 violations, argmin h=10 in 30/30 systems) because rank-≤h classes are nested. h=4 wins only 1/30 trials (~chanc logbook
What PaperCI measured
  • training loss: paper 0.0473, PaperCI measured 0.285303 (n=3, within tolerance: True)
  • training loss: paper 309.598, PaperCI measured 1.273878 (n=3, within tolerance: False)
  • mean latent state recovery statistic (alignment between Kalman filter estimates and learned activations): paper 0.999, PaperCI measured 0.980372 (n=3, within tolerance: True)
  • mean latent state recovery statistic (alignment between Kalman filter estimates and learned activations): paper 0.996, PaperCI measured 0.980372 (n=3, within tolerance: True)

Unfolding Generative Flows with Koopman Operators: Trajectory-Preserving Linearization arXiv 2506.22304 · Deep Learning

Hugging Face community: did not reproduce (75% of 4 independent logbooks; 1 2 3 4) · PaperCI: blocked-infra · no decision · full PaperCI review
PaperCI's reason: builder status gave_up: needs_gpu · build gave_up, 18 agent turns
Claims and evidence from the community reproductions
  • The method formulates Conditional Flow Matching dynamics via a Koopman generator that lifts nonlinear vector-field dynamics into a linear evolution over observables, using the identity L g(x_t) = ∇_x g(x_t)·v(x_t) (Section 4, Eq. 8). verified x4
    [verified] Trained CFM teachers and Koopman students on all three 2D datasets (2 seeds each); learned generator satisfies Eq. 8 identity with max relative residual 0.216584 and exp(tL) reproduces RK4 teacher trajectories with MSE ≤ 0.0203415 across all runs. logbook
  • A decoder-free parametrization concatenates the raw state with learned observables, g̃(x) = [x, g_θ(x)], avoiding reconstruction error from a separate decoder (Section 4, Eq. 14). verified x1, falsified x3
    [verified] Like-for-like trained comparison on Two Moons shows decoder-free reconstruction MSE of 0.0 (by construction) vs encoder-decoder 0.00437493, and decoder-free also preserves trajectories better (MSE 0.0201308 vs 0.0706791). logbook
  • Training combines a Generator loss enforcing Koopman consistency with a Target Consistency loss aligning the exponentiated generator trajectory to the terminal state (Section 4.3, Eqs. 15-16). verified x1, falsified x3
    [verified] Combined Generator+Target Consistency objective was trained on Two Moons with per-term ablation: removing Generator loss raises trajectory MSE from 0.0201308 to 0.0856646, confirming each term enforces its claimed role. logbook
  • On MNIST, one-step (one-shot) generation with the Koopman linearization attains FID 60.6 versus FID 190 for a CFM baseline under the same one-step budget (Section 5, Table 4). toy x1, falsified x3
    [toy] Direction confirmed on real MNIST but at reduced scale: PCA-32 latent space, 10k training images, and RBF-MMD proxy instead of FID. Koopman one-step beats CFM Euler one-step by 3.19x–6.92x in MMD across 2 seeds, but the specific FID values 60.6 vs 190 were never computed. logbook
  • On 2D toy datasets (Swiss Roll, Eight Gaussians, Two Moons), the method achieves MMD scores within about 0.02 of the CFM baseline while enabling full-trajectory linearization (Section 5, Table 1, Figure 4). verified x2, toy x1, inconclusive x1
    [verified] Trained on all three actual 2D datasets with 2 seeds each; seed-averaged MMD gaps are 0.003, 0.013, and 0.024 — two within 0.02 and the worst (Two Moons) at 0.0236, consistent with the claim's 'about 0.02' phrasing, at the paper's own scale. logbook
  • An ablation shows the decoder-free architecture outperforms an encoder-decoder variant, and the combined Generator+Consistency loss reaches FID 20 on MNIST versus worse scores for either loss alone (Section 5, Table 3, Figures 9-10). toy x1, falsified x3
    [toy] Ablation was run on 2D Two Moons (not MNIST): decoder-free beats encoder-decoder on reconstruction and trajectory MSE but ties on sample MMD; the combined-loss-beats-either-alone ordering does not reproduce at 2D scale (gen-only matches combined), and the MNIST FID 20 vs 38 result was never tested. logbook
What PaperCI measured
  • no metric was measured

Universal Representation of Generalized Convex Functions and their Gradients arXiv 2509.04477 · Theory

Hugging Face community: did not reproduce (78% of 9 independent logbooks; 1 2 3 4 5) · PaperCI: toy-scale · reduced scale only (0/4 measured claims matched; no full-scale verdict) · full PaperCI review
PaperCI's reason: ran at reduced scale (Setting A auction design (single buyer, dot-product kernel, no production cost, uniform types on [0,1]^m, outside option). Reduced from the paper's m in {1,2,4,); 4 metric(s) measured · build built, 12 agent turns
Claims and evidence from the community reproductions
  • Theorem 1 (Section V.A) establishes that finitely Ỹ-convex functions form a dense subset of all Ỹ-convex functions, giving a universal approximation property for generalized convex functions. toy x3, verified x6
    [toy] The logbook tests approximation of a single 1D quadratic function f(x)=x²/2 on [-1,1] with up to 65 support points, showing decreasing sup-norm error. This illustrates the density mechanism but does not constitute verification of a universal approximation theorem over general Ỹ-convex functions. logbook
  • Theorem 2 (Section V.A) shows that, under semiconvexity conditions on Φ, gradients of finitely Ỹ-convex functions densely approximate gradients of all Ỹ-convex functions. toy x3, verified x5, inconclusive x1
    [toy] Gradient error is measured on the same single 1D quadratic function with the same finite-support grid, showing decreasing error from 0.5 to 0.0156. This is a simplified demonstration of gradient density, not a verification of the theorem's general semiconvexity conditions. logbook
  • Theorem 4 (Section V.C) proves that the lean subset of finitely Ỹ-convex functions forms a convex parameter space, which is what allows bilevel objectives to be rewritten as single-level problems solvable with standard first-order optimization (Section I). toy x2, verified x5, falsified x2
    [toy] The logbook numerically stress-tests 1000 convex combinations of 9-support max-affine parameter vectors in 1D, finding all remain lean. This is a bounded-scale numerical illustration of convexity, not a verification of the general theorem over arbitrary dimensions and support sizes. logbook
  • Table I reports multi-item auction experiments for n in {1,2,5,10,20} goods in which, for n up to 10, the learned mechanism's revenue matches the Straight-Jacket auction benchmark exactly. falsified x7, inconclusive x1, toy x1
    [falsified] Source audit of the accepted arXiv v3 table shows n-values are {1,2,4,6,12} not {1,2,5,10,20}, and the paper says 'closely match SJa's revenue' not 'exactly matches.' Only n=2 is exactly equal at reported precision; gaps range 0.001–0.008 at other entries. logbook
  • Figures 2-3 show the learned parametrization recovers the known optimal posted price of 0.5 in the single-item auction case and finds a mixed-bundling pricing scheme matching theoretical benchmarks in the two-item case. verified x8, toy x1
    [verified] Exhaustive 10001-point grid search recovers the single-item optimal posted price of 0.5 with revenue 0.25, and 500×500 quadrature with differential evolution finds a mixed-bundling menu (single price 0.673, bundle price 0.866, revenue 0.275/good) with all menu options active, matching the paper's Fi logbook
  • Kantorovich dual solutions for optimal transport are characterized as Ỹ-convex functions in Section IV, so gradients of the learned parametrization directly yield optimal transport maps via a diffeomorphism condition. toy x3, verified x6
    [toy] The logbook tests a single 1D optimal transport problem (U[0,1]→Beta(2,1) with quadratic surplus), recovering T(x)=√x to ~1e-6 accuracy. This illustrates the diffeomorphism formula but is a simplified 1D proxy for the general Kantorovich dual characterization. logbook
What PaperCI measured
  • mean profit per good (learned mechanism, Setting A, small m): paper 0.274, PaperCI measured 0.241717 (n=3, within tolerance: False)
  • mean profit per good (learned mechanism, Setting A, small m, largest reported): paper 0.361, PaperCI measured 0.241717 (n=3, within tolerance: False)
  • mean profit per good (learned mechanism, Setting A, large m): paper 0.412, PaperCI measured 0.241717 (n=3, within tolerance: False)
  • mean profit per good (learned mechanism, Setting A, large m, largest reported): paper 0.45, PaperCI measured 0.241717 (n=3, within tolerance: False)
  • mean profit per good (Setting B, nonlinear hinge surplus, largest reported): paper None, PaperCI measured None (n=None, within tolerance: None)