papercibot reviewed this paper's code · 2026-10-11 11:11
Pipeline
✓Triage2s
✓Claims sealed6s
✗Build1.1m
–Offline run0s
–Reruns0s
!Verdict0s
–Package
–Oracle
–Trainability
Walkthrough
PaperCI fetched the paper's repository into a fresh CPU sandbox. A builder agent (claude-sonnet-4.6) worked for 16 turns (1.1 min), installed 1 package(s) (2), but did not reach a runnable state (gave_up)
4claims sealed
0reproduced
0measured
0offline reruns
16agent turns
$0.19total cost
Claims vs measured
Claim (from the paper)
Paper
PaperCI measured
Tolerance
Status
c1 Overall · YuE2-3B with MIRA
82.1
–
-
Blocked
c2 Overall · ACE-Step with MIRA
76.3
–
-
Blocked
c3 Comp. · YuE2-3B with tool grounding
0.8
–
-
Blocked
c4 Comp. · ACE-Step with adaptive search
0.781
–
-
Blocked
Checks
✓SandboxFresh Docker container: non-root, all capabilities dropped, egress only via an allowlist proxy
✓Claims sealed4 headline claims extracted from the paper text and sealed (sha256 df0faedbd681) before the builder started; the builder never saw the values
✗Buildgave_up after 16 agent turns, 1.1 min
✓No hardcoded answersno sealed claim value appears in anything the builder wrote
✓No copied resultsno output file is a byte-identical copy of a result file in the repo
✓Computed, not replayedthe run computes its numbers instead of loading stored results
✓Uses the paper's codethe run script executes the repository's own code
Suggestions for the authors
1. Make the result reproducible from the public repoThe agent stopped: "(details hidden)"
Pipeline
Walkthrough
PaperCI fetched the paper's repository into a fresh CPU sandbox. A builder agent (claude-sonnet-4.6) worked for 16 turns (1.1 min), installed 1 package(s) (2), but did not reach a runnable state (gave_up)
Claims vs measured
Checks
Suggestions for the authors
Verdict blocked-infra (medium):