✗ Not runnable within the agent budget (18 turns)P-554af5 · repository hidden · run dev-g2
PCI
papercibot reviewed this paper's code · 2026-10-11 11:11
Pipeline
✓Triage2s
✓Claims sealed7s
✗Build1.3m
–Offline run0s
–Reruns0s
!Verdict0s
–Package
–Oracle
–Trainability
Walkthrough
PaperCI fetched the paper's repository into a fresh CPU sandbox. A builder agent (claude-sonnet-4.6) worked for 18 turns (1.3 min), installed 3 package(s) (coverforest, cython, scikit-learn), but did not reach a runnable state (max_turns)
5claims sealed
0reproduced
0measured
0offline reruns
18agent turns
$0.16total cost
Claims vs measured
Claim (from the paper)
Paper
PaperCI measured
Tolerance
Status
c1 coverage · Mice dataset, CV+ method with APS score
0.964
–
-
Blocked
c2 avg_set_size · Mice dataset, CV+ method with APS score
1.589
–
-
Blocked
c3 coverage · Mice dataset, CV+ method with RAPS score
0.968
–
-
Blocked
c4 avg_set_size · Mice dataset, CV+ method with RAPS score
1.761
–
-
Blocked
c5 speedup · Helena dataset, CV+ method
72
–
-
Blocked
Checks
✓SandboxFresh Docker container: non-root, all capabilities dropped, egress only via an allowlist proxy
✓Claims sealed5 headline claims extracted from the paper text and sealed (sha256 30c222ed57ff) before the builder started; the builder never saw the values
✗Buildmax_turns after 18 agent turns, 1.3 min
✓No hardcoded answersno sealed claim value appears in anything the builder wrote
✓No copied resultsno output file is a byte-identical copy of a result file in the repo
✓Computed, not replayedthe run computes its numbers instead of loading stored results
✓Uses the paper's codethe run script executes the repository's own code
Suggestions for the authors
1. Add a one-command reproduction entry pointGetting this repository to produce the headline numbers took more than 18 agent steps. A reproduce.sh (or a README section) that prints each table number would make it checkable.
2. Missing moduleImports failed for a module that is not declared as a dependency; add it to requirements.
3. Api incompatThe code breaks with current library versions; pin the versions you used (pip freeze).
Pipeline
Walkthrough
PaperCI fetched the paper's repository into a fresh CPU sandbox. A builder agent (claude-sonnet-4.6) worked for 18 turns (1.3 min), installed 3 package(s) (coverforest, cython, scikit-learn), but did not reach a runnable state (max_turns)
Claims vs measured
Checks
Suggestions for the authors
reproduce.sh(or a README section) that prints each table number would make it checkable.Verdict blocked-infra (medium):