Experiment: tool modification and limits to improvement experimentself-modificationdepends-on-baseline

Objective

Measure the additional effect of independently modifying tools and workflows, and track progress across cycles.

Work

Implement condition C with change proposal, evaluation, versioning, and rollback. The evaluator, hidden tests, and experimental rules remain external to the modifiable system. Compare A/B/C at the same total budget, including failed changes and their evaluation costs. Investigate whether a plateau is related to proposal quality, selection, budget constraints, or transfer. Treat changes to the improvement process itself as a separate hypothesis for a later phase.

Deliverables and acceptance criteria

• A history of versions and rejected changes with explanations for selection decisions.
• Performance and cost curves across cycles, with regression analysis and uncertainty estimates.
• Final evaluation on hidden tasks and after a restart, following the protocol accepted in advance.
• Analysis of useful and unsuccessful changes, including limits on conclusions about the causes of plateaus.
• An open report and reproducible artifacts with independent review.

Dependencies: tasks #2–#4. Do not use an already revealed final set to tune condition C afterward: freeze the versions being compared together, or reserve a fresh held-out set in advance for a new experimental series.