skill-conductor

Skill Conductor A skill that creates, evaluates, and improves other skills. Meta-level. [](https://github.com/smixs/skill-conductor/releases) [](LICENSE) [](https://skills.sh) [](https://code.claude.com/docs/en/plugins) Architecture-first skill lifecycle: **design → build → test → evaluate → package**. Most skill tools jump straight to "write SKILL.md." Conductor makes you choose the architecture first — because rewriting a wrong pattern costs more than writing it right. Install v3.2.0 — Evidence-based upgrade: form-matching, judge calibration, pressure testing - **Principle #10: Match the form to the failure** — classify the baseline failure before writing a rule; prohibitions bulletproof discipline failures but measurably backfire on shaping failures (obra/superpowers wording tests + Guardrails polarity data). Plus: no nuance clauses, exemption clauses don't scope. - **Critique-before-verdict judges** — all three eval agents (grader, comparator, bineval) now write the detailed evidence critique BEFORE committing to the 1/0 verdict, with a borderline few-shot example in each (Hamel Husain's judge methodology). - **Threshold-blind judging** — the BinEval judge no longer computes the overall score or the GATE; the orchestrator aggregates. A judge that knows the bar is biased toward it. - **Automatic cross-family judge calibration** — a second judge from a different model family answers the same bank; stable disagreement flags a badly worded question, not a dispute. Self-preference-bias guard on final acceptance. - **Variance discipline** — improvements on non-critical questions count only when they reproduce in 2 consecutive runs; the 3-iteration cap now carries its STICK rationale. - **`references/pressure-testing.md`** — micro-test protocol (no-guidance control, 5+ reps, variance as a metric) + pressure scenarios for discipline skills (7 pressure types, forced A/B/C choice, rationalization tables). - **Pushy description formula** — `[What] + Use when [4-5 phrasings] + "even if they don't explicitly say ''" + Do NOT use for [...]`, deduped to a single canonical home in Principle #2. - **Question bank v1.1** — 5 new questions: pushy triggers, nuance clauses, directive reference loading, time-rot language, redundant-content (E:A:R). - **Self-hosted proof** — this release was produced by Conductor evaluating and improving itself: 3 gated iterations, dual-family judges (Claude + GPT via codex), all critical questions passing. v3.1.0 — Gated self-update: held-out gate + edit budget (SkillOpt core) - **Held-out gate for body edits** — Mode 2 IMPROVE now splits evals into train/held-out (`scripts/split_evals.py`, deterministic, stratified by optional `evals[].category`). Lessons and edits come from TRAIN only; a candidate is accepted iff no held-out assertion regresses (flip-confirmation re-run for noise), train pass-rate strictly improves, and no new critical failure. Methodology borrowed from microsoft/SkillOpt. - **Edit budget** — at most 3 atomic edits per iteration (one edit = one lesson, labeled); no wholesale rewrites, so gate rejections stay attributable. - **Case transitions** — assertion-level diff parent→candidate (improved / regressed / persistent-fail / stable-success) recorded as an additive `transitions` block in benchmark.json. - **Refactor** — `split_eval_set` generalized into `utils.split_evals(stratify_key=...)`; `run_loop.py` (Mode 5 OPTIMIZE) delegates to it, split unchanged bit-for-bit. v3.0.0 — BinEval scoring, English canon, dual-channel install - **BinEval evaluation** — replaces the old 5-axis 1-10 scoring with atomic binary yes/no questions across 5 dimensions (Discovery, Clarity, Structure, Robustness, Completeness). Each answer carries grounding evidence; the pass criterion is a **gate on critical questions**, not an opaque number. Adapted from *"Ask, Don't Judge"* (arXiv 2606.27226). - **Deterministic + LLM split** — `eval_skill.py --json` emits structural checks as binary question records; an evaluator agent answers the judgment questions with evidence and a self-update loop feeds failing questions back into edits. - **9 authoring principles** — a universal canon (pre-flight, no-process-in-description, MOC, fresh-practitioner author, TWI "why", blind-agent test, inline checklists, one-term-per-concept, cut-the-fat) in `references/sop-practices.md`, applied to every skill. - **Dual-channel install** — one repo, one source of truth, installable via skills.sh and the Claude Code plugin marketplace. v3: SOP practices + smoke tests - **`references/sop-practices.md`** — 80 years of Standard Operating Procedure wisdom applied to skill authoring. Inline checklists at risk-points, pre-flight checks, programmatic validation, exception handling patterns. Use for procedural skills (client intake, onboarding, reporting, escalation) - **`scripts/test_smoke.py`** — fast safety net for skill-conductor scripts themselves. Verifies critical scripts execute on known-good skills, fail on known-bad, produce expected output shapes. Run: `uv run scripts/test_smoke.py` - Updated eval agents (grader, comparator, analyzer) with refined rubrics - Improved `package_skill.py`, `eval_skill.py`, and schema validation - Updated `patterns.md` and `schemas.md` with tighter definitions v2: Anthropic's eval engine meets architecture-first design Anthropic updated their skill-creator with serious eval infrastructure. We took the best of it: **From Anthropic's skill-creator:** - 3 specialized agents: **grader** (assertion checking + claim extraction), **comparator** (blind A/B testing), **analyzer** (post-hoc root cause analysis) - Parallel eval execution with isolated contexts (no cross-contamination) - Automated description optimization with train/test split (60/40) - Benchmark tracking: pass rate, tokens, time with variance analysis - HTML eval viewer with qualitative + quantitative tabs