Methodology: expert councils

Draft 1, 2026-11-01. This page describes how E3D Pilot forms a council from expert skills, models what it costs, and sets out how we would know whether it helps. Nothing in the results section has been measured. The equations are models, the numbers below are illustrations, and both are labelled as such. See Experts for what exists today.

Definitions

Term	Meaning
Adapter	A provider script that answers a prompt, for example one model service.
Vendor family	The company behind an adapter. Two adapters can share a family.
Skill	An expert: a reusable lens that tells one seat what to look for.
Seat	One participant for the whole debate: a pair of an adapter and a skill. One adapter can hold several seats; one skill is named once.
Round	One pass in which every seat answers once. Round 1 is independent; later rounds read the other seats' previous answers.
Council	The seats, plus a synthesis call that writes the verdict with its consensus level and any dissent.
Multi-model	A council on two or more vendor families. A council on one family is labelled as such and is never called multi-model.

The equations

Notation: n seats, r rounds, Q the question, B the fixed prompt text, L the lens block, A an answer, S the synthesis output, p the chance one seat finds a given problem, ρ the average correlation between seats' misses.

E1. Panel

A council is a set of distinct skills, each on one adapter, with a cap on size. You name the skills; the engine chooses the adapters.

P = {(a_{i}, s_{i})}_{i = 1}^{n}, s_{i} \neq s_{j} (i \neq j), n \leq n_{\max} = 6

E2. Assignment

Each skill goes to the adapter that is least used, vendor first: the fewest seats in its vendor family, then the fewest on the adapter itself, then the skill's model hint (a tie-break only), then pool order. A skill can be pinned to an adapter.

a_{i} = {arg min}_{a \in pool} (seats (f (a)), seats (a), h_{i} (a), π (a))

E3. Token cost

From round 2 every seat reads every other seat's previous answer, so cost grows faster than the seat count.

In = n (Q + B_{1} + L) + (r - 1) n (Q + B_{2} + L + (n - 1) A) + (Q + B_{s} + n r A)

Out = n r A + S, T (n, r) = In + Out, calls = n r + 1

A skill on a seat you already have adds only n*r*L tokens. An extra seat adds a whole call every round.

E4. Effective seats

Seats that miss the same things count for less.

n_{eff} = \frac{n}{1 + (n - 1) ρ}

E5. Coverage

A rough model of the chance the council catches a given problem. The hypothesis behind skills is that a lens lowers ρ between seats on one model.

P (caught) \approx 1 - (1 - p)^{n_{eff}}

E6. Lift and its uncertainty

S_X(t) is a score on task t, for example the share of planted errors caught, over m tasks and k repeats each. Run-to-run noise is large, so every comparison needs repeats.

Δ = E_{t} [S_{X} (t) - S_{Y} (t)], SE \approx \frac{σ_{d}}{\sqrt{m}}, σ_{d}^{2} \approx σ_{task}^{2} + \frac{σ_{run}^{2}}{k}

E7. Decision rule

Fixed before any result is seen: setup X is claimed better than setup Y only if all four hold. A lift of at least 15% of the baseline, an interval that excludes zero, at most three times the cost, and false alarms no more than 10% worse.

Δ \geq 0.15 {\overset{ˉ}{S}}_{Y} \land Δ - 1.96 \frac{σ_{d}}{\sqrt{m}} > 0 \land \frac{T_{X}}{T_{Y}} \leq 3 \land {FP}_{X} \leq 1.10 {FP}_{Y}

Otherwise the result is reported as inconclusive and no claim is made. The bar is not loosened afterwards.

E8. Cost per extra finding

F is the number of valid findings confirmed by a person.

κ = \frac{T_{X} - T_{Y}}{F_{X} - F_{Y}}

Worked numbers

The cost constants are assumptions for a short question, not measurements: Q 100, B1 300, B2 500, Bs 400, L 130, A 300 and S 400 tokens. Real debates are heavier; the ratios between setups are what the model is for. Dollars are the same sums with each vendor's price, which this page does not state.

Council	Seats	Rounds	Calls	Total tokens	vs plain 4 seats, 2 rounds
2 seats, no skills	2	2	5	5,900	0.4x
4 seats, no skills (today's council)	4	2	9	13,300	1.0x
4 seats, a skill each	4	2	9	14,340	1.1x
3 skills, all on one adapter	3	2	7	10,080	0.8x
6 seats (the cap)	6	2	13	24,660	1.9x
8 seats (above the cap)	8	2	17	37,380	2.8x
8 seats, one round (breadth only)	8	1	9	9,940	0.7x
4 seats, one round	4	1	5	5,420	0.4x
4 seats, three rounds	4	3	13	23,260	1.7x
4 seats, four rounds	4	4	17	32,180	2.4x

Coverage, illustrative only, with p = 0.4 and n = 4 (one seat alone gives 0.40):

Seats are	ρ	n_eff	Chance caught
fully independent	0	4.0	0.87
somewhat alike	0.5	1.6	0.56
very alike (one model)	0.8	1.2	0.45

Four seats that are close to identical add little. What matters is how correlated the seats are.

How we would measure it

Each row below is a council setup; every comparison changes one thing.

Setup	Compared with	What it isolates
one strong model	(baseline)
today's four vendors, no skills	one model	does a council beat one model?
one model sampled three times and merged	four vendors	vendor diversity, or just more samples?
one model, three skills	one model sampled three times	do skills add anything, vendor held fixed?
skills spread over four vendors	one model, three skills	what vendor spread adds on top of skills
random skills	skills spread over four vendors	whether choosing skills well is worth anything

What is counted: planted errors caught (checked by a script, no grading), valid points and false alarms (confirmed by a person and a second model), whether the verdict differs from the baseline, run-to-run stability over three repeats, and tokens and time. Tasks are 40 to 50 documents with ground truth where it exists (planted errors, known outcomes) and a blind grader where it does not. A cheaper first test comes before the full run: one model with three skills against one model sampled three times, on planted errors, which the model above puts near half a million tokens.

Results

Comparison	Delta	95% interval	Cost ratio	Status
council vs one model	pending	pending	pending	not run
vendors vs samples	pending	pending	pending	not run
skills vs samples	pending	pending	pending	not run
vendor spread vs skills	pending	pending	pending	not run
chosen vs random skills	pending	pending	pending	not run

Observations so far are anecdotes, not results. In two real runs on one question, each seat's emphasis followed its skill (a skeptic naming failure modes, a customer speaking as a user), and both runs ended "yes, conditionally" with the skeptic as the dissenter.

What is and is not claimed

Claimed: the definitions, the cost model, and a plan for measuring. Not claimed: that skills improve answers, that a council beats one model, or that any ranking of skills or vendors exists. No claim about answer quality is made until the results above have numbers and the decision rule is met.

Limits: ρ is unknown and may differ by task; the coverage model assumes equal p and equal pairwise correlation; the cost constants are for short questions; planted-error tasks may not represent open decisions; graders are subjective; a rule-based suggestion matches words and does not understand a question; a one-vendor council is not independent and is never described as multi-model.