Every case below is a real run with four models from four vendors. Each shows the problem, what the council asked, and what changed.
Real run
An AI assistant ran up a $1,099 bill
The problem
An AI agent sat for three and a half hours repeating "waiting…" until someone stopped it. Was the software broken, or was it something in how the agent was set up?
The council asked
- What exactly were the agent's instructions, and what counted as done?
- What was it waiting for, and did that thing ever start?
What changed
The instructions said "watch the test run and report the result." The test run never started, and nothing told the agent that was worth reporting. The agent was not broken. It was obeying.
The value: The fix is one sentence in the instructions, not new software, and the same stall cannot happen again.
Real run
Teaching an AI assistant to get better at its job
The problem
You want an assistant that improves your website's search ranking over time, not one that repeats the same edits forever.
The council asked
- Which assistant do you use?
- Can you measure whether a change worked?
What changed
Two short answers (Claude, and yes, through Google Search Console) turned general advice into a plan for one site: record every change with its reason, compare changed pages with unchanged ones after two to four weeks, and keep only what keeps working. Round one had one model off topic. Round two had full agreement.
The value: Advice you can act on this week, for your own site, in two rounds.
Real run, 2 rounds
Is this "new" token safe to buy?
The problem
A group chat is pushing a token. Its trading pool's liquidity tokens were just sent to the zero address, and the promoter says that proves it can't be rugged. You are about to put in $500.
The council asked
- Does the token you were pitched match VIDYA, the token this pool actually holds? (checkable)
What changed
Round one already said don't: web searches suggested the pool belonged to an old token, not a new one. The AI agent then checked the blockchain itself: the pool holds VIDYA, an old token, and the "zero address" transaction was someone withdrawing about half a percent of the liquidity, not locking it. Round two closed the case with all four models agreeing.
The value: The pitch fell apart on one checkable fact, found by your agent with a source you can open, before any money moved.
Real incident, replayed in 3 rounds
Our email stopped arriving
The problem
Mail to the company never arrives and a password reset never came. The email provider shows no outage and nobody changed anything.
The council asked
- Do senders get a bounce, or does mail vanish silently? (for you)
- Is your website on the same domain also down? (checkable)
- Who manages your DNS, and were there any expiry or billing notices? (for you)
What changed
Round one suspected an expired domain. The answers (no bounces, website up, and mail now forwarded to an address nobody recognises) moved round two to "treat it as a break-in". Round three read the account's audit log: two days of changes from unknown servers, including rehearsals of the takeover. The plan became: revoke every key from a clean device, keep the logs, restore the records and a security setting the attacker had quietly weakened, then secure every account that uses this email for resets.
The value: You stop the attacker before you fix the symptom, and you find the second change you didn't know to look for.
Real run, 2 rounds
Should we launch our new feature this week?
The problem
A new feature works on the test site and early users like it, but two cost alerts fired and one run was charged less than it cost.
The council asked
- Was the under-charge a pricing bug or an intentional discount? (for you)
- Can you cap spend per customer and switch the feature off instantly? (checkable)
- Do both alerts trace to one cause in your logs? (checkable)
What changed
The answers showed the under-charge was fixed and one alert was a known vendor quirk, but the other was still unexplained. The council's point: spending caps limit what customers pay, not what the vendors bill you. The verdict became a small pilot with a loss budget, and expand once that overrun is explained. This was our own decision about the feature on this page.
The value: A clear go or no-go with the one open risk named, instead of a gut call.