Beacon Chat

Every number is graded before it is spoken. Counter-questions narrow the scope until "at least N" becomes "exactly N," and a sample of citations becomes the complete set.

It will not tell you it "found 10" while showing you 10.

On a construction site, "how many safety findings are on level 3?" is not small talk. That number stops work and settles subcontractor invoices.

And the most common way an LLM chatbot gets a number wrong is not hallucination. It is a chain of layers where nobody lied and the chain lied anyway: the search engine sends a true total, the middle server drops that field while parsing, the chat server silently shrinks the requested count, and the model counts what it received and says "I found 10."

This project breaks that chain with a grading system — and then goes one step further: it raises the grade through conversation.

---

What it is

An LLM chat server (TypeScript) for a reality-capture construction B2B SaaS, plus a panel UI (Angular) that mounts into two different host applications. Users ask questions in natural language about 3D-captured job sites.

The model narrates only. Claims about counts, absence, and failure are adjudicated by code.

How far it has been verified

Three internal demo tiers (CLI → browser → full UI) were run end-to-end against real dev and qa deployments with real accounts and real tokens, and passed. The final tier is feature-complete and internally verified; current work is UI/UX polish.

There is no production end-user traffic yet — no figure in this document claims user scale.

Scale

The test suite is 2.14× the size of the source it tests — 25,283 LOC of server source vs 54,227 LOC of tests (~2,817 cases)

Panel UI 10,284 LOC · 16 browser e2e specs · 116 state-transition rows (illegal transitions included) · 17 CLI reproduction scripts · 103 registered contract boundaries · 18 releases in 31 days.

My role

Design and implementation of the server, the panel, and the verification harness. Drove two API contract changes in a system I had no write access to, using measurement reports as the lever.

---

What makes it different

#	Typical (ChatGPT clone · RAG tutorial · MCP tool-call demo)	This project
1	The model counts the array it received and says "I found N"	Every number that is a candidate for utterance is graded before the model speaks — EXACT / LOWER_BOUND / SHOWN / UNKNOWN — and the grade dictates the permitted sentence form. UNKNOWN forbids stating a number at all
2	Asks a clarifying question when the request is ambiguous (a prompt rule)	The counter-question is a promotion operator: once a narrowed scope fits in one page with nothing beyond it, a sample of citations becomes the complete set and the grade goes up
3	Empty tool result → "I couldn't find anything"	Asserting absence requires standing. The server may only say "there is none" for a payload the upstream accounted for as its own search result. Without standing: silence
4	Tool failure is whatever isError says	Detects failures disguised as HTTP 200 by the absence of a result envelope. Error text is never used for the verdict — so an upstream rewording cannot silently restore the false green
5	Permissions are enforced by system-prompt instruction	Blocked before the tool call goes out. The data never arrives, so leakage is structurally impossible — and the planner is deliberately blind to licensing vocabulary (enforced by a vocabulary gate). 14 dedicated test files / 131 cases on this axis
6	A happy-path screenshot is the whole demo verification	116 transition rows including illegal transitions + 17 CLI reproductions + 16 browser e2e specs — and the stub server the e2e suite stands on is itself contract-tested
7	Built and shipped solo	Changed an external team's API contract twice by measurement report (backend fetch cap 30→200, cursor pagination · reachable set 16→38). Where their delivery deviated from the request, our side sealed it with invariants

---

Three representative cases

① When upstream says "that's all," the code refuses to believe it. The data-query tool injects a row cap into the SQL it generates. When results saturate that cap, upstream pins the total to the cap value and reports has_more: false once pages are exhausted — measured on a project holding 7,961 panoramas: total 3,000, no more. Upstream was confidently wrong. Our grading module detects saturation, reclassifies the total as a lower bound, and forces hasMore: true. The proof that this reclassification cannot produce a falsehood in either direction is written into the source comment.

② There was a code path under a green CI that had never once worked. Four demo scenarios failed on their first real run. I suspected a recent regression and traced the history string by string — the defect was present in all six release tags. Not a regression: dead from birth. Yet 21 work items above that path were already marked complete, and every rule had been followed (merge verification passed 100%). The response was not a bug fix but a pipeline change: mandatory boundary-crossing tests, release refused if the contract sweep is stale, 103 contract boundaries registered, and a new rule — "when you build a gate, answer in the same commit what invokes it." Applying that rule to existing assets immediately surfaced three more dead gates, one of which was a pre-deploy smoke test exiting with code 127 that had never run for anyone doing a clean install.

③ A failure arrived disguised as a success. An upstream API returned 502, but it reached us as isError: false plus an ordinary text block. Our only failure detector was isError, so the model took the error string as data and composed an answer on top of it — and no failure signal reached the user anywhere. The fix was not "zero results means failure." Measuring in the opposite direction showed that genuine zero-result responses carry a result envelope too. So the discriminator became the absence of the envelope, and error wording is never classified — so an upstream rewording cannot break it.

---

Limits

• Latency target not met. The spec target is P95 first-token < 3.5s; measured reality is ~30s cold, under 20s warm. Half the cause is the model-input compaction step that is designed but not yet built — and that is the next priority.
• No production metrics. Instrumentation (metrics emission, warehouse ingestion) is implemented, but there is no traffic to aggregate yet. User scale, error rate, and cost are left blank on purpose.
• Prompt-layer compliance is not guaranteed. Honesty rules that live as model instructions are tested only for injection, not obedience. A case where the model disobeyed was observed in production probes; what saved the user-visible outcome was the deterministic UI layer, not the prompt.
• Exhaustive permission matrix not yet landed. The specified 120-cell exhaustive verification has not shipped; current coverage is the 14 files / 131 cases described above.

---

---

Note on this document: Project codename Beacon · 2026-08. Company, product, personal names, and internal hosts are generalized; code symbols and figures are real.