A constraint can't be skipped. A script can.

Every item this product ever sends carries a human-written urgency and a human-written recommendation, or it does not get approved. That is not a policy anyone has to remember — the application will not accept the update. Every write to the database goes through one module, and its rule reads, in essence: an item cannot move to approved unless its urgency, recommended action, and reviewer are all filled in by a person. An automated test fails if any other code writes to the database, so there is no second door around that rule.

One layer up, the same guarantee holds for a different reason. The classifier’s output is a JSON schema with additionalProperties: false and no key named urgency or recommended_actions anywhere in it. The model is not merely instructed to leave urgency alone. It has nowhere to put it — the schema itself makes the omission structural, not a matter of the prompt being obeyed.

What that guarantee doesn’t reach

Both of those are hard constraints: one enforced on every database write, one by the shape of the API call. Neither one, though, looks inside the two free-text fields the model does fill in — summary_md and impact_md. The schema’s description for each field asks the model directly: “No recommendations, no urgency, no imperatives aimed at the reader.” That is a request written into a prompt, not a rule a parser enforces. Nothing stops the model from writing a summary sentence that reads exactly like advice — “manufacturers should update their labeling immediately” is a sentence the schema has no way to reject, because the schema only checks that summary_md is a string.

That gap is where scripts/verify-classify.ts lives. It is not part of the production pipeline — classify.ts, the job that actually drafts items in the database, never runs it. It is a script a developer runs by hand, usually after touching the prompt: it fetches one real, live FDA document — preferring a guidance document, because guidance prose is “the highest-value document type and the one whose prose is richest, so it exercises the prompt hardest” — classifies it for real, and then runs the drafted text through two short lists of regular expressions. One catches advice-shaped phrasing: you should, we recommend, manufacturers must. The other catches urgency-shaped phrasing: urgent, critical, immediate action, time-sensitive. A hit on either list fails the run and prints exactly which phrase tripped it.

A rule enforced on every database write cannot be forgotten. A rule enforced by a script can be forgotten by simply not running the script.

A spot-check, honestly described

It would overstate what this script does to call it a test suite. It exercises one document, chosen for being a hard case, read by whoever is watching the terminal. It costs one live API call, which is exactly why it is not something a CI pipeline runs on every commit against the full corpus — that would mean a paid model call on every push, against documents that don’t change between them. So it is a deliberate human action: run it after a prompt edit, read the summary and impact it prints, and let the two pattern lists do the part of the reading that is easy to miss under a deadline. FDA’s own guidance on software validation makes a point relevant here even though this product is not itself a regulated device: a test plan is only as good as whether it was built to exercise the failure conditions that matter, not just the ordinary path. A regex list aimed at the two sentences this product is built to never write is exactly that kind of case — a boundary condition, checked on purpose, not caught by accident.

The honest inventory, then, has three tiers, not one. The write-path rule on urgency and actions cannot be bypassed by any code this product ships, and a test checks that. The schema’s missing fields cannot be populated by any response the model returns. And whether the model’s prose merely sounds like it crossed the line into advice or urgency is checked by a script that has to be invoked, read, and acted on by a person — the same three-tier distinction worth asking of any vendor who tells you their AI tool “never gives advice.” The question worth asking back is which of those three shapes the promise actually takes.

None of this changes the bright line itself: this product is regulatory intelligence and method, never regulatory advice, and the classifier is never the thing deciding what a document means for your device. See our editorial standards for the rest of what a model is and isn’t allowed to decide here.