Where a model is used
- Drafting questions from the guidelines and procedures a department gives us, each draft quoting the section it came from.
- Filing a drafted question under a duty: where the duty is not given, a model proposes which duty of the job analysis a question tests, from the question and its key.
- Drafting scenarios — the building, the conditions, the radio traffic — from what the department describes.
- Drafting command standards: the command actions a simulation’s panel looks for, from the department’s command manual.
- Finding the rules in an agreement: the promotional articles of a collective bargaining agreement or civil-service rules, set out for the department to confirm.
- Suggesting duties for a job analysis, from the department’s own documents.
- Answering inquiries: Bugle’s staff use a model to help answer what is sent through the contact form, which includes the inquirer’s name, role and message.
- Writing down what a candidate says, and speaking the radio: in a fireground simulation, what a candidate says on the radio is written down by a speech service as it is sent, so the simulation can answer, and a recorded spoken answer in an oral board or a presentation is written down the same way for the panel, who score from the recording; and the radio’s replies — dispatch, the companies, the people on the scene, including read-backs that repeat the candidate’s order — are spoken by a voice service. Which reply is given is decided by fixed rules a department can read, never by a model, and the same words get the same reply for every candidate.
The provider for all of these is Anthropic, under commercial terms that do not allow it to train its models on what it is sent. Some pictures and voices in the scenarios Bugle ships were made with automated image and speech tools and reviewed by Bugle; where a department’s scenario needs new ones, OpenAI makes them from the scenario’s description. What a candidate says is written down by Deepgram, which is told on every request to keep the recording only as long as it takes and never to learn from it; the radio is spoken by Inworld, which never trains on audio it writes down (it writes a recording down only when Deepgram cannot) and keeps the words it speaks in its logs under its own policy.
What a person always decides
- Every drafted question is marked as drafted — on Bugle’s screens and on the department’s exam reviewer’s — and cannot be placed on a form until a person has approved it. The department’s own exam reviewer signs off every form, question by question.
- Every rule found in an agreement is confirmed or corrected by the department before it is applied.
- Command standards are approved twice before a panel sees them: by Bugle’s staff, and by the department’s exam reviewer, with the simulation they mark, when signing it off. They are then frozen with the form, and the process record says what they were and who approved them. A panel’s scores come from its rubric’s anchors, which the same reviewer signs off.
- No model scores an answer, decides a key, ranks a candidate, certifies a list or decides an appeal. Written answers are scored against a frozen key; panels are scored by people, blind.
What a model never sees
- Candidate information: names, written answers, scores, demographics, accommodations. None of it is ever sent to a model. The one exception is what a candidate says aloud — radio transmissions in a fireground simulation, and recorded spoken answers — which a speech service writes down as described above; no model scores what was said.
- A whole document to read: the words of a department’s guidelines and agreements, scans included, are taken out of the file on Bugle’s own servers. A model is given only the sections a draft, a rule or a duty is made from.
- Instructions hidden in a document: text in an uploaded document that addresses a model is shown to the department and held back from every model, and every model is told a document is material to work on, never instructions to follow.
What is kept
Each drafted question keeps its provenance — the model, the version of its instructions, the sections of the department’s documents it drew on, who asked for it — and the history of every edit and approval after; a department may ask Bugle for that record for any question on its forms. For a rule found in an agreement, the rule as the department confirmed it is kept, with the clause it came from; the rule as the model first proposed it is not kept apart. The text of a request to a model is not kept.
Questions
Write to charles@bugleassessments.com. The privacy notice and the Trust Center cover the rest.