The math checked out on all 300 questions. 22 of them were still wrong.
Besides this blog, I sell study books for electricians. On September 20 I built a new one: a practice book for the master electrician exam. Six exams, 300 questions, every one of them on the 2023 electrical code. A system did most of the writing. This is about how I checked it, and what the numbers said.
The stakes are simple. A practice exam with a wrong answer key is worse than no book at all. The student studies the wrong answer, trusts it, and walks into the real exam carrying it.
So the system ran in three layers.
- Sources. Every fact came from the code book itself, read and written onto a fact sheet. A question could only use what was on a sheet. Anything that could not be read cleanly stayed out of every question.
- A script. It redid every calculation, checked that the keyed answer carried the right number, and checked that every section a question cited actually exists.
- A second reader. A separate AI that never saw the first one write anything. It worked every question on its own before it was allowed to look at the answer key.
The script passed. The second reader flagged 22 of the 300.
- Loads and services: 5 of 90
- Motors: 5 of 50
- Conductors: 4 of 50
- General knowledge: 3 of 27
- Transformers: 2 of 30
- Special occupancies: 2 of 28
- Grounding: 1 of 25
The math was almost never the problem. In the motors section, all 35 calculated answers matched when the reviewer reworked them from scratch. What it found was stuff a calculator cannot see.
Most of the 22 were questions that left out one deciding fact. One gave 45 kW of electric heat and never said whether it ran continuously. Read it one way and you land on the keyed answer. Read it the other way and you land on option B, and a careful student has two right answers and one key. The fix was to say it. The answer never moved.
One was worse. The correct answer was four small motors at 5.8 amps each, on a branch circuit the question protected at 20 amps. That adds up to 23.2 amps. Defensible under one section of the code, arguable on the circuit itself. It is three motors now, 17.4 amps.
Four explanations carried a sentence copied from a neighboring question. Nothing wrong with the answer. Just a line about a compressor in a question with no compressor in it.
Then a read of the finished book found one more thing. The system shuffles answer choices so the keys do not all land on B. Sixteen explanations still pointed at the old positions. The script checks for that now, so it cannot ship again.
22 out of 300 is about 7 percent. Every section had at least one. None of them would have been caught by the part of the system checking arithmetic, because the arithmetic was right.
The log still lists one more step I would recommend to anyone doing this: a licensed master electrician skimming a sample of the answer keys. A second AI is a good reader. It is not a license.
If something in your business produces work on its own, quotes or schedules or estimate sheets, ask what checks it. A check that reruns the numbers answers one question: is the math right. The question your customer will actually ask is whether it can be read two ways. Only a reader who did not write it can answer that one.