Every agent got a perfect score

I wanted to know whether giving an agent guidance for systems engineering helped it make better decisions. Three agents took the same 24 development questions. One had no packaged skills, one had instructions for the task, and the third also had guidance on the subject. Each matched the proposed answer key on all 24.

Look for a shortcut

Before using a score to judge engineering work, try to earn it without doing the work. What else could someone use to pick an answer?

Pick the longest answer

An independent reviewer tried a simpler rule on the full question bank. Picking the longest answer matched the key on 54 of 55 single-choice questions. You can inspect the results and the reviewer’s check.

Try the hospital transport question

In this fictional hospital, a new portal cuts the time to enter a transport request from six minutes to two. Transport still starts after forty minutes. Several wards share the same porters. What would you investigate next?

Write down your answer. Then open the four choices and pick one by length alone. Compare both answers with the proposed key. What would you need to see in an agent’s explanation to judge whether it understood the delay?

What the scores leave open

The quiz gave us no way to tell whether the guidance helped. All three agents also ran on a host that showed them the skill descriptions, so the three setups were not fully separate.