On the evening of June 10, 1995, the cruise ship Royal Majesty was approaching Boston from Bermuda with 1,509 people on board. Its screens said it was right on its planned track, within 200 meters.
It was 17 miles off. A loose antenna cable had sent the GPS back to estimating its position more than a day earlier, and the officers had kept steering by it.
On the bridge all along was a second, independent receiver, a Loran-C. It showed the ship’s true position. The officer on watch had testified that it was not his practice to use it to check the GPS. That night the ship ran aground near Nantucket.
Trust as a setting
The National Transportation Safety Board blamed “overreliance on the automated features” of the ship’s bridge system. Human factors researchers had a more precise word for what went wrong. In a 2004 review, John Lee and Katrina See of the University of Iowa called it a failure of calibration: the match “between a person’s trust in the automation and the automation’s capabilities.”
Calibration can fail in two directions. Overtrust, trust above what the system can do, leads to misuse, like the Royal Majesty’s officers following a GPS that had stopped knowing where it was. Distrust leads to disuse. Lee and See’s example was paper mill operators who rejected automated controllers and lost what the automation could have given them.
Lee and See also pointed out that good trust is specific. It attaches to particular functions of a system, not the system as a whole, and it changes as the situation changes. On the Royal Majesty, the officers’ confidence in the GPS stayed where it had been when it worked. The small “DR” on its screen, the sign that it had stopped using satellites, didn’t move it.
A week of checking
People calibrate trust all the time, usually without naming it. Imagine a warehouse supervisor who gets new scheduling software that assigns pickers to orders. For the first week she checks every assignment by hand. Standard orders come out right. Rush orders, with their special packing rules, sometimes don’t. So she stops checking the standard ones and keeps checking the rush. Months later, after a software update, she spends another few days checking everything before relaxing again.
Her trust follows the software’s record, differs by type of order, and goes back to checking when something changes. It is more work than either trusting the system or ignoring it.
When the AI tells you how sure it is
AI can make calibration harder. In the BCG study of 758 consultants, reports written with GPT-4 were rated more coherent and persuasive even when their conclusions were wrong. Polish is no guide.
One obvious fix is to have the AI say how confident it is. In 2020, researchers at IBM tested that. Participants predicted whether people in 1994 US census records earned more than $50,000 a year, with an AI’s suggestion alongside. When the AI also showed its confidence score, people followed it more on high-confidence cases and less on low-confidence ones. Trust became better calibrated. But the final decisions did not get more accurate. The cases where the AI was unsure were the cases where people were unsure too, so taking over at those moments added little.
In that experiment, better-calibrated trust didn’t make decisions more accurate, because people’s own judgment didn’t cover the machine’s weak spots. The Royal Majesty points to what can: a check that doesn’t run through the same data, such as a second source or knowledge the machine lacks. On the ship, that was the Loran-C. In an office, it might be the original contract, a call to the customer, or a colleague who knows the account.
So for the AI work that matters, keep a record of how it does on your kinds of task, kept separately for each kind. Re-check after a model or tool changes. And keep one independent instrument switched on.