Around 2012, a mechanical problem turned up in some of the diesel cars Volkswagen sold in America. Exhaust hardware was failing on the road.
VW’s engineers suspected they knew why. The cars carried software that recognized the published drive cycles regulators use on a dynamometer, a set of rollers in a lab. On the rollers, the car ran clean. On the road, it cut back the emissions controls. Now cars were staying in test mode too long on real roads, and the hardware wasn’t built for it.
In July 2012 the engineers explained this to two supervisors, separately. Each told them to destroy the document they had used.
Then the engineers did what engineers do with a part that keeps failing. They improved it. The new version of the cheat also checked whether the steering wheel was turning, which it doesn’t on the rollers.
A part that kept failing
On paper, the problem now looked like a component under stress and a detection routine that needed more precision. What the record can’t show is what the people in the room were thinking. It says the two supervisors understood the software’s purpose and significance, and encouraged keeping it hidden.
Treat a choice as a technical or business problem, and its moral side can fade from view.
Seen through that idea, the question worth asking is whether anyone still discussed if the cheat should exist once improving it had become an engineering task. The record shows that some engineers did raise concerns again, and that in April 2013 the head of engine development approved the new function anyway.
In 2004 Ann Tenbrunsel and David Messick named a pattern like this in a paper, “Ethical Fading.” Through self-deception, they argued, people can act in their own interest while “falsely believing that one’s moral principles were upheld,” and the ethical side of the decision fades “into the background.” They pointed to four things that help it along: euphemisms, the slippery slope of small steps, mistakes about who caused what, and the way people see themselves. Ethics training that ignores these forces, they argued, will do limited good. (Tenbrunsel later built on the idea with Max Bazerman in the 2011 book Blind Spots.) That paper is an argument, not an experiment. The evidence that a frame can shift moral choices comes from experiments.
Sears offers a smaller version. Accused by California in 1992 of selling car repairs nobody needed, it denied cheating anyone, but conceded that its sales incentives “created an environment in which mistakes did occur.”
When a fine becomes a fee
In 1999, the pair had tested one piece of it. In three studies, people faced dilemmas where the question was whether to cooperate. A weak sanctioning system produced less cooperation than no sanctioning system at all. The second study suggested why: the sanction changed what kind of decision people thought they were making, “prompting them to see it as a business rather than an ethical decision.”
A field study in Haifa day-care centers by Uri Gneezy and Aldo Rustichini, also told on the crowding out page, pointed the same way. After a small fine for late pickups was introduced, more parents came late, and the number didn’t fall when the fine was removed. The authors argued that the fine “changes the perception of people regarding the environment in which they operate.”
The effect has limits. In a 2020 experimental game, Celia Chui and Manuel Grieder found that as systems for investigating wrongdoing got stronger, people judged it as more unethical, not less. A rule can carry moral weight. And these are games and day-care centers, not boardrooms.
Retention friction
Imagine a product manager asked to cut cancellations. Her dashboard tracks the save rate. The team adds a third confirmation screen and moves the cancel button under “manage plan.” The save rate climbs. The review debates whether the test ran long enough.
Nobody asks how many of the saved customers wanted to leave and are now paying for something they meant to cancel. The meeting was about friction and conversion, and in that language the question has no place to sit.
One thing that brings it back is rewriting the change as a sentence to the customer: “We made it harder for you to cancel.” Another is asking who outside the room pays for the number going up. Both change the frame.
What the model is asked to predict
Building an AI system means choosing what a model learns to predict, a choice that usually looks technical. In 2019 Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan showed how much it can carry.
They examined a commercial algorithm that US health systems used to pick patients for extra care. It predicted future health costs as a stand-in for health needs, and it specifically excluded race. By ordinary measures of accuracy, cost seemed an effective proxy. But unequal access to care, the authors wrote, means less money is spent on Black patients than on White patients. At the score where patients were automatically flagged, Black patients had 26.3% more chronic illnesses than White patients. Removing that gap would have raised the share of Black patients among those flagged from 17.7% to 46.5%.
The authors called the choice of what to predict “perhaps the single most important decision” in building such a tool, and the bias “particularly pernicious because it can arise from reasonable choices.” The manufacturer replicated the results on its own national data. Keeping the same data and training process but predicting an index of health and cost together cut the bias by 84%, by one measure.
This study didn’t examine what the developers were thinking, and the authors described the bias as arising inadvertently. It was a statistical risk score used for screening, not a chatbot. What carries over is the frame. A training target gets filed under settings and judged by accuracy, and the question of who ends up with less care gets no line on the chart.
At Volkswagen, the better cheat reached owners the same way any fix would: as a software update during servicing.