In 2023, Boston Consulting Group let a team of social scientists from Harvard, MIT, Wharton and elsewhere run an experiment on its own people. 758 consultants took part, about 7% of the firm’s individual-contributor consulting staff. Some worked as usual. Others got GPT-4, then the most capable AI model available, and some of those also got a short training on how to prompt it.
About half the consultants spent the session on a product launch for a fictional shoe company: 18 tasks, from “Propose at least 10 ideas for a new shoe targeting an underserved market or sport” to a press release and a memo to staff. With GPT-4, they finished 12.2% more of the tasks, took roughly a quarter less time, and turned in work that graders scored markedly higher.
The other half got a classic business case, the kind BCG uses in its job interviews. A fictional company’s CEO wanted to know which of its brands had the most room to grow. The consultants had a spreadsheet of sales figures and a file of interviews with company insiders. The spreadsheet looked complete. The answer depended on details buried in the interviews, and the researchers had built the case so that GPT-4, given everything, would get it wrong.
Working without AI, consultants picked the right brand about 84.5% of the time. With GPT-4, the two AI groups managed 60% and 70.6%. On average, the help cost them 19 percentage points.
There was a second finding inside the first. Graders who didn’t know the right answer rated each report for clarity, coherence and persuasiveness. The AI-assisted consultants scored higher on that, whether their recommendation was right or wrong. In the paper’s words, AI “improves presentation and argumentation even when the underlying analysis is flawed.”
The researchers called the boundary a “jagged technological frontier.” On one side of it, AI raises the work of even highly skilled people. On the other, it drags them down. Tasks that look about equally hard to a human can sit on opposite sides, and nothing on the screen tells you which side you’re on.
Writing about the study, Ethan Mollick, one of the authors, described two styles among the consultants who used AI well. Some worked like centaurs, splitting the job cleanly between what they did and what they handed to the AI. Others worked like cyborgs, passing pieces back and forth all the way through. Either way, the person kept deciding which pieces went to the machine.