On Wednesday, February 21, 2024, Jack Krawczyk, a senior director on Google’s Gemini team, had some pictures to answer for. Users on X were posting images that Gemini, Google’s AI app, had made on request. German soldiers of 1943 appeared as people of color. Popes and the founding fathers of the United States came in a variety of ethnicities and genders.
“Gemini’s AI image generation does generate a wide range of people,” Krawczyk said. “And that’s generally a good thing because people around the world use it. But it’s missing the mark here.”
The next day, Google stopped Gemini from making images of people at all. The feature had been live for about three weeks.
The range was deliberate. Image generators had been caught making the opposite mistake. A Washington Post investigation the year before, cited by the Guardian, found that Stable Diffusion XL showed food stamp recipients as mostly non-white or darker-skinned, although 63% of recipients in the US are white.
Google’s explanation came on February 23, in a post by Prabhakar Raghavan, a Google senior vice president. When Google built the feature, he wrote, it tuned Gemini against traps seen before: violent or sexually explicit images, depictions of real people. And since its users come from all over the world, a request for football players, or for someone walking a dog, should bring back a range of people. The post also drew a line. Ask for a specific kind of person, or for people in a particular cultural or historical context, and you “should absolutely get a response that accurately reflects what you ask for.”
That line is where it broke. By Google’s own account, two things went wrong. The tuning to show a range “failed to account for cases that should clearly not show a range.” And over time the model grew “way more cautious than we intended,” refusing some prompts entirely and treating very anodyne ones as sensitive. In some cases it overcompensated. In others it was over-conservative.
What the tuning looked like from the inside is harder to know. One user coaxed Gemini into printing what appeared to be its instructions for images, including: “For each depiction including people, explicitly specify different genders and ethnicities terms if I forgot to do so.” The Guardian, reporting it, noted that Gemini might simply have made the text up.
Outside Google, Wendy Hall, a computer science professor at the University of Southampton, pointed to the race. She told the Guardian it looked as if Google had put Gemini “out there before it had been fully evaluated and tested,” because it was in a competitive battle with OpenAI.
On the evening of Tuesday, February 27, Sundar Pichai, Google’s chief executive, wrote to staff about problematic text and image responses in the Gemini app. Some, he said, had offended users and shown bias: “to be clear, that’s completely unacceptable and we got it wrong.” He promised structural changes, updated product guidelines, “improved launch processes, robust evals and red-teaming.”
On March 2, at an AI hackathon in San Francisco, Google’s co-founder Sergey Brin talked about the pictures. “We definitely messed up on the image generation,” he said. “I think it was mostly due to just not thorough testing.”