“Yes, you can take a cut of your worker’s tips.”
The answer was wrong under New York law. It came from New York City’s own business chatbot, on a city website, in tests The Markup published in March 2024.
The bot had arrived with fanfare. On October 16, 2023, Mayor Eric Adams released a citywide plan for using artificial intelligence and, alongside it, a pilot of New York’s first citywide AI chatbot on a new MyCity Business site. It would give business owners “trusted information from more than 2,000 NYC Business web pages,” the city said, on topics like complying with codes and avoiding violations and fines. It ran on Microsoft’s Azure AI services.
That March, the city’s small business commissioner, Kevin D. Kim, talked up the bot on a panel. Trust, he said, was the most important thing when governments use AI. He alluded to Air Canada, whose chatbot had given a customer the wrong bereavement refund rule and ended up in a lawsuit. “That can’t happen to government,” Kim said. “We cannot be in a situation where we lose that kind of trust even one time.”
On March 29, The Markup, with Documented and The City, published what the bot had been telling people. Asked whether buildings must accept Section 8 vouchers, it said no. In New York City, landlords can’t turn tenants away over where their rent money comes from, apart from a narrow exception for small buildings where the landlord or their family lives. Once, a reporter got the right answer. When ten separate staffers asked the same question, all ten were told no.
It told restaurants they could go cash-free, though a 2020 city law requires businesses to accept cash. It said there were no rules about telling staff of schedule changes. The citywide housing director at Legal Services NYC, a legal aid group, tried it after hearing from The Markup and was told it was legal to lock a tenant out. The errors turned up in other languages too.
The page had a warning, one small note saying the bot “may occasionally produce incorrect, harmful or biased content.” The same page said it was “trained to provide you official NYC Business information.” A city spokesperson said the pilot “has already provided thousands of people with timely, accurate answers.”
Four days after the story, Adams addressed it at a press conference. The bot would stay up. “We’re identifying what the problems are, we’re gonna fix them, and we’re going to have the best chatbot system on the globe,” he said. His chief advisor compared it to the early days of MapQuest.
What changed first was the page around the bot. It now called the chatbot “a beta product” and told visitors not to use its responses as legal or professional advice. Later, users had to agree to its limitations before they could type a question.
The bot outlasted the administration that built it. In January 2026, facing a $12 billion budget gap, the new mayor, Zohran Mamdani, singled it out as a place to save money. “The previous administration had an AI chatbot that was functionally unusable,” he said. By his account, it had cost around half a million dollars.
By February 4 it was gone. In its place, the page said the chatbot’s “beta test has ended.”