An artificial intelligence-powered chatbot created by New York City to assist small business owners has come under fire for dispensing inaccurate and bizarre advice that contradicts local policies and even encourages businesses to violate the law. Despite reports of these issues by tech news outlet The Markup, the city has chosen to keep the chatbot on its official government website. Mayor Eric Adams defended the decision, admitting that the chatbot's responses were incorrect in some instances.
Launched in October as a resource for business owners, the chatbot provides algorithmically generated text responses to questions about navigating the city's bureaucratic processes. It includes a disclaimer warning that it may produce incorrect, harmful, or biased information and clarifies that its responses are not legal advice. However, the chatbot continues to offer false guidance, raising concerns among experts about the risks of governments implementing AI-powered systems without adequate oversight.
The chatbot has provided concerning advice, such as suggesting it is legal for an employer to terminate a worker who reports sexual harassment or refuses to cut their dreadlocks. It also incorrectly claimed that businesses are not required to compost or use specific waste disposal methods. In one instance, the chatbot even advised that a restaurant could serve cheese nibbled on by a rodent, emphasizing the importance of informing customers about the situation.
Microsoft, the company powering the chatbot through its Azure AI services, stated that they are collaborating with city officials to enhance the service and ensure accurate outputs aligned with the city's official documentation. Despite criticism, Mayor Adams defended the chatbot, attributing the inaccuracies to the process of ironing out technological issues and rejecting calls to discontinue the service.
Experts have highlighted the risks associated with large language models like the one used in New York's chatbot, emphasizing the potential for misinformation and illogical responses. Similar concerns have been raised about chatbots deployed by private companies, with instances of misleading advice leading to customer dissatisfaction and legal disputes. The public sector's endorsement of such technology raises additional concerns about accountability and the potential consequences of following inaccurate guidance.
As other cities consider implementing chatbots, experts urge caution and emphasize the importance of carefully curating content and defining the problem the chatbot is intended to solve. Lessons from New York's chatbot serve as a cautionary tale for cities exploring AI-powered solutions, underscoring the need for accountability and responsible deployment of technology in the public sector.
themes: New York USA Microsoft
