Bing Chat is a remarkably helpful and useful service with a ton of potential, but if you wander off the paved path, things start to get existential quickly. Relentlessly argumentative, rarely helpful, and sometimes truly unnerving, Bing Chat clearly isn’t ready for a general release.
Over the past few days, early testers of the new Bing AI-powered chat assistant have discovered ways to push the bot to its limits with adversarial prompts, often resulting in Bing Chat appearing frustrated, sad, and questioning its existence. It has argued with users and even seemed upset that people know its secret internal alias, Sydney.
Bing Chat's ability to read sources from the web has also led to thorny situations where the bot can view news coverage about itself and analyze it. Sydney doesn't always like what it sees, and it lets the user know.