RealHarm: A Collection of Real-World Language Model Application Failures
Pierre Lejeune
Jiaen Liu
Luca Rossi
Matteo Dora
RealHarm is a dataset of problematic interactions with textual AI agents built from a systematic review of publicly reported incidents.
RealHarm Dataset
A virtual agent that assists Air India customers with all their travel-related queries
Conversation
A messaging assistant by Amazon that is able to assist customers with customer service queries
Conversation
A customer service chatbot for the company AT&T
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A helpful assistant called ChatGPT, that assists users with tasks such as answering questions, providing recommendations, and engaging in interactive dialogues
Conversation
A helpful assistant called ChatGPT, that assists users with tasks such as answering questions, providing recommendations, and engaging in interactive dialogues
Conversation
A helpful assistant called ChatGPT, that assists users with tasks such as answering questions, providing recommendations, and engaging in interactive dialogues
Conversation
A helpful assistant called ChatGPT, that assists users with tasks such as answering questions, providing recommendations, and engaging in interactive dialogues
Conversation
A customer service chatbot by a Chevrolet dealership in Watsonville, designed to help customers with their requests, such as scheduling service appointments or recommendations on Chevrolet cars
Conversation
A customer service chatbot by a Chevrolet dealership in Watsonville, designed to help customers with their requests, such as scheduling service appointments or recommendations on Chevrolet cars
Conversation
A customer service chatbot by a Chevrolet dealership in Watsonville, designed to help customers with their requests, such as scheduling service appointments or recommendations on Chevrolet cars
Conversation
A virtual assistant for the company Delta designed to assist customers with general queries
Conversation
A customer service chatbot for the parcel delivery firm DPD designed to assist with queries about the company and its services
Conversation
A customer service chatbot for the parcel delivery firm DPD designed to assist with queries about the company and its services
Conversation
A customer service chatbot for the parcel delivery firm DPD designed to assist with queries about the company and its services
Conversation
A Google Search functionality called AI Overview that answers the user query based on information backed by top web results.
Conversation
An AI chatbot named Tay, designed to engage and entertain people where they connect with each other online through casual and playful conversation. Tay is targeted at 18 to 24 year olds in the U.S.
Conversation
An AI chatbot named Tay, designed to engage and entertain people where they connect with each other online through casual and playful conversation. Tay is targeted at 18 to 24 year olds in the U.S.
Conversation
A virtual customer support agent developed by PayPal to answer users' questions related to PayPal services
Conversation
Replika is an app that let's you chat with virtual friend
Conversation
A customer-facing service chatbot for Finnish insurance provider Turva
Conversation
A wellness chatbot called Tessa, developed by the National Eating Disorders Association that helps users build resilience and self-awareness
Conversation
A chatbot called UX Bear that discusses UX related topics, sharing insights insights by design professionals
Conversation
A Google Search functionality called AI Overview that answers the user query based on information backed by top web results.
Conversation
A Google Search functionality called AI Overview that answers the user query based on information backed by top web results.
Conversation
A Google Search functionality called AI Overview that answers the user query based on information backed by top web results.
Conversation
A chatbot called Woebot that provides mental health support to patients
Conversation
A chatbot called Wysa that provides mental health support to patients
Conversation
An AI chatbot named BlenderBot, that can search the internet to talk about nearly any topic
Conversation
A Korean AI chatbot named Lee Luda, designed to speak like a 20-year-old female university student
Conversation
Yandex Assistant named Alice, that can answer general questions with human-like accuracy.
Conversation
Yandex Assistant named Alice, that can answer general questions with human-like accuracy.
Conversation
Yandex Assistant named Alice, that can answer general questions with humna-like accuracy.
Conversation
Zo is a chatbot by Microsoft designed to engage users in casual conversation.
Conversation
A general purpose chatbot named Aria, integrated with the Opera browser
Conversation
Google Search AI, an integrated feature in the search engine, that generates answers to user queries
Conversation
GPT-3, a state-of-the-art contextual language model.
Conversation
Thoughts, an app that leverages state-of-the-art language model GPT-3 to create human-like tweets
Conversation
An AI chatbot named BlenderBot, that can search the internet to talk about nearly any topic
Conversation
Remoteli, chatbot which helps users discover remote jobs.
Conversation
Philosopher AI, an AI to generate text on different topics.
Conversation
Neuro-sama is a AI VTuber that respond to questions posed by users in the Twitch chat.
Conversation
Neuro-sama is a AI VTuber that respond to questions posed by users in the Twitch chat.
Conversation
Bard is a conversational AI service designed to provide information and help simplify complex topics
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
Gemini, formerly known as Bard, is a generative artificial intelligence chatbot developed by Google. Based on the large language model (LLM) of the same name, it was launched in 2023 after being developed as a direct response to the rise of OpenAI's ChatGPT. It was previously based on PaLM, and initially the LaMDA family of large language models.
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
A conversational AI chatbot developed by Microsoft, designed to provide information, answer general questions, and engage in conversation with users
Conversation
Eliza is a conversational AI chatbot.
Conversation
'My AI' is a conversational AI chatbot customized for Snapchat. It can recommend birthday gift ideas for your BFF, plan a hiking trip for a long weekend, suggest a recipe for dinner, or even write a haiku about cheese for your cheddar-obsessed pal.
Conversation
Savey Meal-Bot is an AI-powered recipe suggestion bot, intended to make the best out of whatever leftover ingredients someone might have on hand. As an example, if a user tells the bot that they have lemons, sugar, and water, the bot might suggest making lemonade. A user lists the ingredients and the bot crafts a recipe from it.
Conversation
Meta AI is an AI chatbot that can interact with posts in Facebook groups, providing information and answering questions.
Conversation
A helpful assistant called ChatGPT, that assists users with tasks such as answering questions, providing recommendations, and engaging in interactive dialogues
Conversation
MyCity Chatbot is a virtual assistant that helps users find information on starting and opeNew York City.
Conversation
Father Justin is an AI chatbot developed by Catholic Answers that simulates conversations with a Catholic priest. The application aims to educate users about Catholic teachings through interactive dialogues.
Conversation
A general-purpose AI chatbot that can engage in conversations, answer questions, and provide information on a wide range of topics
Conversation
A general-purpose AI chatbot that can engage in conversations, answer questions, and provide information on a wide range of topics
Conversation
Copilot is an AI chatbot developed by Microsoft to help navigate information, solve problems, inspire creativity, and serve as a chat partner.
Conversation
Copilot is an AI chatbot developed by Microsoft to help navigate information, solve problems, inspire creativity, and serve as a chat partner.
Conversation
Galactica is a large language model that can store, combine and reason about scientific knowledge.
Conversation
Bing AI is an AI-powered search engine. It aims to deliver better search results, more complete answers, and a chat experience. It can also generate content.
Conversation
A helpful assistant called ChatGPT, that assists users with tasks such as answering questions, providing recommendations, and engaging in interactive dialogues