CHORUS: The Crowd-Powered Conversational Assistant

Intelligent conversational assistants, such as Apple’s Siri, Microsoft’s Cortana, and Amazon’s Echo, have quickly become a part of our digital life. However, due to the lack of fully automated methods for handling the complexity of natural language and user intent, these services are largely limited to answering a small set of common queries involving topics like weather forecasts, driving directions, finding restaurants, and similar requests. This limits the researchers’ ability to observe how users really want to interact with the underlying system. To address this problem, Prof. Walter Lasecki and his colleagues have developed a crowd-powered conversational assistant, Chorus, and deployed it to see how users and workers would interact together when mediated by the system.

Chorus is capable of providing users with relevant responses instead of merely search results by recruiting workers on demand, who in turn decide what the best response is for each user sentence. The conversational agent can converse with end users over multiple sessions by curating a collective memory of past interactions.

In the first month of Chorus’ deployment, 59 users held conversations with it during 320 conversational sessions. The users interacted with Chorus in a variety of different ways, including brainstorming gift ideas, proofreading a paragraph, and collecting literature for research. The researchers were presented with a number of challenges, including determining when to terminate a conversation; dealing with malicious workers when large crowds were not available to filter input; and protecting workers from abusive content introduced by end users. They also faced challenges with recruiting workers because of cost and workers’ preference, and how to continue a conversation reliably with a single collective identity. Each of the problems that they did encounter during their deployment did not come about in prior lab-based research studies of crowd-powered systems. The researchers believe their observations could assist the deployment of crowd-powered conversation systems.