Kinda Human / Designing With Theory

Dates: November 2018 to December 2018
Team: Scott Dombkowski
Advisors: Stacie Rohrbach and Molly Wright Steenson
Work Type: Academic

In the Designing with Theory exercise, I aimed to better understand the different models of artificial agents that humans create. I believed that gaining insight of such models would help me to create more reputable visions of the future of artificial agents.

Study Protocol

I developed a two-stage study to generate qualitative descriptions of artificial agents/intelligence (i.e., descriptions of participants' interpretations of an agent and its actions) within contemporary artifacts (e.g., Alexa and Google Search), which ultimately resulted in a number of maps that showed the significant role that context and voice play in users' perception of an artificial agent. The first stage of the study included two conversations with artificial agents, while the second stage included a mapping activity.

The mapping activity focused on information retrieval. Information retrieval represented a common task that users would undertake with the chosen agents.

This task encouraged a user to acquire as much information as possible about the movie The Mighty Ducks. I chose The Mighty Ducks as the content for this study because it is one of my favorite movies and a movie that all my participants were somewhat familiar with. As a result, I hypothesized that their conversations with agents would be exploratory in nature.

First Stage

For two minutes, the participant learns as much as they can about The Mighty Ducks. For the first conversation, they use the Google web search engine. For the second conversation, they use an Amazon Alexa.

After the participant has completed both conversations, I constructed maps representing each conversation. Each map visualized the participant's different searches and requests and the responses they received back.

Second Stage

The participant is instructed to analyze and embellish the two maps. I provided a participant with:

  • printed representations of what they typed or said and what they received back
  • rationale indicators, a place to provide the basis for their action
  • interpretation indicators, a place to explain their understanding
  • adjective indicators, a place for quick reflection. Participants were told to quickly write down a couple of adjectives to describe their experience.
Potential Artifacts For Stage Two

I explored the idea of adding emotion dots (i.e., placed where a participant felt a certain emotion), ambiguity cards (i.e., a place for questions that they wished they could ask the interface), and alternative cards (i.e., a place for other actions they considered). I decided not to add these cards because I sensed the responses I would receive would most likely also be included in the rationale, interpretation, and adjective indicators.

Study Administration

I ran this study with five participants over the course of four days. In total, I received ten maps for analysis.

Google Search Map

Amazon Alexa Map

Study Challenges

During the activity I confronted a number of challenges and obstacles including a delay in between activities (i.e., It took me about 30 minutes to generate the initial map for the second activity. By the time I finished generating the map the participant was typically busy doing something else. In most situations, I would need to wait till the next day to complete the activity. It is not entirely clear how this delay affected the study), and my decision to have participant's complete this activity on paper versus orally through a speak-aloud (i.e., I made the explicit decision to have this activity not be a speak-aloud and instead a written activity. I hoped that this would enable me to receive more qualitative responses.).

Study Outcomes

In all the maps, users took very different approaches. Such approaches include looking through the links of one Google search, running multiple Google searches, each building on the last, asking an Amazon Alexa the same question numerous times, or rewording questions to an Alexa when the participant does not receive an answer they desire.

One common thread found in all the Amazon Alexa maps was a feeling of frustration and an inability to receive an answer the participants would deem appropriate.

Study Synthesis

This activity revealed that users saw a Google search as an expansive, logical, and intuitive experience, while they saw an interaction with an Amazon Alexa as a limiting, confusing, and frustrating experience.

All of these factors ultimately affected a user's:

  • conception of speed. Participants saw Alexa as faster initially, primarily because of voice, but slower over time.
  • perceived effort. Participants saw an experience with Google as instinctual and intuitive, while an experience with Alexa as labored.
  • sense of progress. Participants knew when they were getting closer to the answers they wanted with Google, but had no sense of success with an Alexa.
  • sense of control/patience. Participants did not recognize when Alexa had completed a speaking turn, which eventually lead to a loss of patience.
  • testing of boundaries. Participants felt the need to test the boundaries of an Alexa, but not Google.

Two insights from this study stood out:

  • Participants saw Google as having many strong connections. They also saw it as a source that could easily link them to other sources. For instance, one Google web search could link that participant to thousands of other informational sources, all clearly credited. In contrast, Alexa had a few weak connections and seen as a single entity. Participants were unaware of the information's origin and assumed that Alexa did not have the links to the informational sources that Google has.
  • Participants saw Google as having many strong connections and Alexa as having a few weak connections.

  • Web searches allowed participants to create their own context, whether through the use of tabs (note: It would be interesting to understand more about why some users use tabs and others do not) or Google searches specific to a certain site. In contrast, Alexa, had no such mechanisms in place.

A number of participants use tabs to build context for their searches

Both of these factors played a significant role in the models participants created of the two systems. While using Google, a participant's search remained focused over time. The opposite occurred when interacting with an Amazon Alexa, where searches expanded over time. This insight makes a consideration of both the connections and the mechanisms implied through an artificial agent that much more significant. Such a consideration would enable a designer to ensure the models users create of an interface align with the goals of that interface.

With Google, a participant's search focused over time. While with an Alexa, a participant's search expanded over time.