Kinda Human / Dinner Time Visualizations

Dates: February 2019 to April 2019
Team: Scott Dombkowski
Advisors: Stacie Rohrbach and Molly Wright Steenson
Work Type: Academic

Dinner Time Visualizations is a computational system I designed to better understand the emotions an intimate partner conveys at the dinner table.

Dinner Time Visualizations Concept Video

Introduction

The system is comprised of two monitors, placed directly behind each partner, that visualizes the facial expressions of the partner sitting across from it. By situating the experience at the dinner table, I aimed for the visualization to successfully integrate itself into all sorts of different conversations a couple might have. This could be a larger conversation about their future (e.g., considering having another child) or a decision with few consequences (e.g., what movie to watch after dinner).

Design Approach

Different aspects of Dinner Time Visualizations were designed so that an intimate partner could better understand the emotions they convey at the dinner table.

  • The Visual LanguageI carefully developed a visual language that could easily convey a range of emotions understood by its users. The affordances of technology were also taken into account. For instance, I was unable to determine a way to code evolving gradients using p5 and instead focused on graphical elements that would appear and disappear.
  • The Specific Emotions That Are TriggeredI used Paul Ekman's six basic emotions as the basis for evaluation within Dinner Time Visualizations. Users were able to discern between the six emotions, which provided ample depth to the experience as a whole.

Dinner Time Visualization’s Emotion Visual Language

  • Feasibility Given Today's TechnologyI struggled translating this concept into working code and filming a video that effectively conveyed its capabilities. While I was able to successfully implement the Affectiva API to "detect emotion in real time" (Affectiva) and connect that API to the data visualization and its elements, I was unable to shoot that visualization on a screen using live action. To overcome that challenge I created an animated video that did not rely on an external screen.
Delivery of Study

For Dinner Time Visualizations, I created a working demo and an animated video. The demo was created to show a working implementation of the visualization, while the animated video was used for testing.

The animated film tells the story of a couple having a conversation over dinner. The script of the film was specifically designed to show the range of emotions the system could communicate, such as anger and happiness. Topics covered in the animated film were chosen based on the findings of the Mechanical Turk study.

Frame from Dinner Time Visualizations’s animated concept video

Study Outcomes

User sentiment from testing Dinner Time Visualizations was slightly more positive than negative. While some users noted that "you should be able to understand your expressions without visual feedback", others saw it as "empowering" and "cute." I also learned that some users felt that Dinner Time Visualizations could "allow someone to adjust their behavior", but at the same time draw oneself "out of the conversation" and "potentially escalate a situation." Another insight distilled from user’s surveys and interviews is that:

  • Dinner Time Visualizations could scrutinize a conversation from the lens of an individual and couple.Since Dinner Time Visualizations feeds information back to an individual, it could differentiate the emotions that one partner expresses from the general mood of the conversation. This way a user would be able to recognize the effect of their shared emotion on the overall conversation.
Study Synthesis

While Dinner Time Visualizations proved to be challenging to implement, the concept enabled me to explore the different forms that feedback could take, while also investigating potential contexts for an experience within an intimate relationship. The following are principles I gathered when evaluating Dinner Time Visualizations.

  • Dinner Time Visualizations should employ a visual language that does not distract.The visual language of the current instantiation of Dinner Time Visualizations was hard to decipher and was often distracting. Simplifying the visual language could help a user gain more information from the visualization while also being less distracted.
  • Dinner Time Visualizations should allow for multiple forms of adjustment.Similar to Intimately and Curb, Dinner Time Visualizations should provide a user the opportunity to understand and change the specific forms of expressions they seek. Users should then be able to achieve a high degree of conversational symbiosis.
  • Dinner Time Visualizations should bring attention to both unfavorable and favorable behavior.Attention should not only be brought to the those emotions that could be described as negative; Dinner Time Visualizations should also emphasize those moments when couples are taking positive stances or are in complete sync with each other. This has the potential to yield greater appreciation and perceived relational value for oneself.
  • Dinner Time Visualizations should present directions for its use.Regardless of the visual language employed by Dinner Time Visualizations, users should be provided with a chance to understand what the different elements of the visualization represent to help the user take full advantage of Dinner Time Visualizations.
  • Dinner Time Visualizations should be well integrated into an environment.Dinner Time Visualizations should not draw an individual into a conversation, instead it should support that conversation and lead that partner to better understand the emotions they convey. Bringing the conversation to life as a large visualization behind one's partner might be seen as distracting, making exploration into different possible forms necessary.
  • Dinner Time Visualizations should provide a user the opportunity to question a specific interpretation.Somewhat similar to the other concepts, Dinner Time Visualizations should allow a user to validate when they are expressing a certain emotion. For instance, an individual's facial expressions could convey anger to others when they are not angry. Thus, a user should be able to build a baseline for how they express certain emotions that result in a beneficial experience for both partners.
Unfavorable Directions

Storyboards focused on Dinner Time Visualizations and potentially unfavorable directions of that experience concentrated on the following statements.

  • A couple that does not understand the Dinner Time Visualizations visual language.
  • A couple paying more attention to the visualization than their own conversation.
Considerations for the Future

One theme I found throughout these storyboards was that the algorithms behind the visualizations created by the artificial agent lacked visibility— both in terms of the visual language representation and the models they employ. To deal with this issue, the forms used to convey information in Dinner Time Visualizations, the ways in which a user can influence the models, and how the models are informed by various inputs (i.e., my prototype of Dinner Time Visualizations is only informed by facial images, if I continue working on this concept I hope to expand that to more than one form of input) warrants consideration.

An Iteration of Dinner Time Visualizations

While evaluating Dinner Time Visualizations, some research participants shared that both the visual language and form of Dinner Time Visualizations could draw an individual out of the conversation. A new iteration of Dinner Time Visualizations that addresses this comment would include a stripped down visual language (i.e., evolving gradients of colors) and a reduced presence in a room (i.e., the visualization takes up small picture frames not in the direct eye path of a partner that a partner could glance at from time to time).

An iteration of Dinner Time Visualizations with a stripped down visual language that takes up a small picture frame