Back to Top

Social Agents in Homes

Social agents such as Alexa and Google Home are increasingly entering domestic environments. Close to 40 million social agents have been introduced to homes in the United States within in the last decade. While social agents have been studied in other contexts, little work exists on desired social interactions with social agents, and what people think agents should and shouldn’t be able to do in the home. Furthermore, these agents will be much more capable in the future. Consequently, there are many opportunities associated with the development of these devices, but they don’t come without concerns. We aim to explore the boundaries of these multi-user interactions with future social agents.

Social Agents mockup banner
Cover photo by Linus Rogge


UX Researcher


Spring Semester, 2019


Professors John Zimmerman and Jodi Forlizzi, Ph.D. Candidate Michal Luria, 3 Masters students, 3 undergraduate students


Speed Dating, Affinity Diagramming, Storyboarding, Interviews, Adobe XD


Understand the future role(s) of social agents in multi-user homes + understand how to design for social agents in the home.



This project is conducted through the Human-Computer Interaction Institute at Carnegie Mellon University.

Given current advancements in technology, AI will inevitably have significant potential to augment advanced social interactions with humans. Growing popularity in home agents warrants numerous questions these interactions in the home and what agents should and shouldn’t be able to do. Should household members be able to ask an agent where others are? Should an agent capable of sensing emotion notify other family members of odd behaviors? Who has permission to ask the agent to do various tasks?

These questions will only become more important to address as we design for the future of social agents in the home. We decided to focus on users consisting of families with two adults and at least one child 12 or older for group interviews.

How might we...
  1. Understand the future role(s) of social agents in multi-user homes.
  2. Understand how to design for social agents in the home.

Generating Themes

To begin our research, each team member individually brainstormed potential scenarios—futures—of social interactions in the home. We decided most scenarios should focus on multi-user scenarios involving multiple family members to capture these interactions. We used this to better understand themes before making formal scenarios and concretely defining our users.

some of my individual scenarios
Brainstorming scenarios individually
We used Affinity Diagramming to group scenarios into themes. Five umbrella themes emerged from this exercise
Affinity mapping scenarios
Affinity diagramming scenarios
scenario generation requirements
Formal scenario formatting
affinity diagramming in group
Formal scenario formatting
affinity map of scenarios to topics
Affinity map


Users & Scenarios

After identifying themes, we worked in small groups to identify potential scenarios. We generated a list of ideas (shown above) by theme and labeled them based on the context of their interactions (i.e. 2 parents, 1 child in a room), which we deemed important in ensuring good social multi-user interactions.

computer-like behavior
Computer-like behavior
Social DDD
Social DDD

We split them up amongst ourselves to create more detailed scenarios between 3-5 sentences that would eventually develop into storyboards for interviews. Over 1-2 weeks, we iteratively refined our scenarios and eliminated unnecessary scenarios—ones that either captured a similar interaction or did not provide enough value to the set. Two members of our team developed our personas.

Generating Storyboards

Narrowing down our ideas, we selected close to 30 scenarios to sketch out into storyboards. In our scenarios, it is important to capture the people involved, where it takes place, the activity, trigger, response, and resolution. These had to be reflected in our boards to help illustrate a story of an interaction not yet possible with today’s technology, but that which will be in the future.

storyboard 1

We critiqued each other’s storyboards, eliminated and revised, and then did the same through pilot testing on friends and family in mock interviews over several iterations.

storyboard 2
Example 1
storyboard 3
Example 2
storyboard 4
Example 3
storyboard 5
Example 4
storyboard 6
Example 5
storyboard 7
Example 6

We noticed a major need to showcase more positive storyboards that got in the interactions. When we presented randomized sets in mock interviews that weighed the negative interviews more, subjects would focus on the “how” or the result of the interaction instead of the interaction itself. Eventually, we narrowed our storyboards down to 23.


Interviews & Speed Dating

Speed Dating in research consists of rapid comparisons between design opportunities and speculative futures by presenting low-fidelity concepts in a relatively structured and fast approach, similar to romantic speed dating. Providing participants with exposure to numerous concepts allows for better explanation into what they like and dislike across a wide set of futures and help us better understand user needs in an ambiguous space as a result.

Each interview lasted 1.5 hours, primarily at families’ homes in Pittsburgh, while some were on-campus. In each scenario, 2-3 researchers would conduct interviews to help guide participants through their thoughts and feelings. Scenarios were presented randomly using a Latin Square to help control for bias. Over the course of the interviews, we removed four and added two scenarios to reflect our feedback on whether scenarios provided value or if we were missing a theme.

I participated in running six out of 18 interviews.


We transcribed each interview and marked important quotes from participants. We cut out these quotes to include in another affinity diagramming to identify themes from the qualitative data. From here, we referenced and related themes to derive groupings and narratives shared between multiple families.

affinity diagramming interview quotes
Affinity diagramming quotes


While the project is still ongoing until submission to CHI in Fall 2019, we’ve learned a lot about what families want from social agents in the home, and have more questions to follow. Deriving these insights required looking past scenario-specific cases and pulling out abstract ideas and themes from the quotes. Some of our findings are listed below.

  1. Families desire social agents to understand house rules
    Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.
  2. Many families believe an agent could interfere with their children growing up
    Parents think an agent’s surveillance would inhibit children from learning valuable lessons on their own. Many parents expressed the desire for their children to have room to experiment and make mistakes, and are worried children may additionally become reliant on agents in the home.
  3. Many families wouldn't trust the agent to be proactive in making decisions for them
    Most families want the agent to confirm actions before doing them. People don’t trust the agent would be designed with enough transparency enough to be entirely trustworthy.
  4. Many families think the advanced social agent would remove some autonomy from household members
    The level of surveillance granted to agents will make more things “provable” when family members can be verified or monitored. This is concerning to multiple families. Many see it as an “extra set of eyes."
  5. Many families don’t believe social agents will understand the intimate relationships between family members
    Families think there are too many nuances in the relationships between family members for an agent to ever really understand those connections and act on them. If the agent could, there are mixed reactions on how it should be used to resolve relationship conflicts.

Related research will continue to explore similar ideas in the realm of social agents moving forward next fall.

You can find the research paper here when it is published in September 2019.