Social Agents

Social Agents in Homes

Research on the behaviors of intelligent social agents in homes.

Type / UX Research

Duration / Spring 2019

Role / Researcher

Tools / Storyboarding • User Interviews • UX • UI


This project was a formal study conducted through the Human-Computer Interaction Institute at Carnegie Mellon University.

Given current advancements in technology, AI will inevitably have significant potential to augment advanced social interactions with humans. Growing popularity in home agents warrants numerous questions these interactions in the home and what agents should and shouldn’t be able to do in the home. Should household members be able to ask an agent where others are? Should an agent capable of sensing emotion notify other family members of odd behaviors? Who has permission to ask the agent to do various tasks?

These questions will only become more important to address as we design for the future of social agents in the home. We decided to focus on users consisting of families with two adults and at least one child 12 or older for group interviews.

Google Home device on table

Image Credit: Linus Rogge


1. Understand the future role(s) of social agents in multi-user homes.

2. Understand how to design for social agents in the home.

Generating Themes

To begin our research, each team member individually brainstormed potential scenarios--futures--of social interactions in the home. We decided most scenarios should focus on multi-user scenarios involving multiple family members to capture these interactions. We used this to better understand themes before making formal scenarios and concretely defining our users.

Individually brainstorming scenarios

Affinity Diagramming

We used Affinity Diagramming to group scenarios into themes. Five umbrella themes emerged from this exercise


An agent performs actions without being asked by a user


An agent controls whether an action can be done or access given


An agent deceives a user upon request

Social Dull, Dirty, and Dangerous work (Social DDD)

An agent takes a larger role in the social interactions

Computer-like Behavior

An agent performs algorithmically or computationally, as asked by today’s devices

Affinity diagramming scenarios

Criteria for generating scenarios

Scenario criteria

Sorting potential scenarios

Scenario sorting

Affinity map of scenarios into themes

Affinity map

Users & Scenarios

After identifying themes, we worked in small groups to identify potential scenarios. We generated a list of ideas (shown above) by theme and labeled them based on the context of their interactions (i.e. 2 parents, 1 child in a room), which we deemed important in ensuring good social multi-user interactions.

We split them up amongst ourselves to create more detailed scenarios between 3-5 sentences that would eventually develop into storyboards for interviews. Over 1-2 weeks, we iteratively refined our scenarios and eliminated unnecessary scenarios—ones that either captured a similar interaction or did not provide enough value to the set. Two members of our team developed our personas.

Criteria for generating scenarios

Computer-like behavior

Sorting potential scenarios


Affinity map of scenarios into themes

Social DDD

Generating Storyboards

Narrowing down our ideas, we selected close to 30 scenarios to sketch out into storyboards. In our scenarios, it is important to capture the people involved, where it takes place, the activity, trigger, response, and resolution. These had to be reflected in our boards to help illustrate a story of an interaction not yet possible with today’s technology, but that which will be in the future.

storyboard example 1


We critiqued each other’s storyboards, eliminated and revised, and then did the same through pilot testing on friends and family in mock interviews over several iterations.

storyboard 1

Example 1

storyboard 2

Example 2

storyboard 3

Example 3

storyboard 4

Example 4

storyboard 5

Example 5

storyboard 6

Example 6

We noticed a major need to showcase more positive storyboards that got in the interactions. When we presented randomized sets in mock interviews that weighed the negative interviews more, subjects would focus on the “how” or the result of the interaction instead of the interaction itself. Eventually, we narrowed our storyboards down to 23.

Interviews & Speed Dating

Speed Dating in research consists of rapid comparisons between design opportunities and speculative futures by presenting low-fidelity concepts in a relatively structured and fast approach, similar to romantic speed dating. Providing participants with exposure to numerous concepts allows for better explanation into what they like and dislike across a wide set of futures and help us better understand user needs in an ambiguous space as a result.

Each interview lasted 1.5 hours, primarily at families’ homes in Pittsburgh, while some were on-campus. In each scenario, 2-3 researchers would conduct interviews to help guide participants through their thoughts and feelings. Scenarios were presented randomly using a Latin Square to help control for bias. Over the course of the interviews, we removed four and added two scenarios to reflect our feedback on whether scenarios provided value or if we were missing a theme.

I participated in running six out of 18 interviews.


We transcribed each interview and marked important quotes from participants. We cut out these quotes to include in another affinity diagramming to identify themes from the qualitative data. From here, we referenced and related themes to derive groupings and narratives shared between multiple families.

Affinity diagramming quotes


Deriving meaningful insights required looking past scenario-specific cases and pulling out ideas and themes that emerged from direct quotes. Some findings became apparent in our analysis, many of which call for further exploration. A few of these findings are listed below.


Families desire social agents to understand house rules

Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.


Many families wouldn't trust the agent to be proactive in making decisions for them

Most families want the agent to confirm actions before doing them. People don’t trust the agent would be designed with enough transparency enough to be entirely trustworthy.


Many families believe an agent could interfere with their children growing up

Parents think an agent’s surveillance would inhibit children from learning valuable lessons on their own. Many parents expressed the desire for their children to have room to experiment and make mistakes, and are worried children may additionally become reliant on agents in the home.


Many families think the advanced social agent would remove some autonomy from household members

Families tend to think the agent should be calibrated to individual households. Most families think parents should have primary control of the agent, but this hierarchy was often debated.


Many families don’t believe social agents will understand the intimate relationships between family members

The level of surveillance granted to agents will make more things “provable” when family members can be verified or monitored. This is concerning to multiple families. Many see it as an “extra set of eyes."

The paper for this research was published in CHI 2020.

View Research Paper