Overview
We tend to think of digital and face-to-face communication as separate worlds. But watch any group of friends hanging out and you’ll see something more complicated: people glancing at their phones mid-conversation, sharing memes, scrolling while listening, toggling between the person in front of them and someone texting from across town.
I wanted to understand what’s actually happening in these moments. Not in a lab, but in the wild.
The Research Question
How do smartphones shape face-to-face conversation? Specifically:
- When do people check their phones during in-person interaction?
- How does digital content (like an Instagram post) become part of a face-to-face discussion?
- How do people manage attention between physical and digital conversations happening at the same time?
Methods
I spent 14 months conducting ethnographic fieldwork in Mostar, Bosnia and Herzegovina, embedded in youth social spaces. To capture the interplay between digital and physical interaction, I developed a novel method in linguistic anthropology: recording face-to-face conversations while simultaneously capturing screen recordings from participants’ phones.
Data Collection:
- Video and audio recording of face-to-face interactions
- Synchronized screen recordings from participants’ smartphones
- Ethnographic observation and interviews
Analysis:
- Synchronized all data streams in Adobe Premiere
- Conducted conversation analysis and thematic coding in ELAN
- Mapped participation shifts between digital and physical realms
Key Findings
Finding 1: Phones create layered participation
People don’t simply “check out” when they look at their phones. They strategically manage attention across both digital and physical conversations simultaneously. A participant might be listening to a friend while also scrolling Instagram and both activities inform what they say next.
Finding 2: Digital content enters face-to-face talk
In one recorded session, a participant discovered a friend’s engagement announcement on Instagram. What happened next was striking: the post became a 10-minute group discussion. The phone was passed around, the couple’s relationship timeline was researched in real-time, and the group debated whether to leave a comment and what it should say.
Finding 3: The phone holder shapes the conversation
Whoever holds the phone often controls the conversational direction. They introduce new topics, show content to others, and steer the group’s focus. Even when someone else is speaking, holding the phone gives a participant power to redirect attention.
Framework: Screen-in-Talk Approach
I developed a “screen-in-talk” approach that extends classic interaction analysis to include smartphones as participants in conversation. The phone isn’t just a distraction—it connects people to absent others, archives, and platforms in ways that reshape who’s part of the conversation and how.
Most research treats phone use as either solo activity or interruption. Screen-in-talk captures the moments when digital and face-to-face interaction overlap. In other words, when checking your phone becomes part of the conversation, not a departure from it.
So What? Implications for Design
For product teams building social or communication tools:
- “Phone use” isn’t just solo screen time but is deeply embedded in social moments
- Features like sharing, notifications, and comments shape real-world conversations in ways that aren’t visible in traditional usability testing
- Understanding hybrid digital-physical behavior requires research methods that capture both
For researchers:
- Screen recordings + video observation together yield richer data than either alone
- We need methods that capture the hybrid and multimodal nature of modern communication, not just the digital or the physical in isolation
Publication
“Between Screens and Speech: Screen-in-Talk as a Framework for Digitally Mediated Face-to-Face Interaction”
In preparation for the Journal of Linguistic Anthropology; working-paper copy upon request.
Want to talk about this research?
I’m currently looking for UX Research and Research Scientist roles where I can apply this approach to understanding how people actually use technology. Get in touch: duric@umich.edu
Dejan Duric 