EmpathAR
Ever wanted to read the room? EmpathAR uses your camera to detect posture, expression, and movement in real time, translating them into a live Social Battery overlay for everyone in frame.
Presented at:
Team
Vanessa Cuevas — Product Designer
Yogiti S.
Kriti Shukla
Tools & Materials Used
Software / AI / Data
MediaPipe (pose landmarker + face landmarker, on-device ML)
JavaScript / WebGL / WebAssembly
Vercel (deployment)
Hardware / Sensors
Device camera (laptop or mobile)
No additional hardware required — runs entirely in the browser
Media / Output
Real-time AR web experience
Canvas-based overlay rendering
Other Tools or Materials
ChatGPT (hero image generation)
Claude Code (development assistant)
Project Summary
Project Category: Interactive installation
Most of social intelligence is invisible. We read signals unconsciously: crossed arms, a fading smile, restless movement.
EmpathAR makes those signals visible in real time, rendering a live Social Battery overlay directly over the people in your camera view. It doesn’t tell you what to feel. It tells you what’s already happening and helps you work through it in real time.
Vision
How does our project express a perspective on The New Human?
EmpathAR uses on-device AI to read posture, expression, and movement — translating the unconscious signals we’ve always exchanged into a visible layer of social intelligence. It asks not how AI can think for us, but how it can help us feel more deeply seen: turning invisible emotional states into shared awareness, and creating moments of genuine attunement between people in the same space.
Exploration
What are you experimenting with or discovering through this project?
Our team wanted to create an augmented reality that visually displays emotional body language so we can better understand each other’s needs.
Our goal was to try to understand how much of human connection can be inferred from body language alone, and whether surfacing that inference in real time changes how people relate to each other.
Art and Technology
How do art and technology interact in your project?
EmpathAR uses computer vision as a creative medium. The underlying technology — pose detection, face landmarking, real-time inference — is clinical by design. The art is in the translation: choosing which signals matter, what to name them, and how to render them in a way that creates empathy rather than surveillance.”
Project Demo / How to Experience It
Visit the live web experience at empath-ar.vercel.app Point your camera at one or more people and the system begins reading in real time.
What to look for: Each person in frame gets a persistent Social Battery label (Energized → Engaged → Present → Fading → Needs Space) driven by posture, expression, and movement. The overlay updates continuously and surfaces contextual tips for how to respond to each state.
Desktop: Designed for one person or small group scenario: mount a laptop facing the crowd or the front row at a festival gate or get a live read of the room’s collective energy. Works equally well in interviews and meetings, where it surfaces how your own expression and posture might be landing in real time.
Mobile: Built for 1:1 or small group contexts: hold your phone naturally and get an immediate social read of the person or people in front of you.
Challenges encountered:
2 out of 3 Team members left early
Frown detection proved surprisingly difficult. MediaPipe’s blendshapes fire inconsistently across lighting conditions and face shapes, so we rebuilt the detection using raw geometric landmark measurements.
Reducing jitter across pose, expression, and battery state required multiple layers of EMA smoothing (per-landmark, per-score, and per-position) before the overlay felt stable enough to be useful rather than distracting.
What we didn’t have time to implement:
Audio detection and voice-to-text transcription for context-aware tips based on what’s actually being said in the room (on the roadmap as v1.3).
Testing and hardware:
Currently testing on Oculus Quest 3 and Quest 2.
Actively looking to connect with anyone who is willing to provide Meta smart glasses or Apple Vision Pro to explore what a glasses-native version of EmpathAR could look and feel like — if that’s you, please reach out!
Open questions for feedback:
Where is the line between social awareness and surveillance
How should the UI reflect that tension?
What emotional states matter most in your context: performance, meeting, conversation, crowd?
Who else should this exist for? Accessibility wise, we’d like to support access social learning and inclusion for all.
What’s Next?
Near-future vision: The natural home for EmpathAR is Meta smart glasses. We are currently working to test in Oculus Quest 3 and Quest 2
Our dream, in the solarpunk world of 2036, is to have these integrated into AR contact lenses running EmpathAR as one layer of a broader social awareness stack: a world where emotional attunement isn’t a skill gap, but a shared infrastructure.
*see our full roadmap for future enhancements