Voice-first exploration with WearableAI

Try on TestFlight Watch walkthrough

WearableAI is a side project I built by myself to explore a question I keep coming back to: What does AI feel like when it's no longer trapped behind a keyboard or a screen?

It started as an experiment with the Meta DAT SDK and Ray-Ban Meta glasses, then expanded into a broader product built around a belief I've had for a while: AI and XR are converging into a new interaction layer.

Most AI products still assume the user is sitting in front of a screen, typing into a box, and giving the system full attention. WearableAI explores a different model - one that is voice-first, local-first, privacy-forward, and designed to fit into everyday workflows across wearables, phones, and cars.

TL;DR

Built WearableAI as a solo side project
Started with Ray-Ban Meta and the Meta DAT SDK, then expanded to iPhone and CarPlay
Supports OpenAI ChatGPT, Google Gemini, and xAI Grok through BYOK
Built around voice-first interaction, camera-aware context, and ambient assistance
Integrates with Apple tools like Calendar, Notes, Reminders, Shortcuts, Widgets, and App Intents
Stores data locally by default, with no server at all
Supports local LLMs in addition to cloud providers
Serves as my most direct product exploration of ambient AI in practice

The problem

Most AI products today assume:

a screen
a keyboard
a focused user
a single model ecosystem

Wearables, XR devices, and in-car systems break all of those assumptions.

In those environments, the right interaction model is different:

voice-first instead of text-first
low-friction instead of app-centric
context-aware instead of context-free
present when needed, invisible when not

That was the problem behind WearableAI.

I did not want to build "chat in another app." I wanted to explore what an ambient intelligence layer would look like if it respected the host device, fit around real workflows, and disappeared when it was not needed.

Why I built it

WearableAI combines many of the ideas I've been building toward for years:

context-aware interfaces
natural interaction models
AI systems that work around humans, not the other way around
the belief that wearables, XR, and mobility platforms will become major surfaces for intelligent software

CarPlay became an important part of that thesis for a simple reason: some platforms were beginning to introduce their own built-in assistants, but there were few good alternatives across the broader car ecosystem. Once Apple opened the ecosystem enough to support more serious CarPlay app experiences, it felt like a natural extension of the product.

I built WearableAI by myself because I wanted to test those ideas in product form, not just describe them.

The product

WearableAI lets users interact with LLMs of their choice through:

Ray-Ban Meta
iPhone
CarPlay

It is designed as an ambient, hands-free intelligence layer, not a destination app.

The current product supports:

OpenAI ChatGPT
Google Gemini
xAI Grok
local LLMs

Support is BYOK by design. Users bring their own model credentials, which keeps the system flexible, avoids lock-in, and makes it easier to compare models in real contexts.

This also matters at the product level. Different models are better at different things:

Grok is especially powerful because it combines internet access with X search, which makes it useful for live, socially driven queries
ChatGPT Realtime is strong for conversational interaction
Gemini works well across general tasks and multimodal contexts
local LLMs support privacy-sensitive or offline-oriented usage patterns

That model-agnostic approach is a core part of the product, not an extra feature.

Core product principles

Voice-first interaction

WearableAI assumes the user may be walking, driving, multitasking, or only partially attentive.

The system is built around:

hands-free input
fast commands and follow-ups
conversational continuity
short interactions that fit into movement and interruption

Camera-aware and multimodal context

WearableAI supports live multimodal context, including visual input where available.

That allows the system to move beyond pure text or pure voice and behave more like an assistant that can respond to the environment around the user.

Native integrations instead of app silos

The product integrates with Apple tools like:

Calendar
Notes
Reminders
Shortcuts
Widgets
App Intents

That means the system can help users capture, remember, launch, and act without forcing them into a separate workflow.

There is also a dedicated WearableAI reminders list so the system can be useful without interfering with a user's broader reminder setup.

Local-first and privacy-forward

Wearables and mobility devices are intimate surfaces. They demand trust.

WearableAI stores user data locally by default, including:

conversations
preferences
model configuration
state

At the moment, there is no server at all.

That was a deliberate design decision. Privacy is not a bonus feature for ambient systems. It is a core product constraint.

What makes this hard

Ambient AI is not just about putting a model on a new device.

It is about designing behavior.

The hard questions are:

when should the system respond?
how much context is enough?
how do you stay useful without becoming intrusive?
how do you integrate with real workflows instead of becoming one more app silo?
how do you respect the boundaries of glasses, phones, and cars without flattening them into the same interface?

That is where most of the product work lives.

The technical work matters, but the harder challenge is interaction design under real-world constraints:

partial attention
movement
interruptions
trust
low-UI environments
platform limitations

Why CarPlay matters

CarPlay is not just an extension tacked onto the product. It is part of the larger thesis.

Cars are one of the clearest examples of why current AI interaction patterns break down. A traditional chat interface is the wrong shape for a driving environment. The right interaction has to be glanceable, voice-first, low-friction, and deeply respectful of attention.

When platform support made it possible to build more serious CarPlay experiences, it felt like the right next surface for WearableAI.

The same principles that made sense for smart glasses also made sense in mobility:

minimal UI
high trust
fast entry and exit
AI that works with existing workflows instead of replacing them

The long-term extension of that thinking also points toward Android Auto, where the same interaction constraints exist even if the ecosystem is different.

Beyond current surfaces

Although WearableAI started with smart glasses, it is intentionally built around ideas that extend beyond a single device class.

The same design principles apply to:

XR headsets
Android Auto
future wearable and spatial devices
any surface where screens are secondary and attention is limited

The goal is not to build another destination app. It is to build an intelligence layer that adapts to the surface it runs on.

What I learned

Building WearableAI made one thing clearer: ambient AI only works when the interaction model is as carefully designed as the intelligence behind it.

A model can be powerful and still feel wrong if:

it interrupts at the wrong time
it needs too much setup
it ignores workflow context
it stores too much
or it turns simple tasks into AI theater

WearableAI is where I continue exploring those questions in product form.

It is less about "AI on glasses" and more about what happens when intelligent systems start behaving like part of the environment instead of a destination.

What this project is really testing

WearableAI is best understood as infrastructure disguised as an app.

It is a product, but it is also a testbed for:

ambient intelligence
voice-first UX
model orchestration
local-first AI systems
interaction patterns across wearables, XR, and mobility

That is the real reason I built it.

Designing a VR + LLM communication coach @ DePaul - where the HCI and interaction design thinking started