Sprout: a robot that sees, talks, and remembers you

Sprout is a small mecanum-wheeled robot with a personality. It sees with a camera. It listens and talks hands-free. It recognizes people and remembers them across sessions. It moves to match its words. The brain runs entirely on a Raspberry Pi 5. A laptop is only needed for development.

Demo

Watch the demo video on the GitHub repository.

The voice loop

There is no push-to-talk button. The loop is:

The microphone captures audio. Voice-activity detection finds speech.
Local speech-to-text (faster-whisper) transcribes it on the Pi.
The LLM generates a streamed reply.
Local text-to-speech (Piper) speaks it.

Sprout only acts when you address it by name, or inside a short active conversation window. Overheard speech is transcribed locally and then dropped. It is never sent to the cloud.

Faces and memory

Sprout detects faces with OpenCV YuNet and matches them with SFace embeddings. When it recognizes you, it loads your record into the prompt. When it meets an unknown face, it asks for a name and enrolls the person live.

Embeddings and names are stored locally in SQLite. They are never logged in plaintext. They never leave the device.

Two swappable brains

The LLM backend sits behind one interface. The default brain is a cloud model. The fallback is a local model through Ollama. Both stream. If the cloud is unreachable, Sprout degrades gracefully instead of going silent. The same pattern applies across the project: audio, vision, and the motor link each sit behind an interface, so any piece can be replaced without touching the core.

Three layers of motor safety

Anything that moves is treated carefully. The safety design has three independent layers:

Firmware. The Arduino has a deadman timeout. If no command arrives within the window, the motors stop. Malformed input stops the motors. An on-board sensor reflex refuses motion toward an obstacle or a floor edge. This works even if the Pi crashes.
Pi movement gate. Movement is locked by default. It must be explicitly permitted, and the current spot must be marked safe. A sensor veto overrides permission.
Process. First tests run with the wheels off the ground, at low speed.

The Pi never drives the motors directly. It commands the car over a fixed serial protocol, and the car runs its own firmware. The Pi and the car are on separate power domains.

Expressive movement

The LLM emits bracket cues inside its replies, like [SPIN] or [WIGGLE]. Each cue maps to a named motor routine. The mood of the reply also drives the RGB LED on the car. The model chooses the gesture. Deterministic code performs it.

Stack

Python 3.13 on a Raspberry Pi 5 (8 GB). Raspberry Pi Camera 3 for vision. A SunFounder Zeus Car body with an Arduino UNO, four mecanum wheels, ultrasonic and edge sensors. A pytest suite covers the hardware-free logic.

← All posts