Project 03 / 10 · Vision

PerceptAI

Real-time multimodal perception app combining vision models with a live web interface — see what the model sees, instantly.

01 / Context

The problem

PerceptAI brings multiple perception signals into one live interface, turning a camera stream into interpretable face, gaze, pose, hand, object, and session-level observations.

02 / System workflow

From input to output

  1. 01Camera / image
  2. 02Vision analyzers
  3. 03Identity + event memory
  4. 04WebSocket stream
  5. 05Dashboard + report

03 / Architecture

Inside the system

A1

OpenCV coordinates the frame pipeline while MediaPipe supplies face, hand, and body landmarks and YOLO-World detects more than 60 object classes.

A2

DeepFace Facenet512 embeddings support identity search; ChromaDB stores faces across sessions and DBSCAN groups related embeddings.

A3

FastAPI streams annotated results over WebSocket, SQLite records detections, and Ollama adds session reports with a rule-based fallback.

04 / Engineering decisions

What shaped the build

Decision 01

Multimodal composition

Specialized analyzers handle face mesh, emotion, blink and gaze, head pose, hand landmarks, posture, and scene objects in the same session.

Decision 02

Local-first reporting

Ollama is optional; when it is unavailable the system still returns deterministic rule-based reports.

Decision 03

Inspectable interface

REST endpoints expose image analysis, identity registration, search, reports, clustering, history, and aggregate statistics.

05 / Evidence

Measured, not implied

468

Face landmarks

Dense mesh used for facial geometry and expression signals.

33

Body keypoints

Pose skeleton used for posture interpretation.

60+

Object classes

Open-vocabulary scene detection through YOLO-World.

06 / Delivery

What shipped

  • 01The browser dashboard receives live inference through FastAPI and Uvicorn.
  • 02Docker packages the CPU-oriented stack for Hugging Face Spaces.
  • 03SQLite logging and video recording preserve session evidence beyond the live overlay.

Constraint documented

DeepFace models add a sizable first-run download, and AI commentary depends on local Ollama availability; deterministic analysis remains available without it.