Skip to content

Technical Overview ​

This section serves two different jobs:

  • integration work for teams embedding VAssist into their own host app
  • contributor work for people changing the VAssist runtime itself

If you are looking for user-facing help, start at Start Here or Using VAssist.

If You Are Embedding VAssist ​

Start with the page that answers the next decision you actually need to make.

QuestionRead this page
Which package and entry point should I use?Package Integration
How do I mount and control a non-React instance?@vassist/embed
How do I use the React wrapper and customizations?@vassist/react
Which config field controls the thing I need?Configuration Reference

If You Are Changing VAssist Itself ​

QuestionRead this page
Which runtime owns this feature?This page
Where does the code live?Repository Map
Which engines and model formats are in play?AI and Media Stack
What data is persisted or backed up?Storage and Data
How big is the repo and where are the generated trees?Repository Tree and Metrics

Runtime Model ​

VAssist is one shared application runtime with several outer shells around it.

LayerMain locationWhat it owns
Shared application runtimesrc/Shared UI, stores, services, hooks, config, Babylon integration, and runtime-aware proxies
Browser package surfacespackages/Public browser APIs for plain JS hosts and React hosts
Embed bootstrappingembed/ and src/embed/Custom element definition, host APIs, config normalization, settings policy, branding, portal routing
Extension shellextension/Content/background/offscreen runtime pieces and browser-specific bridge flow
Desktop shellelectron/Electron process wiring, local service management, IPC, packaging
Android shellandroid/ and android-src/Native Android runtime, wallpaper mode, Android-local AI services, Android-facing web entry

Request And Control Paths ​

The same UI can route work through very different execution paths.

SituationPath
Browser host with builtin providersUI -> service proxies -> browser-side provider/runtime code
Browser host with secure bridgeUI -> service proxies -> transport.bridge -> host/backend
ExtensionUI/content -> background and offscreen contexts -> provider/runtime code
Desktop local modeUI -> service proxies -> Electron IPC -> local service/runtime
Android local modeUI -> bridge/store -> Android runtime service

This is why src/services/proxies/ matters: it is the routing layer that keeps one UI working across all shells.

Package Boundary ​

The browser packages are thin public wrappers over the shared runtime.

  • packages/embed/src/ exposes imperative mount, update, remove, and type re-exports.
  • packages/react/src/ wraps the same runtime in React components and adds host-scoped React customizations.
  • src/embed/config.ts is the central public config and type contract used by both packages.

If the docs and the exports ever disagree, packages/embed/src/index.ts, packages/react/src/index.tsx, and src/embed/config.ts are the files to trust first.

Key Libraries ​

AreaLibraryWhy it is here
Shared UIReact 19Common UI layer across shells
3D rendering@babylonjs/coreLive Assistant rendering
MMD model supportbabylon-mmdPMX/VMD loading and playback
Physics@babylonjs/havokCloth and hair simulation
Browser VAD@ricky0123/vad-webVoice activity detection
Browser TTSkokoro-jsONNX-backed speech synthesis in browser contexts
Browser inference runtimeonnxruntime-webONNX execution for supported browser features
Desktop local LLMllama-serverDefault GGUF model execution on desktop
PersistencedexieIndexedDB-backed storage layer
Android bridgeCapacitorWeb-to-Android runtime bridge

Where To Look Next In Code ​

  • src/embed/ for host-facing config, branding, portal containers, and settings policy
  • src/services/proxies/ for runtime routing logic
  • src/components/ for shared product UI
  • src/babylon/stages/ for built-in and imported stage environments, lighting, shadows, seasons, and parallax
  • src/babylon/ar/ for the Android AR session and placement flow
  • packages/ for the public browser API boundary
  • electron/, extension/, and android/ for shell-specific code