Skip to content

AI and Media Stack ​

A map of the AI backends, speech engines, and Live Assistant asset formats VAssist uses.

Language models ​

OptionWhere it appearsWhat it usesWhat you manage
Desktop LocalDesktopllama-server or node-llama-cpp, both on top of llama.cppGGUF models, inference engine and runtime backend, context and token limits, local model folder
Android LocalAndroidNative llama.cpp for GGUF and LiteRT-LM for .litertlm modelsGGUF or LiteRT-LM models stored in the app, compute unit, on-device limits
Chrome AIBrowser and extension when supportedBrowser-side Gemini Nano and related Chrome AI APIsBrowser flags, model download, output language, Chrome availability
OllamaAll platformsOllama serverEndpoint URL and model name
OpenAIAll platformsOpenAI hosted modelsAPI key and model choice

The optional Decision API is configured separately from chat generation. It can use a remote endpoint, an embed host bridge, or the desktop-local decision worker. It is off by default. Android can use a remote endpoint but does not include the native decision service.

GGUF and mmproj notes ​

  • GGUF is the local model format used by the llama.cpp-based desktop and Android paths.
  • Android also runs .litertlm models through LiteRT-LM.
  • Some multimodal GGUF models also need a matching mmproj file for image support.
  • That is why the local model controls in VAssist talk about model libraries, imports, downloads, and vision support instead of just asking for one model name.

Speech output ​

OptionWhere it appearsWhat it usesWhat you manage
KokoroBrowser, extension, desktopkokoro-js using the Kokoro-82M ONNX model, runs in a shared audio worker via onnxruntime-web so inference does not block the main threadVoice, speed, backend, cache, model initialization
Android Local TTSAndroidsherpa-onnx voice packs: VCTK, Kitten Nano, and SupertonicInstalled pack, speaker, speed
Desktop Local TTSDesktopGPT-SoVITS or Supertonic 3 hosted through the desktop runtimeEngine, install state, reference or built-in voice, language, speed, quality, sampling settings
GPT-SoVITS RemoteAll platformsRemote GPT-SoVITS serverServer URL, saved voices, language, speed, sampling settings
OpenAI TTSAll platformsOpenAI speech modelsAPI key, model, voice
OpenAI-compatible TTSAll platformsOpenAI-style speech serverEndpoint, optional key, model, voice

Speech input ​

OptionWhere it appearsWhat it usesWhat you manage
Android Local STTAndroidOn-device Whisper, SenseVoice, or Dolphin through sherpa-onnxLanguage, model, compute provider, optional Snapdragon QNN pack
Desktop Local STTDesktopfaster-whisper or whisper.cpp through the desktop runtimeEngine, runtime pack, Whisper model, language, threads, local endpoint
Chrome AI STTBrowser and extension when supportedBrowser-side Chrome AI speech recognitionOutput language, model download, required flags
OpenAI WhisperAll platformsOpenAI remote transcriptionAPI key and remote model
OpenAI-compatible STTAll platformsOpenAI-style remote transcription serverEndpoint, optional key, model, language

Voice activity detection ​

All platforms use @ricky0123/vad-web running the Silero VAD v5 ONNX model for voice activity detection.

  • VAD decides when the user has started and stopped speaking so VAssist knows when to begin and end recording without a push-to-talk button.
  • In voice conversation mode, VAD also watches for speech while the assistant is replying. If it detects confirmed human voice during playback, it automatically interrupts TTS and starts recording the new input.
  • VAD runs inside the shared audio worker alongside Kokoro so it does not block the main thread.

Live Assistant engine ​

The Live Assistant is rendered using BabylonJS (@babylonjs/core v8) with two additional layers:

LayerPackageWhat it adds
MMD model and animation supportbabylon-mmd v1PMX and BPMX model loading, VMD and BVMD motion playback, morph targets, physics bones
Physics@babylonjs/havokHavok physics engine for cloth, hair, and accessory simulation on supported models

Live Assistant formats and assets ​

VAssist uses MMD-style asset formats for the Live Assistant.

Asset typeWhat it is used for
PMX and BPMX modelsAvatar and stage geometry
PMX stagesBackground scenes for the Live Assistant
VMD and BVMD motionsAnimation clips used for idle, talking, thinking, and other motion categories
EmotesNamed combinations of audio, motion, and optional camera movement

What those file names mean in practice ​

  • PMX is the model format most users see when importing MMD-style avatars and stages.
  • BPMX is the binary packed model form used by some built-in resources and converted assets.
  • VMD is the common motion format for imported animation files.
  • BVMD is the binary packed motion form used by built-in and converted motion assets.

Speech-driven animation ​

  • The Live Assistant can use spoken output for lip sync.
  • VAssist also contains worker-side helpers for generating VMD motion data from audio and converting it into BVMD when needed.
  • That is why the speech stack and the Live Assistant stack are connected in the codebase even though they look like separate features in the UI.