Persona

A single microphone can independently drive Live2D mouths and the vowel shapes of VRM avatars. This capability functions entirely independently of face tracking: it works reliably without a webcam or mobile phone, and smoothly blends with face tracking whenever camera tracking is active.

Settings reside in the Lip Sync section of the Tracking panel, located directly beneath Webcam Tracking. These are global application settings preserved across scene switches, and the Tracking page in the Web Console configures the exact same settings. Audio processing runs entirely on the local machine: incoming audio is never played through output speakers or streamed over the Plugin API. Plugins receive only instantaneous volume levels and vowel recognition weights, never the raw audio stream.

SetupCopy Anchor Link

1. In the Lip Sync section, turn on Enable Lip Sync (disabled by default). On macOS, the operating system prompts for microphone access the first time this is enabled.
2. In the Microphone menu, select Default Microphone or choose a specific audio input hardware device.
3. Speak into the microphone and verify that the Input Level and the A, I, U, E, and O vowel meters respond dynamically; adjust Levels as needed.
4. In each avatar's layer inspector under Tracking, Microphone Lip Sync defaults to Always; once the global master switch is turned on, avatars in the scene immediately begin syncing their lip movements.

MicrophoneCopy Anchor Link

Default Microphone follows the system-wide default input device, or you can bind directly to a specific physical hardware input. The menu lists input hardware available on the computer running Persona (including in the Web Console, which never lists microphones from the client browser's device). Virtual audio devices (such as virtual cables or software soundboards) appear alongside standard hardware microphones. The status line below indicates the active input device currently being captured.

If the selected microphone becomes disconnected or unavailable, Persona does not switch to another input automatically: the dropdown displays Unavailable Microphone and the status switches to unavailable until you select another device or click Retry.

Persona does not offer audio monitoring pass-through to output speakers, nor does it capture system desktop audio output directly.

LevelsCopy Anchor Link

The noise gate evaluates signal levels after Input Boost has been applied. The dynamic range from the gate threshold up to full scale is mapped linearly to mouth opening from fully closed to fully open. If the mouth remains open in silence, decrease Input Boost or raise the Noise Gate threshold; if normal speech barely moves the mouth, adjust them inversely. Changes to all three parameters take effect immediately on active audio analysis; switching microphones or toggling lip sync reinitializes the audio capture pipeline.

Status and MetersCopy Anchor Link

The indicator beside Enable Lip Sync reports the current operational status:

When lip sync is active, the Input Level and five vowel indicator meters appear beneath the settings. These meters display similarity weights rather than probability distributions: ambiguous pronunciation can cause two adjacent meters to rise simultaneously, while non-vocal sounds cause all vowel meters to fall back while opening the mouth based on volume alone.

Audio capture halts and switches to unavailable in three scenarios: system microphone permissions were denied (grant Persona access in OS settings), the selected device is missing or disconnected, or the audio analyzer failed to start. Clicking Retry attempts to reinitialize capture on the selected device.

Calibrate VoiceCopy Anchor Link

While the microphone is actively capturing, the Voice Profile row displays the current profile status: showing Calibrated for ‹microphone› when a custom profile exists, or Default Profile otherwise. The Calibrate Voice… button is interactive only when the status reads listening.

The calibration wizard covers six recording targets (A, I, U, E, O, and Background Noise), each with spoken prompts:

1. At your normal speaking volume and distance from the microphone, click Record for an item and sustain the target vowel steadily for two seconds; for Background Noise, remain silent and breathe normally.
2. Recorded items show a check mark, and their button changes to Retry so you can re-record any item on its own. Vowel samples must exceed the Noise Gate threshold to register; background noise has no minimum level. If a take falls short, the dialog keeps the previously saved baseline and explains why, for example:
   • "That sound was too quiet. Speak above the noise gate and retry."
   • "Not enough audio was captured. Keep the sound steady and retry."
3. Once all six items are recorded, clicking Preview Profile applies the calibration draft immediately. Speak A, I, U, E, and O in sequence: the corresponding meter should lead distinctly, and avatars in the scene should match your vowel shapes. Re-record any confused vowel and preview again.
4. Click Save Profile to save the calibration. Clicking Cancel or closing the dialog discards the draft and restores the previously saved or bundled profile.

If the microphone already has a calibration profile, the dialog preloads existing data so individual vowels can be refined without repeating all six recordings. Disabling lip sync, switching microphones, hitting a capture error, or clicking Retry discards unsaved calibration drafts.

Profiles are saved per physical microphone hardware device and load automatically whenever that device begins capturing. The Reset Profile button appears only when a custom calibration exists for the active device; clicking it deletes the device-specific calibration and reverts to the bundled profile without affecting other microphones. When Default Microphone is selected, Persona attempts to resolve the underlying physical hardware; if identity cannot be determined, select the specific device before calibrating.

Prior to calibration, Persona applies a general baseline profile indicated as Default Profile. This generic profile may not match every vocal register or microphone frequency response. If vowel recognition accuracy is suboptimal, perform a personalized calibration. Importing external MotionSync profiles is not supported.

Microphone Lip SyncCopy Anchor Link

Each avatar can independently configure how it responds to microphone audio. Select an avatar layer, navigate to Tracking in its inspector, and locate Microphone Lip Sync, which consumes the microphone configured under Tracking:

In Always mode with active camera face tracking, Persona blends between both sources using the smoothed audio volume envelope: during silence, the mouth retains natural camera tracking details, including subtle twitches, silent yawns, jaw motion, and resting shapes; as speech volume rises, microphone vowels take precedence smoothly. When camera tracking drops, mouth control hands over entirely to the microphone without interference from held landmark poses.

Newly added avatars and those in existing scenes default to Always; however, because Enable Lip Sync is disabled globally by default, no audio driving occurs until the master switch is turned on. This synchronization mode belongs to the individual avatar instance and is saved within the scene; duplicate instances of the same model can use different modes, while Parameter Bindings remain bound to the underlying model asset.

Driving ModelsCopy Anchor Link

Live2DCopy Anchor Link

No additional sidecar files are required. Microphone values are applied on top of these standard mouth inputs, so a model's existing mouth bindings work out of the box:

• MouthOpen, MouthSmile, JawOpen
• ARKitJawOpen, ARKitMouthClose, ARKitMouthFunnel, ARKitMouthPucker

Mouth opening blends vowel shape and volume continuously, so consonants don't cause jarring jumps.

Parameters in the model's LipSync group (declared in .model3.json and listed in the Lip Sync row of Model Info) receive the volume directly unless an explicit binding rule drives them. Models with non-standard mouth parameter names therefore move without custom bindings. If a model has a separate mouth parameter for each vowel, map VoiceA…VoiceO to them under Parameters.

When audio bundled with Live2D motions or speech played through the Plugin API is playing, the playing audio stream takes precedence to drive the mouth, and microphone input yields automatically.

VRMCopy Anchor Link

Each vowel drives an expression on the avatar: A → aa, I → ih, U → ou, E → ee, O → oh. Each expression's weight is the volume multiplied by that vowel's similarity share. Avatars without vowel presets but with the standard ARKit jawOpen expression open the jaw by volume alone. How good the mouth looks depends on the avatar's own expression blendshapes.

When blending with live camera tracking, vowel presets and conflicting ARKit mouth expressions hand over control dynamically based on audio volume. Microphone expression weights are restored to neutral at the end of every render frame, preventing lingering artifacts in held camera tracking poses.

Voice InputsCopy Anchor Link

In the Mouth group of the Parameter Binding editor, Persona provides twelve dedicated voice inputs for both Live2D and VRM models:

The composite inputs VoiceVolumePlusMouthOpen and VoiceFrequencyPlusMouthSmile combine the voice with camera face tracking:

• The summed value can go beyond the standard limits; the binding's own input and output ranges clamp it.
• Without camera face tracking, they use only the audio component. Poses frozen during a tracking dropout are not added.
• When the microphone is inactive for an avatar (lip sync disabled, mode set to Off, or When Face Tracking Is Unavailable deferring to live face tracking), they keep only the camera component, so bindings keep following face tracking.

ImportingCopy Anchor Link

When importing .vtube.json configuration files, input parameter names are case-insensitive.

Persona employs a custom, optimized audio analysis pipeline; consequently, volume levels and vowel classification metrics for the same voice sample may exhibit slight differences compared to VTube Studio.

AutomationsCopy Anchor Link

In Automations, the Microphone Level trigger shares the same physical microphone, Input Boost, Noise Gate, and Smoothing configuration. It operates even when Enable Lip Sync is turned off, listening for audio thresholds without driving avatar mouths.

The Play Audio action can drive avatar lip sync directly from an audio file: assign its Drive Lip Sync target to an avatar (Live2D or VRM), and the avatar's mouth will follow the file during playback while microphone input yields. This analysis uses the bundled default voice profile, processes audio prior to volume adjustments or fades, and does not open the physical microphone. If multiple audio files drive the same avatar concurrently, the most recently started playback takes precedence; once playback finishes, microphone and face tracking smoothly resume control.