Persona

A single microphone can independently drive Live2D mouths and the vowel shapes of VRM avatars. This capability functions entirely independently of face tracking: it works reliably without a webcam or mobile phone, and smoothly blends with face tracking whenever camera tracking is active.

Settings reside in the Lip Sync section of the Tracking panel, located directly beneath Webcam Tracking. These are global application settings preserved across scene switches, and the Tracking page in the Web Console configures the exact same settings. Audio processing occurs strictly on the local machine: incoming audio is never played through output speakers or streamed over the Plugin API—plugins receive instantaneous volume levels and vowel recognition weights, never the raw audio stream itself.

SetupCopy Anchor Link

1. In the Lip Sync section, turn on Enable Lip Sync (disabled by default). On macOS, the operating system prompts for microphone access the first time this is enabled.
2. In the Microphone menu, select Default Microphone or choose a specific audio input hardware device.
3. Speak into the microphone and verify that the Input Level and the A, I, U, E, and O vowel meters respond dynamically; adjust Levels as needed.
4. In each avatar's layer inspector under Tracking, Microphone Lip Sync defaults to Always; once the global master switch is turned on, avatars in the scene immediately begin syncing their lip movements.

MicrophoneCopy Anchor Link

Default Microphone follows the system-wide default input device, or you can bind directly to a specific physical hardware input. The menu lists input hardware available on the computer running Persona—in the Web Console as well, which never lists the microphone devices of the client browser machine. Virtual audio devices (such as virtual cables or software soundboards) appear alongside standard hardware microphones. The status line below indicates the active input device currently being captured.

If the selected microphone becomes disconnected or unavailable, Persona will never silently fall back to an alternate input: the dropdown displays Unavailable Microphone and the status switches to unavailable until you select another device or click Retry.

Persona does not offer audio monitoring pass-through to output speakers, nor does it capture system desktop audio output directly.

LevelsCopy Anchor Link

The noise gate evaluates signal levels after Input Boost has been applied. The dynamic range from the gate threshold up to full scale is mapped linearly to mouth opening from fully closed to fully open. If the mouth remains open in silence, decrease Input Boost or raise the Noise Gate threshold; if normal speech barely moves the mouth, adjust them inversely. Changes to all three parameters take effect immediately on active audio analysis; switching microphones or toggling lip sync reinitializes the audio capture pipeline.

Status and MetersCopy Anchor Link

The indicator beside Enable Lip Sync reports the current operational status:

When lip sync is active, the Input Level and five vowel indicator meters appear beneath the settings. These meters display similarity weights rather than probability distributions: ambiguous pronunciation can cause two adjacent meters to rise simultaneously, while non-vocal sounds cause all vowel meters to fall back while opening the mouth based on volume alone.

Audio capture halts and switches to unavailable in three scenarios: system microphone permissions were denied (grant Persona access in OS settings), the selected device is missing or disconnected, or the audio analyzer failed to start. Clicking Retry attempts to reinitialize capture on the selected device.

Calibrate VoiceCopy Anchor Link

While the microphone is actively capturing, the Voice Profile row displays the current profile status: showing Calibrated for ‹microphone› when a custom profile exists, or Default Profile otherwise. The Calibrate Voice… button is interactive only when the status reads listening.

The calibration wizard covers six recording targets—A, I, U, E, O, and Background Noise—each with spoken prompts:

1. At your normal speaking volume and distance from the microphone, click Record for an item and sustain the target vowel steadily for two seconds; for Background Noise, remain silent and breathe normally.
2. Successfully recorded items show a check mark and the button changes to Retry, allowing individual items to be re-recorded. Vowel samples must exceed the Noise Gate threshold to register; background noise has no minimum level requirement. Incomplete takes display feedback (e.g., "That sound was too quiet. Speak above the noise gate and retry." or "Not enough audio was captured. Keep the sound steady and retry."), preserving the previously saved baseline.
3. Once all six items are recorded, clicking Preview Profile applies the calibration draft immediately. Speak A, I, U, E, and O in sequence: the corresponding meter should lead distinctly, and avatars in the scene should match your vowel shapes. Re-record any confused vowel and preview again.
4. Click Save Profile to persist the calibration. Clicking Cancel or closing the dialog discards the draft and restores the previously saved or bundled profile.

If the microphone already possesses a calibration profile, the dialog preloads existing data so individual vowels can be refined without repeating all six recordings. Disabling lip sync, switching microphones, encounter capture errors, or clicking Retry will discard unsaved calibration drafts.

Profiles are saved per physical microphone hardware device and load automatically whenever that device begins capturing. The Reset Profile button appears only when a custom calibration exists for the active device; clicking it deletes the device-specific calibration and reverts to the bundled profile without affecting other microphones. When Default Microphone is selected, Persona attempts to resolve the underlying physical hardware; if identity cannot be determined, select the specific device before calibrating.

Prior to calibration, Persona applies a general baseline profile indicated as Default Profile. This generic profile may not match every vocal register or microphone frequency response. If vowel recognition accuracy is suboptimal, perform a personalized calibration. Importing external MotionSync profiles is not supported.

Microphone Lip SyncCopy Anchor Link

Each avatar can independently configure how it responds to microphone audio. Select an avatar layer, navigate to Tracking in its inspector, and locate Microphone Lip Sync, which consumes the microphone configured under Tracking:

In Always mode with active camera face tracking, Persona blends between both sources using the smoothed audio volume envelope: during silence, the mouth retains natural camera tracking details, including subtle twitches, silent yawns, jaw motion, and resting shapes; as speech volume rises, microphone vowels take precedence smoothly. When camera tracking drops, mouth control hands over entirely to the microphone without interference from held landmark poses.

Newly added avatars and those in existing scenes default to Always; however, because Enable Lip Sync is disabled globally by default, no audio driving occurs until the master switch is turned on. This synchronization mode belongs to the individual avatar instance and is saved within the scene; duplicate instances of the same model can use different modes, while Parameter Bindings remain bound to the underlying model asset.

Driving ModelsCopy Anchor Link

Live2DCopy Anchor Link

No additional sidecar files are required. Microphone analysis values are applied directly over standard mouth inputs—including MouthOpen, MouthSmile, JawOpen, plus ARKitJawOpen, ARKitMouthClose, ARKitMouthFunnel, and ARKitMouthPucker—allowing existing standard mouth bindings to function out of the box. Mouth opening blends vowel shape and volume continuously to prevent jarring transitions during consonant articulation.

Parameters declared in the model's .model3.json under the LipSync group (listed in the Lip Sync row of Model Info) receive volume directly whenever no explicit binding rule drives them, allowing models with non-standard mouth parameter names to move without custom bindings. For models authoring discrete per-vowel mouth parameters, map VoiceA…VoiceO under Parameters.

When audio bundled with Live2D motions or speech played through the Plugin API is playing, the playing audio stream takes precedence to drive the mouth, and microphone input yields automatically.

VRMCopy Anchor Link

A, I, U, E, and O map directly to the aa, ih, ou, ee, and oh Expressions, weighted by signal volume multiplied by the vowel's similarity share. For avatars lacking vowel presets but containing the standard ARKit jawOpen expression, the jaw opens based on volume alone. Visual fidelity depends on the quality of the avatar's authored expression blendshapes.

When blending with live camera tracking, vowel presets and conflicting ARKit mouth expressions hand over control dynamically based on audio volume. Microphone expression weights are restored to neutral at the end of every render frame, preventing lingering artifacts in held camera tracking poses.

Voice InputsCopy Anchor Link

In the Mouth group of the Parameter Binding editor, Persona provides twelve dedicated voice inputs for both Live2D and VRM models:

The composite inputs VoiceVolumePlusMouthOpen and VoiceFrequencyPlusMouthSmile can sum beyond standard limits, clamped by the binding's own input-output ranges. In the absence of camera face tracking, composites evaluate only their audio component, and frozen landmark poses from tracking dropouts are excluded. When microphone input is inactive for an avatar (lip sync disabled, mode set to Off, or When Face Tracking Is Unavailable deferring to live face tracking), composites retain only their camera tracking component, ensuring bindings continue responding naturally to visual tracking.

ImportingCopy Anchor Link

When importing .vtube.json configuration files, input parameter names are case-insensitive.

Persona employs a custom, optimized audio analysis pipeline; consequently, volume levels and vowel classification metrics for the same voice sample may exhibit slight differences compared to VTube Studio.

AutomationsCopy Anchor Link

In Automations, the Microphone Level trigger shares the same physical microphone, Input Boost, Noise Gate, and Smoothing configuration. It operates even when Enable Lip Sync is turned off, listening for audio thresholds without driving avatar mouths.

The Play Audio action can drive avatar lip sync directly from an audio file: assign its Drive Lip Sync target to an avatar (Live2D or VRM), and the avatar's mouth will follow the file during playback while microphone input yields. This analysis uses the bundled default voice profile, processes audio prior to volume adjustments or fades, and does not open the physical microphone. If multiple audio files drive the same avatar concurrently, the most recently started playback takes precedence; once playback finishes, microphone and face tracking smoothly resume control.