Eidolon

Intro

LAPLACE Eidolon is a voice double of a streamer's persona — visitors walk into the LAPLACE flower shop through a browser and talk in real time with an AI that speaks and remembers

Technically it is a cascaded voice bot built on Pipecat, chaining STT, LLM, and TTS, with optional multi-layer RAG and per-viewer long-term memory on top, so the double can "remember" past stream segments, persona settings, and its private exchanges with each visitor

Live Demo

The AI chat page on laplace.live runs on this very project. Open it to talk to the double:

Pipeline Structure

transport.input → STT → user_aggregator
    → Mem0 (per-viewer long-term memory, optional)
    → StreamArchive (RAG over past streams, optional)
    → PersonaFacts (RAG over persona facts, optional)
    → LLM → TTS → transport.output → assistant_aggregator

Tech Stack

• STT: Soniox (stt-rt-v5)
• LLM: OpenRouter (~google/gemini-flash-latest by default)
• TTS: Fish Audio (s2.1-pro)
• Transport: SmallWebRTC (Pipecat self-hosted)
• Vector store: LanceDB (local)
• Long-term memory: self-hosted Mem0 over Qdrant (optional)

Only self-hosted SmallWebRTC is supported. It relies on your own reverse proxy to terminate TLS, and does not integrate with Daily or Pipecat Cloud

Four Layers of Persona Memory

The bot feeds the LLM context from four levels of persistent storage. Every layer can be turned on or off independently, and the bot runs stably on a minimal configuration with only persona.md enabled:

1. persona.md: always-on identity, tone, and hard rules, loaded straight into the system prompt at startup
2. Persona facts: hand-curated atomic facts of "things she has said", retrieved by semantic relevance (the LanceDB index is built by laplace-ingest-facts)
3. Stream archive: past stream segments refined a second time by an LLM, retrieved by semantic relevance (the LanceDB index is built by laplace-ingest-streams), with a broadcast catalog layered on top for whole-stream overviews
4. Mem0: per-viewer episodic memory extracted automatically from conversation, isolated by Bilibili UID; vectors live in Qdrant and the engine runs embedded in the bot process

Hybrid Retrieval and Reranking

The Persona facts and Stream archive RAG layers share one hybrid retrieval pipeline, run in real time on every conversation turn:

1. Query construction: combines the last few user turns so that clipped utterances (such as "Hmm, what about that?") still retrieve with context
2. Single embedding pass: embeds the query once with EMBED_MODEL (Gemini by default, 3072 dimensions)
3. Date prefilter (persona facts only): when the query mentions a date (2026年5月9日 / 5月9号), the candidate set is narrowed by the date column first
4. Hybrid search: vector retrieval layered with jieba-tokenized Chinese BM25/FTS, pulling PERSONA_FACTS_CANDIDATES (30 by default) or STREAM_ARCHIVE_CANDIDATES (15 by default) candidates, with stop words stripped so BM25 scores do not collapse
5. Cross-encoder reranking: cohere/rerank-4-fast runs through OpenRouter by default, on the same OPENROUTER_API_KEY the LLM uses, narrowing the candidates down to top_k (8 for facts, 3 for chunks); set RERANK_ENABLED=false and retrieval falls back to LanceDB's RRF fusion

Stream Recall and the Broadcast Catalog

Beyond retrieving roughly five-minute stream segments by semantic relevance, the stream archive keeps a broadcast catalog — stream_catalog.json, built locally from refined transcripts by laplace-build-stream-catalog, with no model or embedding calls. The catalog groups Bilibili recording parts by date and BV ID, and keeps per-broadcast activity counts, representative excerpts, and every scene summary

That layer lets the double tell a whole-broadcast question apart from a topical one:

• Recaps ("what did you stream last week") read the matching catalog records directly, skipping embedding, reranking, and persona-fact retrieval
• Topical questions scoped to a period go through the same hybrid retrieval with a metadata prefilter
• Anything whose time reference needs the model's judgment goes to the recall_streams tool, which accepts periods such as latest, last_week, past_days, and date_range, or catalog IDs already supplied in context — at most twice per user turn

Dates and periods resolve against PERSONA_TIMEZONE: "last week" means the previous Monday through Sunday, and "the past seven days" includes today. A missing or unreadable catalog fails open, and segment retrieval keeps working as before

Self-Hosting

Only one command is needed:

docker compose up --build

The image bundles the Pipecat runtime and prebuilt LanceDB indexes, and listens on port 7860. The POST /api/offer endpoint handles SmallWebRTC SDP signaling, so a frontend can connect to it directly, or through @pipecat-ai/client-js and @pipecat-ai/small-webrtc-transport

A production deployment must terminate TLS at the reverse proxy, and configure TURN as needed to punch through symmetric NAT

With TURN configured, the server gathers relay candidates only: the origin's public IP never reaches a viewer's SDP, and every NAT type connects on the first candidate, at the cost of relaying all media and making TURN a hard dependency. STUN-only deployments keep the old behavior

Viewer Identity and Rate Limiting

Optional loginSyncToken verification runs through the LAPLACE Login Sync worker: the Bilibili UID it resolves becomes the Mem0 namespace and skips anonymous rate limiting; visitors that carry no token, or whose token fails verification, fall into a per-IP sliding-window quota of 5 connection attempts per hour by default

Signed-in visitors can also manage their own Mem0 memories — GET /api/memories lists them, POST /api/memories/forget clears them. Both authenticate with the same loginSyncToken carried in the WebRTC handshake, and are strictly scoped to the caller's own Bilibili UID

Runtime Voice Switching

The double can switch Fish Audio voices mid-call: when a visitor says a cue such as "be gentler" or "be more upbeat", the change_voice_style tool swaps in the matching reference_id in real time. Set FISH_VOICE_ID_GENTLE and FISH_VOICE_ID_LIVELY to enable it; leave both empty to turn the capability off

The frontend can also pick the Opus encoding bitrate per session by audio quality (EIDOLON_OPUS_BITRATE_*, 96 kbps by default), trading off weak networks against audio fidelity

Public MCP Service

The bot additionally exposes a read-only Model Context Protocol endpoint at /mcp/ (Streamable HTTP), opening the same two RAG layers the voice pipeline uses — search_persona_facts and search_streams — to any MCP client. The public instance sits at https://eidolon.vrp.moe/mcp and is anonymous by default. See MCP Service for details

Offline Tools

Index building, bulk downloading, and transcription each ship as a standalone console script, all invoked through uv run:

Source Code