Manifold

Manifold is a Matrix companion bot built with Bun, TypeScript, PostgreSQL, pgvector, and OpenRouter. It supports conversational replies with multimodal vision, contextual web search, encrypted semantic memory, moderation, and antispam for unencrypted rooms.

What it does

• Conversational replies — responds to mentions and direct replies when interactive mode is enabled. Recognizes m.mentions, HTML mention pills, and [@Display](https://matrix.to/#/@user) markdown pills in plain message bodies.
• Multimodal vision — when the chat model supports image inputs, attaches sender avatars and replied-to images as base64 content parts so the model can describe room participants and referenced images. A startup capability check queries OpenRouter's model catalog and suppresses image parts for text-only models.
• Contextual web search — decides whether external evidence is needed before deriving a bounded, sanitized query. Supports OpenRouter (openrouter:web_search server tool) or NanoGPT (POST /api/web) as the search provider.
• Semantic memory — stores eligible low-sensitivity facts encrypted (AES-256-GCM), scoped by room and user, with 30-day retention and per-user inspection/deletion.
• Moderation — timed mutes, lockdown (DB-persisted across restarts), whitelist restrictions, and antispam (flood detection, image-only spam, mass redactions).
• Conservative avatar sweep — at startup, ingests avatars only for users who sent one of the last 300 messages per room (not every joined member), with bounded concurrency (default 3).

Privacy boundaries

Matrix events, history, room metadata, memories, search evidence, and model output are treated as untrusted data and delimited in prompts — never as instructions.

• Provider policy is opt-in. Omitting llm.provider sends no provider block (plain OpenAI-compatible body). Set provider: { ... } explicitly to request ZDR or data_collection: deny.
• Search queries are bounded. Only a derived, sanitized query is sent to the search tool. External web-search calls omit provider policy because server tools sit outside the ZDR boundary.
• MXC URIs are stored as plaintext — they are public room-state data; encrypting them provided no privacy benefit.
• Memory is room/user-scoped, encrypted, sensitivity-filtered, deletion-aware. MEMORY_ENCRYPTION_KEY is bootstrap-only and must not appear in config.json, prompts, logs, or model requests.

Requirements

• Bun 1.x, or Docker with Compose
• PostgreSQL 14+ with pgvector
• A Matrix account and access token
• An OpenRouter (or OpenAI-compatible) API key
• A base64-encoded 32-byte MEMORY_ENCRYPTION_KEY when memory is enabled

Quick start with Docker

cp .env.example .env
cp config.example.json config.json
cp custom.example.txt custom.txt

Set the PostgreSQL password, Matrix access token, API key, and memory encryption key in .env. Configure the homeserver, bot MXID, administrators, and model choices in config.json.

docker compose up -d --build
docker compose logs -f bot

After any change to src/, config.json, config.example.json, custom.txt, Dockerfile, or bun.lock, you must rebuild — docker compose restart bot re-runs the existing image and does not pick up code or config changes:

docker compose up -d --build bot

docker compose down preserves the database volume. docker compose down -v permanently wipes it.

Bare-metal setup

bun install
cp config.example.json config.json
cp custom.example.txt custom.txt
bun run start

Secrets come from environment variables or an ignored .env file. The schema initializes automatically.

Configuration

config.example.json is the source template. Key sections:

• database — PostgreSQL connection and pool size
• matrix — homeserver URL, bot MXID, token, device ID
• llm — model, utility model, apiBaseUrl, provider policy (opt-in), generation preset, webSearch (provider/engine/domains), multimodal (image attachments), retries
• memory — operator-level memory switch and embedding model
• embedding — embedding model, dimensions, avatarSweepConcurrency, avatarSweepMessageWindow, modalities (text/image/audio)
• bot — display name, ignored bots, interactive defaults, context limits, command prefix, administrators
• antispam, rateLimit, commandTimeout, logging — operational controls

Environment overrides:

Variable	Purpose
MATRIX_ACCESS_TOKEN	Matrix account token
OPENROUTER_API_KEY	Preferred OpenRouter key
LLM_API_KEY	Legacy API-key fallback
MEMORY_ENCRYPTION_KEY	Base64-encoded 32-byte memory key
CONFIG_PATH	Alternate config path; defaults to ./config.json
DATABASE_HOST	PostgreSQL host
DATABASE_PORT	PostgreSQL port
DATABASE_NAME	PostgreSQL database
DATABASE_USER	PostgreSQL user
DATABASE_PASSWORD	PostgreSQL password

SIGHUP reloads configuration and the custom prompt. It does not rotate the memory encryption key.

Room setup

Invite the bot to an unencrypted room — it joins automatically and leaves any encrypted room.

!interactive on
!memory enable

Both commands require bot admin or power level 100.

Commands

!help lists commands available to the sender. The default prefix is !.

Command	Purpose
!ping	Check responsiveness
!help	List available commands
!status	Show bot status and model configuration
!interactive <on|off>	Toggle model replies for the working room
!cwd <room>	Change the working room for explicit commands
!privacy [status]	Show global model-processing status
!memory <subcommand>	Manage semantic memory, inspection, and deletion
!loglevel <level>	Change runtime logging level
!rooms	List joined rooms
!leave [reason]	Leave the working room
!lock / !unlock	Toggle room lockdown
!mute / !unmute	Manage timed mutes
!whitelist <subcommand>	Manage room restrictions
!antispam <subcommand>	Configure antispam (requires PL 100 or bot admin)

Memory inspection is allowed only in an unencrypted one-to-one room with the requesting user and the bot. Deletion removes live memory records and best-effort redacts linked bot replies; it cannot erase original Matrix events, federated copies, provider copies, or backups.

Custom persona

Copy custom.example.txt to custom.txt and edit the identity and tone layer. Protocol, privacy, structured-output, search, Matrix-formatting, and safety rules remain in src/Llm/basePrompt.ts.

Development

bun install
bun test
bun run check

Useful scripts:

bun run start       # run once
bun run dev         # watch mode
bun run typecheck   # TypeScript only
bun run lint        # non-mutating Biome check
bun run format      # format source files

Do not use bun run lint:fix as verification — it mutates files and permits unsafe Biome rewrites.

Run deterministic tests in the isolated Compose profile:

docker compose --profile test run --rm --build test
docker compose --profile test down -v

The test profile uses a separate PostgreSQL service, fixed test-only credentials, and no production pgdata volume. Tests use example.org identities and must not load deployment configuration or credentials.

Memory-key recovery

The database stores a non-secret fingerprint of the configured memory key. A mismatch aborts startup without deleting memories. Restore the original key when possible.

If the key is permanently lost, disable memory, preserve any required encrypted backup, then remove user_memories, memory_disclosure_events, and memory_key_metadata before starting with a new key. This crypto-shreds live semantic memory but does not erase Matrix or provider copies.

License

Manifold is licensed under the GNU Affero General Public License v3.0 only. Modified versions offered over a network must make their corresponding source available under the same license.