Multi-Modal AI Systems

RapBotRito is more than a chat surface—it is a gated, multi-modal, tool-using agent that sits inside the primary dapp. The runtime blends Next.js streaming (dapp/app/lib/llm/handler.ts) with a Model Context Protocol server (dapp/app/lib/mcp/server/index.ts) and dozens of inline UI affordances (dapp/components/chatBot). This page catalogues the moving parts you will dive into throughout the AI Systems section.

Next.js route handlers stream tokens via Server-Sent Events, while the client ToolAwareTransport mirrors tool lifecycle events into the inline chip UI.

Runtime Surfaces

JWT-gated access, Redis-like durable quotas, and Pinecone-backed semantics keep the agent safe while still acknowledging on-chain context.

Governed Agent

Tools are registered centrally, authenticated per-call, and surfaced to the model based on chat mode.

Extensible Tools

Code Map

Use this mental map while reading the detailed pages:

Key Capabilities

Multi-provider orchestration. ai.server.ts lets you switch between OpenAI and local LM Studio models, define dedicated vision models, and configure the image generation backend.

Agentic chat modes — dapp/app/lib/llm/modes/configs/*.ts define aggressive rap battles, freestyle sessions, and agent battles, each with bespoke tool allow-lists.
Inline tooling UX — dapp/components/chatBot/ToolActivity renders per-tool chips, while useHydrateToolImages.ts feeds base64 images to the client without ever placing large blobs in the chat stream.
Semantic context — pinecone.config.ts, the seeding scripts under dapp/pinecone, and the MCP pinecone_search tool let the agent pull memes, rhymes, and lore on demand.
Crypto-aware automation — send-crypto.ts, send-crypto-agent.ts, and mark-key-used.ts demonstrate how JWT claims, quotas, and chain configs combine to gate real transfers.

Continue with Runtime Architecture to see how these pieces stream together.