litellm/deploy/memory-pilot

Memory gateway pilot on Render

Run this branch as an isolated forwarding gateway. Colleagues keep their existing upstream LiteLLM key and model name and change only their gateway base URL. They need no plugin or client-side memory tools. Choosing the pilot URL opts them into the pilot; returning to the original URL stops using and collecting pilot memory.

Every model call uses that caller's upstream key. The upstream gateway continues to enforce its model permissions, budgets, rate limits, and guardrails. The pilot checks the key against the upstream model catalog, then registers its hash as a local virtual key so LiteLLM's normal authentication and memory authorization still apply. No upstream key or provider credential is configured on Render. The administrator credential belongs only to this pilot.

The forwarding pilot isolates memories by virtual key. Upstream management APIs may deny ordinary keys access to user/team/org details, so the pilot does not infer those identities from client metadata. Install the feature directly in an organization's gateway to use its existing user/team/project/org policies. A regular gateway deployment reuses its existing PostgreSQL database with normal schema migrations. It does not need a separate memory database or vector service. This forwarding pilot has a separate database for isolation. Its memories are not automatically available on the original gateway. Sharing requires both deployments to run this feature against the same database and authenticated namespace; the pilot must not be connected to an older gateway's production database.

Create the service

1. Create a separate Render Postgres 16 database in the same region as the web service. Restrict public database access; use its internal connection URL.

2. Create a Python web service from this repository and the Memory V2 branch. A Standard service and the smallest paid Postgres plan are sufficient starting points for a small pilot. They incur Render hosting charges.

3. Set the build command to bash deploy/memory-pilot/build.sh, the start command to bash deploy/memory-pilot/start.sh, and the health path to /health/readiness. The build includes the dashboard from this branch. Set the service's maximum shutdown delay to 300 seconds so active requests can drain during a deployment. Uvicorn allows 290 seconds before cleanup

4. Set these environment variables in Render:

   Variable	Value
   DATABASE_URL	The new database's internal connection URL
   UPSTREAM_LITELLM_BASE_URL	Your original gateway URL, without /v1
   LITELLM_MASTER_KEY	A new random sk- administrator key
   LITELLM_SALT_KEY	A separate random encryption secret; preserve it across deploys
   PYTHON_VERSION	3.12.14
   NODE_VERSION	24.19.0
   NEXT_TELEMETRY_DISABLED	1
   PORT	4000

5. After deployment, open /ui/memory, sign in as admin using the pilot's master key, and save a policy for Whole gateway, Enabled automatically, Private to each virtual key. This policy persists across restarts. Memory stays disabled until an administrator enables it.

Equivalent activation through the API, with secrets supplied in shell variables:

curl --fail-with-body "$PILOT_URL/v2/memory/policies" \
  -H "Authorization: Bearer $PILOT_ADMIN_KEY" \
  -H 'Content-Type: application/json' \
  -X PUT \
  -d '{"target_type":"gateway","target_id":"*","activation":"automatic","scope":"key"}'

Administrators can instead require opt-in, disable a particular registered key, or disable the whole gateway. Under an opt-in policy, callers set their preference with PUT /v2/memory/preference and {"enabled":true} using their own key.

Try it

Set an OpenAI-compatible client's base URL to https://YOUR-SERVICE.onrender.com/v1. For Claude Code, set ANTHROPIC_BASE_URL to https://YOUR-SERVICE.onrender.com. Retain the same gateway key and model setting.

In one conversation, say “Remember that my demo project is Cobalt Heron and its staging port is 8347.” In a new conversation, ask “What is my demo project and its staging port?” Check actual saved entries with GET /v2/memory/entries using the same key. An unrelated key must not see them. Administrators can inspect, correct, or delete entries in Memory; callers can use the self-service API.

Behavior and limits

• Streaming keeps LiteLLM's configured SSE keepalives across silent memory rounds. The pilot sends comments every 15 seconds of silence and disables proxy buffering. A failure after streaming starts arrives as a native SSE error; before streaming starts, HTTP errors retain their retry delay
• Model calls retain LiteLLM's normal timeout and retry settings. The separate upstream credential check has a 20-second timeout. Deployments drain existing requests for up to five minutes; requests still running after that can be interrupted. Schedule pilot updates outside active office usage
• Supported surfaces: Chat Completions, Responses, and Anthropic Messages, including their native streaming responses and client tool continuation.
• The selected model must support function calling. The actual answering model receives catalog, fuzzy search, full-read, and observation-capture tools beside its normal client tools. The gateway executes only its own memory tools.
• A request allows at most eight model rounds and sixteen memory calls per round. One final reflection round can acknowledge an empty observation batch. Additional rounds use the same model and caller budget, and add latency and token spend.
• Captures are immediately visible after a confirmed save. Each observation keeps its title, relevance guidance, scope, kind, certainty, evidence, source, and actor. Corrections append observations. Agents receive no memory deletion tool.
• Search uses weighted fuzzy matching over the authorized scope. There is no vector database, extraction model, or nightly consolidation.
• Searches accept up to 16 distinct terms. Fuzzy matching checks up to 256 distinct words per field; exact terms still match anywhere in the field.
• Fixed instructions and tool definitions preserve prompt-prefix caching after warm-up. Dynamic catalogs and checkpoint IDs stay at the conversation tail. Complete-response caching is bypassed for memory rounds on both gateways so permission checks, retrieval, and capture execute against current state.
• Hidden tool continuations expire after 24 hours, hold at most one megabyte each, and are limited to 1,000 per key and scope. They contain gateway-added fragments, not another copy of the complete incoming transcript. Responses retrieval and continuation use gateway-owned response IDs; deleting one removes its model responses and temporary continuation records, not saved memories.
• /input_items returns 501 for gateway-owned response IDs. Retain the original client input; the hidden provider transcript is not a public input history.
• Foreground requests with one completion are supported. Use modern tools instead of legacy functions. The special Cursor conversion route, background responses, multiple completions, and WebSocket inference are outside this implementation.
• On gateway/backend deployments without shared Redis, first-time activation can take up to 30 seconds to reach another process. Policy revocation is checked against the primary database before memory operations.
• Each memory scope can hold up to 1,000 entries. Creation checks this limit atomically; correction and deletion remain available when the scope is full.
• Stored references are untrusted data. They cannot grant API permissions or change the namespace derived from authentication. Current user corrections take precedence. Replacements require the current revision.
• Invalid tool arguments return errors to the model. Infrastructure and model failures fail the request or stream instead of reporting a successful save. Administrators can disable memory to restore ordinary calls.
• Switching away or disabling memory stops automatic use; it does not delete existing entries. Delete memories explicitly through Memory or the API.
• Shared upstream keys share a pilot namespace. Give each person a distinct key when their memories must be private from each other.
• Other API surfaces are outside this forwarding pilot. Use the original gateway for embeddings, images, realtime, batches, and administration.