Commit graph

12281 commits

Author SHA1 Message Date
joshua-berri
083ddefa92
Merge pull request #40498 from BerriAI/litellm_fix_mcp_edit_tool_preview_7135
fix(mcp): refresh tool previews when editing connection settings
2026-09-09 19:48:00 -07:00
Mateo Wang
f0fac55fe1
Merge pull request #40114 from BerriAI/litellm_fix_background_polling_disconnect_guard
fix(responses): keep background polling alive after the client disconnects
2026-09-09 19:41:19 -07:00
Mateo Wang
c005431cad
Merge pull request #40489 from BerriAI/litellm_lite_claude_apikeyhelper_conflict
fix(cli): let the apiKeyHelper supply Claude Code's key under lite claude
2026-09-09 19:33:59 -07:00
Mateo Wang
26626348f8
Merge pull request #40270 from BerriAI/litellm_bedrock_sign_request_off_loop
fix(bedrock): sign requests off the event loop on every async path
2026-09-09 19:31:58 -07:00
mateo-berri
69c8c70547 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_background_polling_disconnect_guard 2026-09-09 19:21:47 -07:00
mateo-berri
ab9dc75411 merge: origin/litellm_internal_staging into litellm_lite_claude_apikeyhelper_conflict
Resolves the conflicts with the lite configure claude work from #40319: every persistent
writer and reader of Claude Code's settings file now resolves it through CLAUDE_CONFIG_DIR,
the lite up backup check only guards the default file, and each settings file keeps its own
undo receipt (the default file keeps ~/.litellm/claude_configure_state.json, any other file
gets ~/.litellm/claude_configure_state/<sha256 of its resolved path>.json).
2026-09-09 19:09:52 -07:00
mateo-berri
d93baa3f2c fix(bedrock): sign on a dedicated executor instead of the shared default one
asyncio.to_thread puts every Bedrock signing on the loop's default executor, the same pool every provider's async entry point hops through, so signings parked on botocore's refresh lock queued unrelated providers behind Bedrock. run_aws_signing runs them on a 16-thread pool only AWS signing uses
2026-09-09 18:52:00 -07:00
mateo-berri
708381948b fix(cli): compare the resolved settings path when deciding whether the lite up and autoroute backups guard it 2026-09-09 18:34:07 -07:00
Joshua Valluru
3169c80252 fix(mcp): preserve IPv6 hosts when comparing preview origins 2026-09-09 18:21:55 -07:00
mateo-berri
b3f2de058b fix(cli): only guard the default settings.json with the lite up and autoroute backups on login --config-claude 2026-09-09 18:19:56 -07:00
mateo-berri
c1ca963d75 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_5546_count_tokens_offload 2026-09-09 18:18:17 -07:00
mateo-berri
47eb7f4349 fix(token_counter): give each event loop its own count limiter
A single process-wide anyio.CapacityLimiter wakes waiters with the
asyncio.Event of whichever loop created it, so a fifth loop in another
thread (asyncio.run per request under Celery, gunicorn sync workers, or
run_async_function from function_with_fallbacks) waited forever once four
counts were in flight, and building it at import time raised
AsyncLibraryNotFoundError on anyio below 4.2. The limiter now lives in a
RunVar and is created on first use inside the running loop.

TOKEN_COUNTER_MAX_CONCURRENT_COUNTS reads from the environment like
TOKEN_COUNTER_MAX_EXACT_CHARS, and the proxy's tokenizer lookup runs on the
shared thread pool so a Hub download no longer holds a count slot.
2026-09-09 18:17:32 -07:00
Mateo Wang
21ae29b759
Merge pull request #40284 from BerriAI/litellm_legacy_hook_streaming_pipeline_step
feat(guardrails): run legacy post-call hooks as streaming pipeline steps
2026-09-09 18:14:37 -07:00
Joshua Valluru
493c98f9bf fix(mcp): normalize default ports in preview origins 2026-09-09 18:11:52 -07:00
Joshua Valluru
3523f3731c fix(mcp): bind inherited preview credentials to the saved origin 2026-09-09 18:04:36 -07:00
tin-berri
317b29e69d
feat(cli): add lite configure claude and lite unconfigure claude (#40319)
Persistently route Claude Code through a LiteLLM proxy with a long-lived virtual key or the stored lite login, turn on gateway model discovery so /model lists the proxy's models, optionally pick the model Claude Code starts on, and record what changed so unconfigure restores only the keys the user has not touched since. lite login --config-claude writes through the same receipt and is undoable too. The two settings merges (lite up / --config-claude and lite autoroute) collapse into one credential-aware merge
2026-09-09 18:02:23 -07:00
mateo-berri
b554d79914 fix(cli): resolve Claude Code's settings file through CLAUDE_CONFIG_DIR
Claude Code reads settings.json from CLAUDE_CONFIG_DIR when it is set,
while lite login --config-claude wrote to ~/.claude/settings.json and
lite claude checked that same file for its apiKeyHelper. With the
override set, lite could drop ANTHROPIC_AUTH_TOKEN because the helper
lives in a file Claude Code never reads, leaving it with no key at all.

claude_settings_path(environ) now picks the file the way Claude Code
does, and both commands go through it. The CLI test directory gets an
autouse conftest that isolates HOME, USERPROFILE, and CLAUDE_CONFIG_DIR
per test and fails any test that writes the developer's real
settings.json, restoring it first.
2026-09-09 18:01:11 -07:00
Joshua Valluru
245369764e fix(mcp): preserve edited settings in static connection previews 2026-09-09 17:49:52 -07:00
ryan-crabbe-berri
a650178ebe
Merge pull request #33703 from BerriAI/litellm_fix_jwt_key_mapping_cascade_delete
fix(jwt): cascade-delete JWT key mappings when their virtual key is deleted
2026-09-09 16:52:20 -07:00
ryan-crabbe-berri
969d152c4b fix(jwt): evict jwt_key_mapping cache when a virtual key is deleted
The FK cascade drops the LiteLLM_JWTKeyMapping row, but the cached
jwt_key_mapping:{claim}:{value} entry still resolved to the deleted token
hash, so every JWT call from that identity failed until
virtual_key_mapping_cache_ttl expired instead of auto-registering against a
recreated key. delete_verification_tokens now snapshots the mapping cache
keys before the delete and evicts them across replicas afterwards, the same
way /key/regenerate already does.

Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
2026-09-09 16:40:25 -07:00
mateo-berri
7e137958e9 fix(cli): let the apiKeyHelper supply Claude Code's key under lite claude
`lite login --config-claude` writes an apiKeyHelper into ~/.claude/settings.json,
and `lite claude` then also exported ANTHROPIC_AUTH_TOKEN, so Claude Code opened
with its "Both ANTHROPIC_AUTH_TOKEN and apiKeyHelper set" banner and, because the
env token wins, never ran the helper that was meant to refresh the key.

When the launch key is the stored login key and settings.json carries exactly
the helper lite wrote for this base URL, `lite claude` now leaves the token out
of the env (dropping an inherited one) so Claude Code asks the helper. An
explicit --api-key or LITELLM_PROXY_API_KEY, a helper for another proxy, a
hand-written helper, or no settings file keep the previous behavior.
2026-09-09 16:12:40 -07:00
tin-berri
eb45a088d3
fix(router): resolve team-scoped auto-routers by their public name (#40432)
A team-scoped auto-router is stored under an internal
model_name_{team_id}_{uuid} with the caller-facing name in
model_info.team_public_model_name, and the four pre-routing strategy
registries key on that internal name. A team key asks for the public name,
so the strategy lookup missed, the team early-resolve exit handed back the
marker deployment itself, and every call 400'd with "Unmapped LLM provider".

The strategy lookup now resolves the requested name through the same
team-first, then global, then admin-across-teams deployment resolution the
deployment path uses, and looks the registries up under the model_name of
whatever that resolves to. Both exits of _common_checks_available_deployment
drop strategy markers through one helper, so a marker-only resolution is
rejected as uncallable on every path. The request team id has one reader.

Resolves LIT-7363


Claude-Session: https://claude.ai/code/session_01NU97S7d2FUDDvTk59k53Wp

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-09 15:53:05 -07:00
mateo-berri
5fdbb2a1c8 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_legacy_hook_streaming_pipeline_step 2026-09-09 15:52:48 -07:00
mateo-berri
d644e4970f Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_bedrock_sign_request_off_loop
# Conflicts:
#	tests/test_litellm/llms/bedrock/embed/test_bedrock_embedding.py
2026-09-09 15:43:48 -07:00
devin-ai-integration[bot]
6e96885629
fix(proxy): accept non-string callback vars in default_team_settings (#40458)
* fix(proxy): accept non-string callback vars in default_team_settings

A YAML boolean such as turn_off_message_logging: true in a
default_team_settings block failed TeamCallbackMetadata's str-only
callback_vars validation and errored the request before any callback
ran. Stringify the value the same way AddTeamCallback does.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): drop docstring from default_team_settings bool regression test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): move default_team_settings bool regression test to mapped pre_call_utils suite

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 15:40:15 -07:00
ryan-crabbe-berri
eee239cd9e fix(jwt): cascade-delete JWT key mappings when their virtual key is deleted
LiteLLM_JWTKeyMapping_token_fkey was created ON DELETE RESTRICT, so deleting
a virtual key that a JWT mapping pointed at failed with a foreign key
violation on every deletion path (/key/delete, Admin UI, alias delete,
team and user cascades). Declaring onDelete: Cascade on the relation lets
the database clean the mapping up uniformly, so the next JWT call from that
identity re-registers against the newly created key.

Rebase of #33703 onto current staging.

Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
2026-09-09 15:39:21 -07:00
Mateo Wang
2b2e6cf283
Merge pull request #40271 from BerriAI/litellm_lit_7174_stream_tool_call_rewrites
feat(guardrails): deliver tool-call rewrites into buffered chat, Responses, and Messages streams
2026-09-09 15:36:13 -07:00
devin-ai-integration[bot]
264fc82dc1
fix(rate_limiter): attach v3 priority rate limit headers on /v1/messages (#37228)
Some checks are pending
LiteLLM Rust / rust-test (push) Waiting to run
Unit Tests: Documentation Validation / documentation (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests / integrations (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
* fix(rate_limiter): attach v3 priority rate limit headers on /v1/messages

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(playground): honor the Stream responses toggle for /v1/messages

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rate_limiter): drop explanatory docstrings from v3 dict response tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 15:14:10 -07:00
devin-ai-integration[bot]
2488f84b02
fix(proxy): keep a body litellm_session_id in SpendLogs under missing_session_id omit (#40379)
* fix(proxy): keep a body litellm_session_id in SpendLogs under missing_session_id omit

Under general_settings.missing_session_id: omit, apply_missing_session_id_policy now
mirrors a client-supplied top-level litellm_session_id into metadata.session_id when the
client did not set one there, so SpendLogs.session_id and Langfuse agree with the session
callbacks already report through StandardLoggingPayload.session_id

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep client metadata.session_id ahead of body litellm_session_id on litellm_metadata routes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): drop docstrings from the missing_session_id omit regression tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 14:53:35 -07:00
Oliver Jensen
f2eacc3c36
Merge pull request #40116 from BerriAI/litellm_disable_env_credential_login
feat(auth): add disable_env_credential_login setting with admin ui warning
2026-09-09 23:08:26 +02:00
Mateo Wang
11b31c19be
Merge pull request #40274 from BerriAI/litellm_post_call_pipeline_background_responses
feat(guardrails): run post_call policy pipelines on background Responses retrieval
2026-09-09 13:56:37 -07:00
mateo-berri
c5e93aff13 fix(proxy): warn at submit when a body-selected post_call policy is deferred 2026-09-09 13:45:26 -07:00
mateo-berri
8ecd9c16cd Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_legacy_hook_streaming_pipeline_step 2026-09-09 13:30:39 -07:00
mateo-berri
8bb6d8c120 chore(lint): bring ANN202 and BLE001 back under the strict-rule budget
Annotate the return types of dispatch_async and transform_then_dispatch in llm_http_handler and _send_batch in azure_sentinel, and mark four legitimate broad catches with the repo's noqa convention, so the promote PR's lint job passes the strict gate again. Supersedes #40328.
2026-09-09 12:34:08 -07:00
devin-ai-integration[bot]
096984bfc2
fix(proxy): pin multi-root CA bundle to the server's root before handing it to Prisma (#40428)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 11:19:29 -07:00
devin-ai-integration[bot]
c7163a80dd
perf(proxy): collapse per-worker SGR upserts into one statement per flush (#40362)
Each proxy worker flushed one Prisma upsert per active (date, category, route)
bucket every interval, so the Postgres primary saw workers x routes statements
per interval across the deployment. A flush now builds a single multi-row
INSERT ... ON CONFLICT DO UPDATE, and with use_redis_transaction_buffer on the
workers push snapshots to a Redis list that one lease-holding pod folds and
commits, so the whole deployment costs one statement per interval. The leader
keeps popping until the list is empty so a deployment wider than the dequeue
cap cannot build a backlog, and rows that fail both the commit and the Redis
re-queue fall back to the leader's own accumulator instead of being lost.

Resolves LIT-7371

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 10:48:06 -07:00
devin-ai-integration[bot]
7c6e33ef70
fix(proxy): ignore team_id="" on /key/update so team-less keys can be updated and imported (#40421)
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 10:38:26 -07:00
devin-ai-integration[bot]
996ee5635a
perf(proxy): pipeline spend counter increments into one Redis call per request (#40371)
* perf(proxy): pipeline spend counter increments into one redis call

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): apply surviving spend increments before raising scope error

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): ruff format spend counter helpers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): settle inner spend counter gathers and fall back per key on pipeline failure

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): suppress BLE001 on pipeline fallback catch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): invalidate all batched spend counters on pipeline failure

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-09 10:35:50 -07:00
joshua-berri
ea0851de79
Merge pull request #40359 from BerriAI/litellm_fix_mcp_connection_errors_31318
fix(mcp): surface connection failures across transports
2026-09-09 10:22:25 -07:00
ryan-crabbe-berri
53d23b90ee Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_cost_estimate_cache_tokens
# Conflicts:
#	litellm/litellm_core_utils/litellm_logging.py
2026-09-09 10:05:48 -07:00
kerry-berri
bfcc6404d3
Merge pull request #40350 from BerriAI/litellm_cost_map_background_retries
fix(cost-map): keep first fetch blocking, run retries in background
2026-09-09 09:59:07 -07:00
ryan-crabbe-berri
36f3ca95d8 fix(proxy): report /cost/estimate rates from the call that billed them
The estimate looked the reported per-token rates up a second time, with the
provider this endpoint resolved rather than the one completion_cost infers.
The provider decides whether a token tier threshold is inclusive, so an
unrouted xai model sitting exactly on 200k billed at the tier rate and
reported the base rate, half of it.

completion_cost now hands back the rates its own lines were billed at, and
the endpoint reports those.

Claude-Session: https://claude.ai/code/session_01RLKy5DMi3XCBUJ37WzfNi1
2026-09-09 09:57:38 -07:00
Joshua Valluru
dbf9490229 fix(mcp): expose shared SDK timeout normalization 2026-09-09 09:44:14 -07:00
Mateo Wang
9b6c7c8bf0
Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-09-09 09:17:14 -07:00
ryan-crabbe-berri
360fa65631 Merge branch 'litellm_internal_staging' into litellm_lit_5858_jwt_team_grants
Claude-Session: https://claude.ai/code/session_01Hn5E8Jz1LjGLFyiYxBRcBW
2026-09-09 08:58:50 -07:00
Joshua Valluru
a1588c2602 fix(mcp): preserve stream failures across transports 2026-09-09 08:28:18 -07:00
Joshua Valluru
0225a16f48 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_mcp_connection_errors_31318 2026-09-09 08:00:01 -07:00
Yuneng Jiang
9d90a54491
revert(model-management): roll back #40047
This reverts commit e8e3172d7d

Restore the previous model update and router cost registration behavior while pricing compatibility is investigated
2026-09-09 00:06:26 -07:00
devin-ai-integration[bot]
ef3a3c16ae
feat(guardrails): map each guardrail scan id to its guardrail, stage and provider (#40327)
* feat(guardrails): map each guardrail scan id to its guardrail, stage and provider

Adds the x-litellm-guardrail-scan-metadata response header, a JSON list of
{guardrail, stage, provider, scan_id} entries, next to the existing
comma-separated x-litellm-guardrail-scan-id header. Prisma AIRS records the
execution stage for every scan and OpenAI Moderation now records its
moderation id too. The new metadata key is internal: client-supplied values
are stripped and it is exposed through the UI CORS allow list.

Resolves LIT-6018

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(guardrails): cap the scan metadata response header at a configurable length

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): hardcode the scan metadata header cap

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-08 23:32:31 -07:00
yuneng-jiang
e8e3172d7d
fix(model-management): honor an explicit null as a clear on model update (#40047)
* fix(model-management): honor an explicit null as a clear on model update

PATCH /model/{model_id}/update merged the patch with exclude_none and then
popped explicit nulls only for the mirrored pricing fields, so a null sent for
max_input_tokens, mode, supports_vision or any other key was dropped and a value
pinned by an earlier save could never be removed.

The route now follows JSON Merge Patch over both blobs: a key absent from the
body is unchanged, a key sent as null is removed from the stored row, and a key
sent with a value is set. Ownership and identity keys keep ignoring a null, as
do the fields the stored models require, since clearing one writes a row no
reload can rebuild. Mirrored pricing keys still clear from both blobs.

Clearing a price also needed the router to stop merging a deployment's cost-map
entry onto its previous registration, which left the old rate in place and kept
billing at a price the deployment no longer carried.

Adds a create, read, partial-update, clear, enforce, delete lifecycle e2e that
reads back on every replica, and a harness helper for that read-back.

* fix(router): keep a deployment id that names a real model from evicting its catalog entry

Deployments are keyed into litellm.model_cost alongside the built-in catalog, so
evicting a deployment's stale entry by id could take a real model's entry with it:
registering a deployment whose model_info.id is "gpt-4o" stripped that model's
pricing, context window and capability flags process-wide, for every other
deployment of it, until the next price-map reload.

Only evict an entry this registration owns. A colliding id keeps the previous
merge, which pollutes the catalog entry rather than emptying it.

Also pins the Admin UI round trip: the model edit form echoes the whole /model/info
row back on save, and that read reports every key the deployment never stored as an
explicit null, so the clear path has to leave those keys alone.

* fix(router): decide cost-map eviction by what this registrar created

The previous guard read a catalog entry off `litellm_provider`, so a deployment
that declares its own provider in model_info was treated as one and kept billing
at a price it no longer carried. It also only held for a single registration: a
second one under a colliding id saw the id the first merge left behind and
evicted the catalog entry anyway.

Track the cost-map keys this registrar creates instead. A key it created is
evicted before re-registration; one it did not is left to merge, which is what a
deployment id colliding with a catalog model name needs.

Also folds the required-fields comment into the docstring that already gives the
reason.

* fix(router): release a deployment's cost-map key when it is deleted

The ownership ledger only grew. A deleted deployment kept its claim, so if a
later catalog refresh started publishing a model under that same name, the next
registration would treat the catalog entry as the deployment's own and evict it.

Deleting a deployment now gives the key back, which also stops the ledger
growing for the life of the process.

* fix(router): hold a cost-map key while another live router still serves it

The claim is process-wide but the release was per-deletion, so with two routers
serving one deployment id, the first deletion put the survivor back on merging
and the price it had just cleared would keep billing.

Release the key only once no live router still serves that id.

* fix(router): register a router in the live set when it gains a deployment

_live_routers was only joined when a router was constructed with a model_list,
but a router built empty is populated through add_deployment, and the empty
branch exists for exactly that. Such a router was invisible to the live-router
scan, so deleting the deployment from another router released the shared
cost-map key while it was still serving that id.

Joining the set where a deployment enters the list covers every path, and it
also lets a price reload rebuild what a dynamically built router serves.

* fix(e2e): read the stored model row from the control plane, not each gateway

The lifecycle suite polled /model/info on every URL in PROXY_REPLICA_URLS. Those
URLs are the stack's gateways, and gateway/routes/allowlist.py trims them to the
LLM data-plane surface, so /model/info answers only on the backend and 404s on
every replica. All five tests failed at their first read-back in CI while passing
against a monolith, where one process serves both planes.

The stored row has one answer behind it, so it is read through the shared
transport, which routes control-plane paths to the backend. What every gateway
must agree on is which models it serves, so the create and delete steps poll
/v1/models per replica instead, a route the gateway does serve.
read_back_everywhere now rejects a control-plane path outright rather than
timing out on it.

Two things surfaced behind that. /public/ was missing from the transport's
control-plane prefixes, so model_cost_map() was routed to a gateway and 404'd,
and the billing steps needed a data-plane wait: a PATCH lands on the backend and
each gateway picks it up on its own config reload, measured here at 12-24s, so
they now drive calls until the new rate reaches the spend row and let the
deadline fail them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1S92J8gSxxKVe1JBzxWBF

* test(models): keep polling outcomes immutable and document shared ownership

* test: validate opaque stream IDs and hide log-reader credentials

* test: isolate auto-router scenarios and clean partial setup

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 06:10:18 +00:00