Main deleted router_python_core.py when routing moved to the Rust router, so the stale thread 404 skip now sits in RustHealthCallbacks.failure_exits, which the attempt path, the failure callback and the stream failure reporters all read
* fix(proxy): bill an OpenAI pass-through background response once across repeated reads
A finished background response now bills only on the first pass-through call
that sees it finished with usage. The claim is an insert on the new
LiteLLM_BackgroundResponseSettlement table, keyed by response id, so it holds
across workers and replicas. Every later retrieve, cancel, streamed create, or
resumed stream logs like an unbilled read. With no database or a failed insert
the call still bills.
* fix(proxy): keep pass-through background billing fail-open without prisma and skip foreground stream re-parse
* fix(proxy): claim background responses on failed streams and bound the claim
A resumed stream that fails after response.completed billed its usage through
the partial-usage path without claiming the response id, so a later read billed
it again. That path now claims the id and logs $0 when an earlier read paid.
The claim uses create_many(skip_duplicates=True) so a repeat read no longer
raises a unique violation in Postgres, and it waits at most 2 s before billing,
since the claim now runs before the client gets its response.
* fix(pass_through): keep the background response claim and its spend write together under cancellation and managed ids
* test(pass_through): cover a failed stream whose failure logging cannot be scheduled
* fix(proxy): start the background response settlement migration with a lock timeout
* fix(passthrough): leave a failed background stream's charge to its next read
A stream that failed after response.completed claimed the response id while its failure row billed $0, so no later read paid. The failure path is back to main's behavior, so the next finished read claims and bills once
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The test stores its recording Redis client in litellm's shared client
cache with the default ttl. When earlier tests in the same process have
filled that cache with longer lived clients, the next insert evicts the
recording client, the pin is read through a fresh client, and the test
sees no commands on the wire.
* fix(proxy): let the OS assign Prisma engine ports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): clear Prisma connection state after failed startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer
Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525,
first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown
Codex versions still get the catalog cut to 1 MiB, since they drop a larger one.
* fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog
Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those
clients counted as too old for the uncut catalog, their custom reasoning levels and stock
entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does.
* test(proxy): type the Codex catalog byte-limit test
* refactor(proxy): gate the uncut Codex catalog on the 0.166 release line
Read only the major and minor numbers of client_version instead of a shared
semver comparison, and leave the reasoning level and stock entry version
checks as they are on main.
* refactor(proxy): read every Codex client_version with one helper
Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version
parser rejects. The catalog byte limit, the per-model minimum version and the
custom reasoning level checks now share one reader for Codex's format, which
also bounds digit counts so an absurd version can't fail the listing.
* fix: make scheduled spend resets idempotent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover reset roundoff with shared Redis workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: isolate reset workers and type raw adapters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep updated_at moving on zero-spend budget resets
* fix: never leave spend below zero after a budget reset
* fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged
Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time
* fix(proxy): keep a resent budget reset time only once it is due
A save that resends the same budget_duration keeps the stored reset time
only when that time has already passed, so the scheduled sweep still
resets the period it is due for. A row that is not due yet realigns its
reset time on save, the same as before.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(transcription): treat libsndfile's unknown frame count as no duration
* refactor(transcription): inline the unknown frame count check
* fix(transcription): read the frame count once to keep the type gate flat
* test(transcription): annotate the new passthrough test's locals as Final
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(budgets): add per-model tpm and rpm limit maps to budgets and customers
Budgets gain model_tpm_limit and model_rpm_limit (model name to whole number, 0 or more).
/budget/new, /budget/update, /budget/info and /customer/new carry them, and the auth
path loads a customer's maps onto UserAPIKeyAuth as end_user_model_tpm_limit and
end_user_model_rpm_limit so the limiter can enforce them in a follow-up.
* fix(budgets): keep org, team and project per-model limits in metadata; lock_timeout and type gate fixes
Organizations, teams and projects store model_tpm_limit / model_rpm_limit in their own metadata,
which the limiter reads, so the shared budget allowlist leaves those two fields to customers.
Migration starts with SET LOCAL lock_timeout. Mapping annotations and rebind-ok reasons satisfy
the type discipline gate. Customer info tests expect the two new budget keys.
* fix(organizations): keep stripping budget_id and flattening nested per-model maps on update
The update path pops every budget field from the organization row again, so a caller's budget_id
never changes the attached budget, and a nested litellm_budget_table still flattens the two maps
into org metadata. Drops the explanatory comment on CUSTOMER_ONLY_BUDGET_FIELDS.
* fix(types): keep basedpyright and ruff strict counts flat for the new budget fields
Reads the per-model maps off the end-user params with membership checks instead of .get on an
untyped dict, writes the inline customer budget through the client's typed jsonify_object, and
builds budget rows in handle_budget_for_entity with model_validate so the two Mapping fields never
receive object-typed kwargs.
* chore: drop unrelated reformatting from project endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: update the spend tests that still expected the pre-ledger accounting queues
#49872 made the SpendLogs ledger the default accounting path on purpose. A batch with no billable request now writes no cost row, a call the guardrail fails closed during a vendor outage keeps one spend row of either status while its key is still charged once, and charges left at a graceful shutdown are applied by the next start. The daily spend batch shutdown tests exercise the old queue, so they pin use_spend_logs_ledger to false.
* test(integration): price the guardrail relay model so the outage test can assert the charge
The relay model had no price, so every stream cost $0 and the outage test could not tell a charged call from an uncharged one. Both rigs now price it, and the test expects each served stream to cost 9 input and 4 output tokens and the key to be charged for all sixteen.
Also notes the skipped batch cost row case in the e2e rules file, next to the other explicit skips.
* test: repair four tests that fail for reasons outside the product
The OpenAPI docs gate test rebuilds the Rust extension with uv, and the integration lane blocks outbound traffic, so uv could not refresh its PyPI index for maturin. The test now runs uv offline against the cache the dependency install already filled.
The response cache test expected a chat reply blocked with empty content to be cached. Since #50057 a reply with no output is never cached, so that case now expects the repeat to reach the upstream and the real answer to be cached.
The Gemini cache hit e2e test gave the model 16 tokens, which its thinking can use up, and a reply with no text is not cached. It now allows 512 and checks the first reply has text.
The Together cache read e2e test picks the cheapest model with a cache read price. Since #50491 that is a -latest alias, whose target reports no cached tokens. Aliases are now left out of the pick.
* fix(proxy): give rag ingest managed file ids and team credentials on vector store file routes
RAG ingest through a team's OpenAI deployment now registers the uploaded file as a LiteLLM managed file and returns its managed id, so GET /v1/files lists it for the caller. Re-ingesting a managed id attaches the provider file and returns the caller's managed id
Model-less vector store file calls (create, list, retrieve, content, update, delete) from a team key now use the team's OpenAI deployment that ingest uploads through, via the router, and fall back to the direct path when the key may not use one. Explicit api_key, api_base, litellm_credential_name, and forwarded provider keys still win, and blank values count as absent
* fix(proxy): route managed vector store file ids to their team deployment and keep missing-file 404s off the router
Model-less team vector store file calls now go straight to the provider with the team deployment's
credentials, so 404s on missing files no longer put the team deployment in cooldown.
A managed file id on the vector store file routes now uses the deployment named by its
llm_output_file_model_id when that deployment is proxy-wide or belongs to the caller's team, and falls
back to target_model_names otherwise.
* fix(rag): keep managed ids and team checks on every file and search path
Ingest no longer registers a raw file id when file_url is empty, a managed
file id on a proxy without the managed files hook answers 400, and vector
store search and /v1/rag/query hits show the managed id ingest returned.
Model-less vector store file calls use the team deployment only after the
key, team, and member model checks pass, fall through to the router when the
team has no OpenAI deployment, and log the deployment they used.
* fix(rag): keep provider ids on lookup failures and mint managed ids only for keys that can use them
A database error while mapping search hits to managed ids keeps OpenAI's ids instead of failing a finished search. Ingest registers a managed id only when the caller passes the team deployment's model checks, so a model-limited key keeps the raw id it can still use, and the file routes share that check.
* fix(rag): attach the managed file copy that matches the ingest credentials
* fix(vector_store_files): route a managed file id to the caller team's copy
* fix(vector_store_files): keep the upload copy for a store registered with its own credentials
* fix(vector_store_files): use a managed file's team copy only when the store holds it
* fix(vector_store_files): attach a managed file's team copy and keep named credential options out of the store check
* fix(vector_store_files): attach the team copy only to a store that routes to the team deployment
* fix(vector_store_files): keep a managed store's routing and check the upload copy through its own deployment
* fix(vector_store_files): route a managed file to its upload deployment only when the key may call it
A managed file id names its upload deployment, and with require_managed_files off that id is not checked against an ownership record. The routes now use that deployment only when the caller may call its public model, else they keep target_model_names routing as on main.
* fix(vector_store_files): log a pinned managed file call under its deployment's public model name
* fix(vector_store_files): send a pinned managed file call's deployment key only to that deployment's base
* fix(vector_store_files): keep stored credential headers intact and fail the store check on errors other than 404
* fix(vector_store_files): answer a failed store check with the provider status and copy credentials read by name
* fix(vector_store_files): send a call to the upload copy when the store check fails without a 404
A store check error other than a 404 now leaves the call on the upload copy, the way main always routed it, so the real call keeps the router's retries, the provider's status, and its failure log instead of failing on one transient error
* fix(vector_store_files): keep a credential's no-log key out of the store check
A named credential carrying a no-log key made the store check call aretrieve with no-log twice, so every managed id call through that deployment answered 500 where main answered 200
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): log keys and teams still on the deprecated empty models defaults at boot
* fix(proxy): read the repository tables inside the empty models boot check so a client error never fails startup
* fix(proxy): say which empty-models rows keep access in 2.0.0
Teamless keys listing all-team-models get no models in 2.0.0, so the
boot warning no longer says every counted row keeps its access. It
names each upgrade outcome only when its kind is in use, says the counts
include blocked and expired rows, and words each kind by what the
setting means rather than what it grants
* fix(proxy): drop the unbacked 2.0.0 rewrite promise from the boot warning
The warning no longer says the upgrade keeps access for empty models
lists, since no shipped code or public page backs that yet. It says
2.0.0 removes the defaults and how to keep today's access, and keeps
the advice for teamless all-team-models keys. The integration test
drops an empty-database assertion that ran before the check finished
* fix(proxy): leave rows with access groups out of the empty models boot warning
* fix(proxy): count access group rows, skip management and read-only keys in the empty models warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add customer spend breakdown and filtering by current team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: recover legacy customer team spend and tolerate lookup outages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf: resolve customer team keys once per daily activity query
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: paginate customer team activity in SQL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust
* fix(model-cost): keep main's compat helpers pure and move network tests to integration
ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Google AI Studio answers generationConfig.candidate_count "" with a 400 before it generates anything, so the truncated 200 this case scripts is a reply Google never gives. The other malformed count and cap cases stay.
* fix(anthropic-messages): apply additional_drop_params before the unsupported-param gates
* fix(anthropic-messages): apply additional_drop_params before the unsupported param checks
Share the tolerant drop list parser between the Python handler and the Rust host route, clear listed params before the Rust gate, and keep the prompt cache baseline body in step with the real request
* fix(anthropic-messages): leave additional_drop_params keys out of the Rust host body
* test(anthropic-messages): annotate the new test locals with Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): merge default metadata under the request's instead of replacing it
* fix(router): join default metadata tags after the request's tags
A router default_litellm_params.metadata.tags list replaced the request's tags, so the deployment and credential tags the router adds never reached the logged request tags. Default tags now follow the request's tags, skipping ones already present, like the proxy's key tag merge
* fix(router): keep masking a router default turns on
Merging default metadata under the request let a caller's mask_input or mask_output false switch off masking an admin set in default_litellm_params.metadata. A masking flag the default turns on now stays on, and a caller can still turn masking on. Also gives this PR's test suppressions a reason and adds a missing blank line
* fix(router): keep the router's cache namespace and drop malformed caller tags and dicts
A caller's metadata.redis_namespace no longer beats a router default, so it
cannot pick another tenant's cache partition. A caller tags value that is not a
list, or a non-dict value under a dict default, is treated as absent the way the
proxy's merge_tags treats malformed tags, so the router defaults still apply.
* test(router): check that mask_output stays on when the router default turns it on
* fix(router): keep a router default guardrail whatever guardrails the request lists
* fix(router): keep the admin's settings for a router default guardrail when the caller lists it
* test(router): check that a router default guardrail entry keeps the admin's settings
* fix(router): keep a router default private Langfuse trace private whatever the caller sends
* fix(router): keep a router default function such as langfuse_masking_function over the caller's value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Several integration tests on main still assert behavior that later changes replaced on purpose, so they fail on every scheduled run, and one of them stops the whole sdk group at import.
The batch cost case expected cached tokens to bill at the Batch input price when a price row has no Batch cache price. Since #50062 they bill at the Standard cache price, so the case now expects 0.000235 input cost and is renamed to say so.
The OTel v2 attribute test pinned langfuse_otel and langtrace as carrying no request metadata baggage. Since #50373 every OTel v2 destination carries it, so all seven callbacks share one expectation and the Langfuse baggage test that was deferred under LIT-9128 runs again.
The clean-EOF outage test trips its database relay on the text of the spend log insert. The ledger writes that insert without the "public" schema prefix, so the relay never tripped and the test timed out before it checked anything. The trigger now matches the insert on both write paths.
The usage routing tests expected cooldown, tpm and rpm keys in one MGET. Since #50170 the cooldown keys are read in their own MGET, at most once per DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS. Both the proxy and the SDK test now expect the two MGETs, and the proxy test waits out that interval after its /model/info poll, which reads the same keys.
The concurrent deployment delete test held its writer inside Router._update_deployment_indices_after_removal, which went away with the Python router core in #50178, so the file failed at import and stopped the whole sdk group. It now holds the writer where the registry reads the model list again after router-core applied the change.
* feat(proxy): let a key's logging entry reference a proxy credential
A virtual key's metadata.logging entry can name a proxy credential with
litellm_credential_name. At request time the credential's values, which config
credential_list resolves through the secret manager at boot, are merged under the
entry's callback_vars, with the entry's own values winning. An entry whose
credential is not defined is dropped, so it never falls back to the proxy's own
callback environment.
Only proxy admins may change the logging entries of a key when the stored or
requested list holds a reference (403), and a reference to an undefined
credential is a 400. The check runs on key generate, update, team bulk update,
regenerate and /user/new, and compares decrypted entries so re-saving metadata
read back from /key/info is not a change.
Resolves LIT-10061
* fix(proxy): drop a key logging entry whose referenced credential holds no logging values
* fix(proxy): only require newly added credential references to exist on key writes
* fix(proxy): log a dropped credential reference as an error so /key/health reports it
* fix(proxy): gate credential references on bulk-created user keys
* fix(proxy): keep failing requests on keys with non-object logging metadata
A key whose stored metadata.logging is not a list of objects failed every
request with a 500 before this branch. The credential reference rewrite
turned those values into no entries, so the request quietly succeeded
without the key's logging. Raise a TypeError for those shapes again so the
request path behaves as it did on main, while empty values and objects with
unknown fields still go through the existing skip.
* test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging
The malformed logging cell expects the base behavior that 7ac4ab1356
restored, a 500 for a key whose stored metadata.logging is not a list of
objects and a 200 for an empty list
* fix(proxy): keep a 500 for teams with non-object logging metadata
On main, a team whose stored metadata.logging held non-object items failed
chat requests with a 500, because AddTeamCallback(**item) raised a TypeError.
This branch switched to AddTeamCallback.model_validate, whose before-validator
raises AttributeError on those items, and the proxy maps AttributeError to a
400 invalid_request_error. Raise a TypeError for non-mapping items before
validation so the status matches main again, as the key path already does.
* fix(proxy): read an absent team membership row as zero spend under fail-closed budget enforcement
A team user under a default member budget has no LiteLLM_TeamMembership row until spend is persisted. SpendCounterReseed.from_db returned None for that absent row, the same value as a failed DB read, so fail_closed_budget_enforcement rejected every request from that user with 503. The DB answering that the row is absent is a verified zero, so from_db now returns 0.0 for a team-member counter without a row and still returns None when the read fails.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only read an absent membership row as zero when the team member key parses unambiguously
A team_id or user_id containing ":" makes the counter key split onto the wrong membership row, so that miss keeps returning None instead of a verified zero. Type the new unit tests.
* fix(proxy): refuse a fail-closed team member with no membership row with a 403 that names /team/member_add
Replaces the earlier reading of an absent LiteLLM_TeamMembership row as verified zero spend. A key minted with team_id and user_id but no /team/member_add has no row and the spend writer never creates one, so admitting it at zero would leave its member spend tracked only in Redis. With fail_closed_budget_enforcement on, _check_team_member_budget now rejects that key with a 403 that names the user, the team and the member_add call, before any spend read. With the flag off nothing changes. The membership load raises on a database failure and returns None only for a genuinely absent row, so the 403 never fires during an outage. spend_counter_reseed.py and the counter prefix constant go back to main.
* test(proxy): drive the no-membership 403 tests through the real membership and spend readers
Fake Postgres rows and the FakeSpendCounters double stand in for the external stores instead of patching get_team_membership and get_current_spend. Strip CR/LF from the ids in the fail-closed warning log.
* fix(proxy): read fail_closed_budget_enforcement from general_settings in the membership check
Importing proxy_server's private helper grew reportPrivateUsage past the type gate ceiling. Type the integration helper's return as JsonValue.
* fix(proxy): expose fail_closed_budget_enforcement_enabled for the membership check
Reading general_settings directly from auth_checks grew the unknown-type counts past the type gate ceiling; a small public wrapper in proxy_server keeps the gate flat.
* test(proxy): drop docstrings that restate the no-membership tests
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover claude-fable-5-2 on Vertex AI and Bedrock Converse
Scripted-upstream wire tests for the Fable capability flags on both routes, the
vertex_ai and us.anthropic cost cases every other Fable row has, Fable 5.2 in
BEDROCK_CONVERSE_MODELS, and bedrock/claude-fable-latest repointed to 5.2 like
the Anthropic and Vertex aliases already are.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): add GovCloud Bedrock rows for claude-fable-5-2
bedrock/us-gov-west-1 and bedrock/us-gov-east-1 rows copy the Fable 5.1 field
set with the GovCloud rates from the Bedrock price list.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(auth): grant access groups through principal attribute rules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): satisfy cache tracing and recursion CI checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): close access group rule authorization bypasses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): reject self-editable identity rule attributes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Centralize JWT model authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Cover centralized key checks and name denied JWT models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Test centralized authorization for custom auth and JWT mappings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve model normalization before budget validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use a controlled clock for WebSocket session deadline tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Unify JWT credential admission and simplify team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Simplify request-time access group authorization
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Use real requests in custom auth budget fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover claude-fable-5-2 capability flags and prompt-cache pricing over Anthropic direct
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): replace the fable-5-2 e2e suite with scripted-upstream wire and cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add claude-fable-5-2 wire and cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(accounting): stop delivery metrics from scanning retained receipts on every flush
The backlog query now reads only pending and parked deliveries, so its cost follows the backlog instead
of retained receipt history. Batch count and table size are measured once per check interval per pod,
and the storage and delayed-delivery alerts repeat at most once per interval instead of every flush
Fixes#50475
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): prove the storage query reruns only when its interval is due
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read the check interval on each flush so YAML environment_variables apply
The writer is built at proxy import, before the config's environment_variables load, so reading
the interval there ignored values set in config.yaml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): wait for router accounting instead of sleeping in user-config limit tests
Each request ended with a 0.3s sleep and hoped the router's usage writes had landed, so under CI load a
concurrent request could read counters mid-update. The tests now flush the logging worker and wait for
every task the requests scheduled, and a retired router's counter flush lands before the next router reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(completion): accept a tool_choice that forces an OpenAI hosted tool
litellm.completion rejected tool_choice={"type": "web_search"} before any request, while /v1/responses took the same choice. The chat validator now forwards a hosted tool type unchanged and a provider gate turns it into a 400 off openai, azure and litellm_proxy, where the choice would otherwise be dropped or mangled.
Fixes#50566
* fix(utils): keep a non-string tool_choice type a 400, not a TypeError
A list or dict in tool_choice["type"] is unhashable, so the hosted tool lookup needs the string check first. The bridge test that used file_search as an untranslatable choice now uses a type that really is unknown.
* refactor(completion): forward any Responses API tool_choice type instead of keeping a hosted tool allowlist
Chat completions now accepts any tool_choice dict with a string type it has no shape for and sends it as is on openai, azure and litellm_proxy, where OpenAI validates the type itself (a typo is still a 400, from OpenAI). The provider gate keeps the 400 everywhere else. The HOSTED_TOOL_CHOICE_TYPES list is gone, so a new OpenAI hosted tool needs no change here.
* Revert "refactor(completion): forward any Responses API tool_choice type instead of keeping a hosted tool allowlist"
This reverts commit 5a6d5a8e08.
* fix(utils): add tool_search to the hosted tool_choice types
OpenAI lists it among the supported tool_choice.type values (live error on 2026-11-08), alongside the ones already in the set.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): let one streaming request opt into usage.cost with x-litellm-include-cost-in-streaming-usage
The global include_cost_in_streaming_usage flag moved every stream off the
fast path. A request can now send x-litellm-include-cost-in-streaming-usage: true
to get usage.cost on its usage-carrying stream event (/v1/messages and the
Anthropic and OpenAI pass-throughs) while requests without the header keep the
fast path. The global flag keeps working unchanged.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the new stream cost header tests
* fix(types): keep reportUnknownArgumentType at its merge-base count
* fix(proxy): the stream cost header only accepts true
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hold MODEL_RECONCILE_LOCK around credential create, update and delete
The config reload snapshots the credential table, awaits config rows, then prunes and
upserts litellm.credential_list from that snapshot. A credential write landing inside
that window was undone in memory until the next reload. The three endpoints now take
the same lock the reload holds.
Fixes#50273
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): take MODEL_RECONCILE_LOCK before the credential update reads its row
Two PATCHes queued behind a reload read the same old row and the second
overwrote the first. The relay now keeps enough bytes to spot a split
after marker, and the new unit tests are fully typed.
* test(credentials): annotate the reconcile lock test locals with Final
* refactor(proxy): rename MODEL_RECONCILE_LOCK to DB_RECONCILE_LOCK
The lock gates every db-to-memory reconcile and now the credential writes too, so
the model-only name undersold what it protects.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
On an auto-router request the LLM span reported the name the caller asked
for (the router or an alias of it) as gen_ai.request.model (OTel v2) and
gen_ai.response.model (v1 and v2). Both now name the model group the router
picked, read from the request's routing_decision. In v2 the span name and the
vendor keys built from the request model (Langfuse, OpenInference, Langtrace,
Weave) follow. Requests that were not auto-routed are unchanged.
Both restart tests boot two 2-worker proxies but capped the case at a fixed
180s, below even one calibrated 2-worker readiness budget. On a loaded runner
each boot took 80-90s, so pytest-timeout fired mid-boot or mid-shutdown
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): surface mid-stream upstream errors that arrive without an event type
Bedrock's native Responses route can end a stream with a bare {"error": {...}}
chunk. It parsed as a typeless GenericEvent and crashed the iterator with
AttributeError, so clients saw that instead of the upstream error. Such chunks
are now read as error events, which raise the mapped upstream error and reach
the client as response.failed.
* refactor(responses): inline the untyped error chunk check to keep LIT001 at the base count
* fix(anthropic): preserve streamed tool argument fragments verbatim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): send the schema-less stream test the null properties form providers accept
* fix(anthropic): unwrap values when a schema-less stream sends its whole tool input in one piece
* fix(anthropic): unwrap only a one-piece values input and never emit Infinity in schema-less streams
* fix(anthropic): keep JSON mode answers verbatim when they cannot be written back as strict JSON
A tool input holding a number past a float or an integer past Python's digit limit now passes through as the model wrote it on the stream and non-stream paths, instead of coming back with Infinity or raising. One-piece values rows for false, null, a plain object and too-deep nesting pin the remaining unwrap branches.
* fix(anthropic): reject non-finite floats through pydantic instead of a raising parse hook
* fix(anthropic): keep split JSON mode answers whole when a field cannot be written back as strict JSON
* fix(anthropic): keep out-of-range numbers as the model wrote them in JSON mode answers
Decode tool input with parse hooks that keep a number past a float or an integer past the digit limit as its source text, and write it back unchanged, so held, wrapped and name and value split answers stay whole instead of falling back to raw blocks.
* fix(anthropic): keep numbers that round to zero as the model wrote them in JSON mode answers
* refactor(anthropic): encode JSON mode answers without a recursive helper
* fix(anthropic): decode deeply nested JSON mode answers without slow recursive validation
* fix(anthropic): type the decoded JSON mode answer as object and test non-object answers
json.loads output now goes through the existing TypeAdapter(object), which returns it unchanged, so no Any reaches the object guard. New stream and non-stream cases check that a list, string or number answer comes back as written.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve Responses SSE event framing
* style(proxy): format Responses stream terminator condition
* fix(proxy): validate Responses guardrail event names
* fix(proxy): keep Responses first-event errors and keepalive failures in Responses framing
* test(proxy): use controlled clocks for Responses keepalive failures
* fix(proxy): end late Responses failures with an error event the SDK raises on
Errors that arrive after keepalive pings, and guardrail error strings yielded
mid-stream, now end /v1/responses with a named `error` event carrying the
same error object chat sends, so the OpenAI SDK still raises APIError with a
clean message and provider_specific_fields. Background polling passes the
same framing and records an `error` event as a failed response. The first
data line of a frame is found by splitting on newlines only, so a U+2028 in
an error message no longer hides a first-event error.
* fix(proxy): match OpenAI's Responses error event and stream late passthrough blocks as block responses
* fix(proxy): keep the reason when a background Responses request fails
A background poll of a request a post-call guardrail blocked saved a
null code and message, because the guardrail's error event nests them.
A request whose first chunk was an error polled as completed with no
error. Both now poll failed with the same code and message a late
failure gets.
* test(proxy): type the first-chunk poll test parameters
* fix(proxy): keep first-chunk error detection to lone error frames and handle string poll errors
* fix(proxy): keep the HTTP status on late Responses errors and end Responses streams on error events
A late error body without a code now takes the response's status, an upstream error event counts as the stream's terminal event, and a Responses stream that fails before its first chunk ends with an error event instead of data: [DONE].
* test(proxy): type the first-chunk failure test's parameters
* fix(proxy): keep cursor chat framing and JSON-encode Responses hook errors
* fix(proxy): encode non-dict Mapping Responses chunks as JSON objects
* fix(proxy): encode mappings and models nested in Responses stream chunks as JSON objects
* test(proxy): freeze the hook-output fixture
* fix(proxy): keep masked mappings and odd keys safe in Responses stream frames
Only read-only dict views and pydantic models nested in a hook chunk or error event expand into JSON objects. Other mappings, such as httpx.Headers, keep their own string form so masked values stay masked. Keys JSON cannot take become strings, and a chunk JSON still cannot encode goes out as the plain data frame it was before.
* fix(proxy): never fail a Responses stream frame on a value JSON cannot encode
Only dicts and read-only dict views expand into JSON, at the top level too, so a header mapping yielded as a whole chunk keeps its masked string form. Any failure while encoding a chunk falls back to the plain data frame the base sent, and an error event that cannot encode still goes out with its code and message, so a late block never drops the connection.
* fix(proxy): keep what a nested value's repr hides out of Responses stream frames
* fix(proxy): print values in Responses stream frames the way each base path did
* fix(proxy): send string and number subclasses whose repr shows their value as that value in Responses stream frames
* fix(proxy): print container subclasses and SDK models in Responses stream frames the way base did
* fix(proxy): read hook container storage the way repr does and send error-event models as str()
* fix(proxy): frame streamed Responses reads and send unreadable hook values as base did
Track pre-framed and raw SSE frames so a late failure never follows a terminal event, accept CRLF and CR line breaks in a first named error, frame GET /v1/responses/{id}?stream=true as Responses events, print a whole hook proxy chunk once and fall back to its text, ignore a non-integer status_code in a hook error frame, read a late-block dict subclass through items() when that view differs from its storage, and keep the order and inline form base's repr printed for nested mappings
* fix(proxy): match base on Responses stream edge cases
Read a decimal-string or integral-float status_code as its integer again, so
"429" and 429.0 still map to rate_limit_exceeded. Print each hook value's repr
once and reuse it, so a repr that changes between calls cannot reach the
client. Track only pre-framed sequence numbers from 0 to 2**63-1, so a huge
one cannot break the failure event. Build nested JSON with one frame per level
again, which restores the nesting depth sent as JSON. Count a pre-framed frame
as terminal from its data type, and a named error frame only when its data
carries an error, the way the OpenAI SDKs read them.
* fix(proxy): print hook proxies once and keep base's chunk tracking
Read-only mappings in hook chunks now emit the text from their first print, hook
chunks are read without their error field, only a non-empty error object makes a
framed error event terminal, and framed sequence numbers stay below 2**63-1 while
hook chunks keep base's unbounded tracking
* fix(proxy): print each read-only mapping item once and poll framed error events as failed
A MappingProxyType in a hook chunk or error detail now expands from the
mapping it wraps, so each item is printed once, the way base printed it.
A value whose repr changes between calls stays masked, a value whose repr
works once is kept, and deep nesting keeps one stack frame per level
The background reader takes each frame's type from the same terminal rule
the stream uses, so an event: error frame with an error object polls
failed even when its data names another type or none
* fix(proxy): print self-holding hook values once and poll how the stream ended
A container that holds itself now goes out as the marker its repr prints, like {...}, [...] or mappingproxy({...}), and the dict behind a read-only mapping joins the cycle check, so no item is printed a second time. A dict whose item repr adds a key keeps going after the items already shown, the way dict's repr does, instead of printing the whole dict again
A whole read-only mapping chunk that cannot go out as JSON is printed once by the formatter, so a print that fails fails the stream as it did on the base
The background poll reads the event type from the block it parses, as on the base, and keeps the first error frame the stream treats as terminal anywhere in a chunk. That frame fails the poll only when no response.completed, response.failed or response.incomplete follows it
* fix(proxy): name hook chunk events from the sent JSON and keep polls on frames they cannot read
A mapping chunk now takes its event: line from the JSON it sends, so a masked type value never reaches that line, and a chunk the event model cannot read goes out as a plain data frame instead of failing the stream
A dict whose item repr deletes or swaps keys keeps printing every key not yet shown, and an error code past the integer digit limit reads as no status
The background poll skips a frame whose shape it cannot read, as base skipped frames it could not parse, stops reading a chunk at [DONE], and lets only the last terminal frame in a chunk decide its status
A Responses stream's first chunk answers as a JSON error only for codes 400 and up
* fix(proxy): read every frame of a polled chunk and name model chunks from the sent JSON
A background poll parsed only the first data frame of each chunk, so a terminal event behind an error or delta frame in the same chunk was lost, and the poll stored a status its own stream never ended with. It now reads every frame up to [DONE]
A terminal event whose response is a string or null kept no status and polled completed. Its status now comes from the event type, and final output items that are not objects are skipped, so the rest of the final output is still stored
Model chunks now take their event line from the JSON they send, as mapping chunks already do, so a serializer that drops type no longer sends event: response.completed over {}
* fix(proxy): read polled chunks as SSE events and keep draining past a mid-chunk [DONE]
A background poll split a chunk into data lines, so one event with two data lines parsed as two payloads, and a frame with an integer past the digit limit raised a ValueError the parse guard did not catch, which failed the poll and dropped its spend row. The poll now joins each event's data lines the way an SSE reader does and skips any frame it cannot parse
A [DONE] behind another frame in one chunk ended the poll and left the stream unbilled. Only a chunk that opens with [DONE] ends the poll now, as on main, and the poll stops reading a chunk at its first terminal event, so two terminal events in one chunk poll the first one as main did
* fix(proxy): JSON-encode only plain hook chunks on Responses streams
A hook chunk whose whole tree is builtin JSON types streams as its named
event. Any other chunk keeps the base data frame, so nothing a custom
repr hides reaches the wire. Error event details encode the way the chat
error frame does.
* fix(proxy): skip containers already visited when checking hook chunks for JSON
* fix(responses): keep streaming past an event with no type
The Responses stream iterator read event.type directly when it noted a
yielded event, so an event whose JSON had no type, like a custom hook's
dict that the proxy now sends as JSON, raised AttributeError and ended
the stream. Read it with getattr, as the iterator's other checks do.
* fix(proxy): keep hook dicts with no type on the base Responses frame
A hook dict with no type is not a Responses event, so it gets no event
name either way. Sending it as JSON only made litellm SDKs that read
type on every event stop the stream. Keep such dicts on the frame the
base sends, so every SDK version reads them as before.
* fix(responses): keep a background poll's first terminal status and take a first-chunk error code from its HTTP status
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): forward Codex agent messages that carry encrypted content intact
Codex multi-agent v2 sends worker messages as agent_message items whose
encrypted_content block only the model server can decrypt. Bedrock Mantle and
Bedrock Runtime used to 400 on agent_message, so LiteLLM rewrote every one into
assistant text, pasting the ciphertext into the prompt and dropping the sender
and recipient. Bedrock now decrypts these items itself, so the rewrite left the
recipient model reading an opaque token instead of its task.
Agent messages with an encrypted_content block now go to Bedrock untouched on
both endpoints. The Bedrock configs declare supports_encrypted_agent_messages,
and the shared normalizer keys off that flag so the two cannot drift. Mantle's
/v1/responses route for open-weight models still rejects agent_message in any
form, so it keeps the old rewrite. Plaintext-only agent messages keep the
rewrite on every route.
* fix(bedrock): only keep worker messages locked for OpenAI GPT models
Gemma and Grok on Mantle's OpenAI route never lock Codex worker messages, so
Codex puts plain text in the locked slot and Bedrock rejected it with
invalid_encrypted_content once the route alone decided to forward it intact.
Decide from the model instead, and restore the full-value rewrite assertion.
* fix(otel): seed request metadata baggage for team-only OTel v2 destinations
A team newrelic (or other preset) destination exported spans without the
operator-allowlisted litellm.metadata.* keys unless a proxy-wide otel callback
was listed, because only the OTel logger's async_pre_call_hook seeded
requester_metadata into Baggage and that hook runs only for loggers in
litellm.callbacks. ProxyLogging.pre_call_hook now re-seeds identity with the
request data once requester_metadata exists.
Resolves LIT-9783
* fix(otel): seed only request metadata from the proxy pre-call hook
The pre-call re-seed also passed the request model, which by then can be an
alias target, mcp-tool-call or an internal embedding model, so it rewrote
gen_ai.request.model on the server span for V2 deployments without otel listed.
Seed only the request metadata and keep the model auth seeded. Adds a unit test
for the pre-call seed and the guardrails_only skip, drops the explanatory
comments, and requires one server span plus the model span in the integration
test.
* test(otel): annotate new test variables Final and build test config without mutation
Addresses review: the new unit and integration tests now follow AGENTS.md
(LIT010 Final on every variable, no in-place mutation, read-only parameter
types, one for-clause per comprehension).
* test(otel): cover key-level callbacks and assert tenant spans carry no credentials
Parametrizes the tenant metadata test over team and key callbacks, and adds a
test that sends credential headers and a planted New Relic key with an
allowlist naming credential headers, then asserts no exported span carries the
master key, the virtual key, the team's New Relic key or any credential header
value, and that every export used only the team's own New Relic key.
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
* feat(proxy): serve Codex its own release's model catalog, fetched from GitHub
When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that
release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the
fetch fails, times out after 2 seconds, does not validate, the version has no tag, or
LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight
most recent versions are kept. The integration tests ask as the bundled release so their expectations
hold whether or not the proxy can reach GitHub
* fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions
Failed versions now live in their own cache so they never push out a fetched catalog, requests for a
version already being fetched share that fetch, each worker starts at most six fetches a minute and
serves the bundled catalog past that, and the version check only accepts ASCII digits
* fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap
Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine
failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the
codebase
* fix(bedrock): run realtime guardrails on Nova Sonic sessions
* fix(bedrock): quote the spoken violation message and skip it after a blocked Nova Sonic tool output
* fix(bedrock): screen every message role, end Nova Sonic sessions on blocked transcripts, drop unparseable frames
* fix(realtime): apply guardrail masks, keep block reasons out of spoken notices, fail closed on guardrail crashes
* fix(realtime): screen joined text parts, keep the documented notice, honor key opt-outs
Text parts of one message are joined before screening again, so a phrase split
across parts is still blocked, and a mask collapses them into one part. The
spoken notice is realtime_violation_message or the block reason, quoted only on
Nova Sonic. Client-written output_text and text parts are screened. Key and team
guardrail opt-outs reach regular and translation realtime sessions. A non-string
upstream transcript skips screening as before.
* fix(realtime): honor guardrail role scope, clear stale response.create drops, screen split keywords, report Nova guardrail outages
* fix(realtime): keep every per-part mask when a joined text masks differently
* fix(realtime): screen one joined text per item and pass the session's tenant context to guardrails
Multi-part items reach each guardrail once as the space-joined text, as on main, and only an item a
guardrail rewrites gets its parts screened again to keep per-part masks. Guardrails now receive the
session request data with its metadata, so key and team policies apply, and a ModifyResponseException
block is reported as a block. Nova scopes every message item as the user turn it sends to Bedrock.
* fix(realtime): catch phrases split mid-word across text parts without re-screening single-part items
* fix(realtime): screen each joined text in its own guardrail call and keep a Nova opening message for the retry while it is screened
* fix(realtime): screen text parts in a loop so large messages keep their masks
* fix(realtime): fail closed on a non-string transcript under a guardrail and give each turn its own guardrail metadata
* fix(realtime): fail a non-string transcript only when a guardrail would screen the user's turn
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): pin /v1/responses requests for prompt caching affinity
* fix(router): pin previous_response_id turns and keep native tools in the Responses pin key
A Responses turn that continues a stored response now goes to the deployment
that answered it, unless that deployment already failed this request. Native
Responses tools are hashed into the pin key as sent instead of converted, so
two requests with different native tools no longer share a pin and a native
request no longer logs a false dropped-tool warning. The pin keeps its
cacheable deployment ids in a sorted tuple so loggers can serialize request
metadata.
* fix(router): leave a Responses request with a malformed tool to the provider
* fix(router): keep the Responses prompt cache pin out of logged metadata
The pre-call check now keeps the pin in a router-owned in-flight store and
puts only a random token in litellm_metadata, so loggers that ship metadata
never see the routing key or deployment ids. The store is shared across check
instances, so turning prompt_caching off and on at runtime keeps pinning, and
a retry drops its previous attempt's token. The skipped-input debug log now
names only the error type.
* fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries
The prompt caching check no longer narrows the candidates inside the callback
filters. It names the deployment it prefers, and the router core applies that
preference at the pinned deployment step, after tag, region, pre-call and plugin
filters, only when the deployment is still a candidate. A pin can no longer
send a request to a deployment its tags do not allow.
A retry skips the pin when the pinned deployment was already tried in this
request, read from the retry breadcrumbs in both metadata buckets and the
excluded and retry-skipped ids. When the retry count outruns the breadcrumbs
the check stands down, so a pinned deployment that keeps failing is not
retried on every attempt.
* test(router): list the prompt caching check as an in-memory cache caller
The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers
* fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records
* test(router): lift test cooldowns through the cooldown store's evict_local
* fix(router): key Responses tools like their chat twin and drop the pin token after success
A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict.
* test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
