* fix(proxy): bound tool and guardrail index create_many by the spend-log statement budgets
One flush drains up to MAX_LOGS_PER_INTERVAL source transactions or logs, but a
transaction fans out to one LiteLLM_SpendLogToolIndex row per tool and a log
to one LiteLLM_SpendLogGuardrailIndex row per guardrail, so the index
create_many payload was unbounded. Both index writes now go through
spend_log_write_batches(SPEND_LOG_WRITE_BATCH_MAX_BYTES, SPEND_LOG_WRITE_BATCH_MAX_ROWS).
The tool index write moves out of the rollup batch_() so the split reduces
the query-engine payload; replayed index rows are no-ops under
skip_duplicates, and the daily rollup upserts stay in one transaction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): pin the row budget in the index fan-out tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Router.log_retry used to copy the failed attempt's kwargs and metadata into
metadata.previous_models. Nothing downstream read those copies, but they carried
client credentials into spend logs and grew the payload on every retry. Each
attempt now leaves a flat record (model group, deployment id, exception type and
string, attempt number), which drops RETRY_BREADCRUMB_EXCLUDED_KWARGS and the
per-retry credential masking.
num_retries_per_request was enforced from len(previous_models), which only
looked at the metadata bucket and never exceeded four records. The sync and
async client wrappers and the Rust lifecycle guard now read attempted_retries
from whichever metadata bucket the call carries.
Resolves LIT-7505
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The bridge probe asked `responses_api_bridge_check` with the summary read straight off
the Responses object, but `litellm.completion` reads it from `optional_params` via
`peek_reasoning_summary_aliases`, which the bridged request never populated. So gpt-5,
gpt-5.1 and azure/gpt-5 answered "bridging" to the probe and "not bridging" for real,
and the object still landed on Chat Completions, which only takes a string
`reasoning_effort` is now always the effort string, and `summary` rides the
`reasoning_summary` alias that main.py already reassembles into `{effort, summary}` on
the bridged path. The alias is emitted only when the probe says the model bridges, so
no chat provider ever sees it, and the probe is now asked with the exact params this
transform emits
The Responses API takes reasoning as an object, {effort, summary}. Chat
Completions takes reasoning_effort as a string enum and has no equivalent of
summary, but the completion bridge forwarded the whole object whenever summary
was set, which agentic clients set on every request.
Bedrock Converse guards its mapping with isinstance(value, str) and has no else
branch, so the object fell through, thinking was never enabled, and the caller
was billed for a non-thinking turn with nothing in the response to explain it.
The object is still forwarded for the one caller that can consume it: a model
whose cost-map mode is responses, which litellm.completion bridges back onto the
Responses API and reassembles {effort, summary} there. That decision is delegated
to responses_api_bridge_check, the same check litellm.completion runs, rather
than a second copy of the rule that could drift from it. An object carrying no
effort now yields no reasoning_effort at all.
Multi-IdP deployments can now set virtual_key_claim_field and
unregistered_jwt_client_behavior on a JWTIssuerConfig entry. Tokens from
that issuer use the issuer-specific claim path and no-match policy for the
virtual key mapping lookup; issuers that omit them keep the global values.
The auth flow now enters the mapping lookup when any issuer configures the
field, not only when the global field is set.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/model/info fills a deployment's missing pricing in from the model cost map so the
Admin UI has a rate to display. Clients echo that whole model_info blob back on save,
and update_db_model merged it into the row, so editing an unrelated setting turned
that day's catalog price into a real per-deployment override. After that the
deployment ignored the cost map and Reload Price Data could no longer move it,
because the reload replays each deployment's stored pricing over the fresh catalog.
Drop the derived pricing from incoming model_info on the two write paths. The
drop-set is read off the same objects the read path uses, CustomPricingLiteLLMParams
plus the tiered *_above_N_tokens pattern that get_model_info passes through and no
model declares, so it cannot drift as new rates are added. output_vector_size is
exempt: it lives on the pricing model but is an embedding dimension, not a rate.
A deployment's own pricing still rides litellm_params, which is untouched, as is the
explicit-null clear, which reads the incoming model rather than the filtered dict.
The filter sits in the endpoint bodies rather than _add_model_to_db, which master-key
rotation reuses to re-serialize every stored deployment.
Codex hides catalog entries whose supported_in_api is false when it runs
with an API key, so a stock entry the proxy serves now carries
supported_in_api true alongside its list visibility.
`lite codex` now asks the installed Codex for its own model list through
`codex debug models` before writing the catalog. A proxy model whose id
matches a stock Codex slug keeps that Codex's entry (reasoning levels,
base instructions, context window and the rest) and only its picker
position, visibility and upgrade nudge come from the proxy. Unknown slugs
still get the plain entry built from the bundled base instructions. The
catalog directory is created before the stock call so a fresh CODEX_HOME
does not make Codex refuse to run
Adds general_settings.custom_key_policy, one coroutine that receives the
operation ("generate", "update", "regenerate"), the existing key row, the
effective row as it will be written, and the raw request, and can deny with a
403. It runs after the request has been normalized and before the first DB
write on /key/generate, /key/service-account/generate, /key/update,
/key/bulk_update, /team/key/bulk_update and /key/{key}/regenerate. The two
legacy hooks keep running unchanged on the raw request.
* fix(proxy): attribute gate-rejected requests to their endpoint in cache analytics
Requests rejected before dispatch (bad key, blocked key, budget, rate limit, malformed body) were spend-logged with an empty call_type because the synthesized logging object never reached the failure lifter. The caching dashboard rolled all of them, plus failed calls on info routes such as /model/info, into one Unknown group.
Resolve call_type from the matched route first, falling back to body shape, and keep the synthesized logging object on request_data so the lifter sees it. Log bare auth exceptions with the 401 ProxyException the client gets so error_code is never empty. Exclude info routes from the cache analytics groups and error breakdown. The dashboard explains the Unknown group when older rows still produce one.
Resolves LIT-5884
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the raw auth exception for failure callbacks
Record the client-facing status in the spend log through a separate client_exception argument so custom failure callbacks still receive the exception auth raised.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the route for multi-operation endpoints and exclude info routes from cache filter options
Routes such as /v1/files map to several operations (create, list) and the
method is not available in the failure hook, so a rejected request there is
filed under its route instead of the first mapped call type. The key alias and
model filter-option queries now apply the same info-route exclusion as the
groups and error breakdown, so every offered filter value returns data. The
info-route exclusion and Unknown grouping are now covered against a real
Postgres in tests/proxy_behavior/spend/test_cache_activity.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop client_exception, the spend log row never used it
The DB spend row for a gate rejection is written by _ProxyDBLogger from the
original exception, so the status-bearing copy only reached the in-memory
logging payload. Live runs at the tip still recorded bare auth exceptions as
Unknown/Exception, the same as the base branch. Removing the plumbing keeps
this PR to endpoint attribution and the info-route exclusion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Codex 0.130 and 0.145 require supports_reasoning_summaries and
supports_parallel_tool_calls on every catalog entry, so a catalog written
for 0.154 made those releases exit at startup with a parse error. Every
field some release since 0.105.0 deserializes without a default is now
written, with Codex's own fallback values, and the catalog is read back
once through the installed binary (`codex debug models`) before launch.
A Codex that rejects it, or one older than 0.130 with no such command,
gets the skip notice and launches on its built-in catalog instead.
The provider_config path skipped run_realtime_guardrails for transcription
sessions to avoid sending response.create, which also dropped every
realtime_input_transcription guardrail: no violation error reached the
client and on_violation / end_session_after_n_fails never fired. Run the
guardrail for every completed transcript and only suppress response.create
when the session has no assistant turn.
Every proxy and router cost lookup goes through get_model_info, which copies
cost map keys explicitly, so the new audio cache-read branch always fell back
to the text cache-read rate there. Copy the key so models whose audio
cache-read rate differs from the text one bill cached audio correctly.
When a slow backend times out many logging callbacks at once, each timeout
hit verbose_logger.exception in _process_log_task and produced a full ERROR
traceback, clustered milliseconds apart. Count timeouts instead and arm a
debounced flush that logs one WARNING with the burst count, callback name,
timeout and cumulative total. Real programming errors keep their traceback.
