* fix: enforce disable_custom_api_keys from general_settings
The gate in _check_custom_key_allowed read the persisted UI settings row
through get_ui_settings_cached, which had two consequences.
A config-file general_settings.disable_custom_api_keys was never enforced,
because the gate only ever looked at the stored ui_settings row. POST
/key/generate with a custom key value returned 200 even with the flag set
to true in config.yaml.
The read went through a DualCache with a 600s TTL, which is per worker
without Redis, and only the worker that served PATCH /update/ui_settings
refreshed it. A gate flipped through the UI was then a coin flip across
workers for up to ten minutes.
Both go away by routing the flag the way the other runtime UI flags are
already routed. Adding it to _RUNTIME_GENERAL_SETTINGS_FLAGS and to the
settings rules' _UI_SETTINGS_FIELDS makes SettingsStore resolve it from the
ui_settings row with the config file winning, and every pod re-reads it on
its own settings sync rather than holding a private cached copy. The two
lists have to stay in step: a flag in one and not the other resolves
against the wrong stored row and silently never reaches a reader, so there
is a test for that invariant.
Writes to the ui_settings table did not publish on the config-sync channel,
so other pods only discovered a change on their next periodic reload. Adding
litellm_uisettings to _CONFIG_SYNCED_TABLE_NAMES puts it on the same pubsub
path model and SSO config writes already use, which cuts cross-pod
propagation from tens of seconds to a few.
The value reaching the gate is run through coerce_bool first. Resolution
hands back the raw YAML value, so a quoted "true" in config.yaml is a str
and the old `is True` check let custom keys straight through.
* test: assert both directions of the coerced config value
* test: assert the runtime flags read back instead of inspecting the registry
Python 3.13+ strips the common indentation of docstrings at compile time and 3.12
keeps it, and the 429 error description in ERROR_RESPONSES came straight from
RateLimitError.__doc__, so regenerating litellm/proxy/_lazy_openapi_snapshot.json
on a 3.13+ venv produced a one-line diff that the check-ui-api-types job (Python
3.12) rejected. Run the docstring through inspect.cleandoc before it lands in the
spec, regenerate the snapshot once, and pin the behavior with a test
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): add admin-only /debug/report sharing the bug report environment fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add verbose=true to /debug/report listing every config key with typed values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): inject auth into /debug/report through Annotated to keep the B008 budget flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the verbose config walk, drop nested-list recursion from the safe renderer, regenerate schema.d.ts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pass-through and mcp header maps plus operator-named budget maps in verbose /debug/report, single-exit scalar renderers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after dropping the verbose query from /debug/report
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(arize): per-team success and error sampling rates for the Arize AX callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(arize): fail open on invalid sampling rates and type the new sampling code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(arize): type the sampling test helpers and parametrized fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): opt-in litellm_call_id in JSON error bodies
Add general_settings.include_call_id_in_error_body. When true, the value
already on the x-litellm-call-id response header is copied into JSON error
bodies: as error.litellm_call_id on the OpenAI-shaped routes, /v1/messages,
and streaming first-chunk errors, and as a top-level litellm_call_id on
pass-through routes. Off by default, so error bodies stay byte-identical
unless an admin opts in
* chore(proxy): drop helper docstring and restore lazy OpenAPI snapshot
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A string such as "*" or "..." has no base64 characters, so it decoded to no bytes and read as an empty plaintext under any key. The migration would have counted it and overwritten it with a ciphertext of the empty string. Also read from the writer database instead of a read replica, report a database error during the migration instead of crashing the boot, skip columns the connected schema lacks across every schema on the search path, cap the JSON walk depth for the recursion detector, and move the boot wiring into one tested function.
Rotating through POST /key/regenerate needs a running proxy, which a refused boot does not have. The refusal now counts the stored values that decrypt under the unsafe key. When there are none it only asks for a new key. When there are some it also asks for LITELLM_MIGRATE_FROM_MASTER_KEY, and the next boot with a safe key re-encrypts them and logs that the variable can be deleted. Leaving the variable set afterwards is a no-op with one notice.
Shipped proxy configs now read general_settings.master_key from
os.environ/LITELLM_MASTER_KEY, the .env examples ship a blank value with
the openssl generate command above it, and READMEs, the missing env vars
page and Admin UI code snippets show a generate command or the
<your-master-key> placeholder instead of the literal sk-1234
The two CircleCI docker runs that mount proxy_server_config.yaml and
oai_misc_config.yaml now pass LITELLM_MASTER_KEY so their runtime key is
unchanged
PR #40243 started carrying the upstream error body on InternalServerError so the Responses response.failed event can report the provider's code and message, and openai's APIError.__init__ took the body's type along with it. The proxy then answered an OpenAI-compatible upstream 500 with type server_error while a 502 and a 503 kept internal_server_error, and the integration contract in test_observed_routing.py went red. Pin the type the way RateLimitError pins throttling_error, keeping the body.
A temporary-only member update no longer clones the team default budget into the private row. The row stores just the temp pair and auth, spend admission and reservation add the active increase to the current shared default, so a later lowering of the default reaches members with an active grant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Forking a shared budget row rebuilt budget_reset_at from the duration, so editing
an unrelated limit restarted the member's window while their spend carried over:
a tpm bump quietly handed them a fresh period. The fork now inherits the source
row's deadline, and only recomputes when the patch actually sets budget_duration.
The bulk member budget route now writes one audit entry per call, a team-scoped
'updated' row carrying every written member's limits before and after, matching
what /team/member_add already records for membership changes. It honors the
litellm-changed-by header like the other audited team routes.
Cuts the new docstrings back to the parts a reader cannot get from the code,
and fixes a stale reference: the walk this one is modelled on is
_reset_windows_for, not _reset_windows_for_source.
The truncation test reached in and replaced MockTable.find_many. The mock takes
a scheduled read failure instead, the way it already takes canned rows.
UserApiKeyCache's batch delete ran the two partitions in sequence, so a Redis
failure on the hashed token partition returned before the ordinary management
keys were touched. Both partitions are attempted now and the first failure is
re-raised for the caller to report.
The customer walk kept its position in two locals it reassigned each page. It
now mirrors the window walk in the same file: a page helper returns where the
walk goes next, and the driver rebinds one value.
Greptile review follow-ups on the paged end-user cache invalidation.
UserApiKeyCache keeps hashed token keys in a second in-memory partition, and
routes delete_cache / async_delete_cache there. It inherited the new batch
delete unchanged, so a budget cascade cleared the main partition and left the
key object sitting on its pre-reset spend. Override it the way
async_set_cache_pipeline already partitions its entries.
The spend counters and the management cache shared one exception handler, so a
Redis failure on the counters returned before the management cache was touched
at all. Each cache gets its own await and its own handler now.
A failed page read returned the same empty tuple that ends the walk normally,
so a truncated pass was reported as a complete one. The window is advanced by
then and no later tick comes back for the customers past that page, so the walk
now says it was cut short and the service log carries it.
The budget-tier reset read every customer linked to an expiring tier into
one result set before the write, then invalidated their caches one key at
a time. Both of those scale with the customer count, so a large enough
deployment can OOM the proxy pod on the read, and the tail of the
population sits on a stale spend counter while the per-key invalidations
drain
PR #40639 moved the reset write itself to a link-based UPDATE, so that
pre-commit read no longer feeds the write. It only fed cache invalidation
and the service-logging counts, which means it can move after the commit.
This replaces it with a keyset walk over litellm_endusertable ordered by
user_id, taking RESET_BUDGET_JOB_BATCH_SIZE rows per page, the same shape
_reset_windows_for_source already uses, with no per-run page cap for the
same reason that walk has none: the cursor cannot survive the run, so a
cap would restart at the first customer on every tick and never reach the
tail
Each page's counter and cache keys now go out as one batched delete
through a new DualCache.async_delete_cache_keys, which drops the
in-memory entries and chunks the Redis DELETE at
DEFAULT_MAX_REDIS_BATCH_CACHE_SIZE
num_endusers_found and num_endusers_updated now report the customers
whose caches were invalidated after the commit rather than the rows read
before it, so both read 0 when the cascade write fails
Adds a persistent total_spend column to LiteLLM_VerificationToken and LiteLLM_DeletedVerificationToken, incremented in the same write as spend and left alone by budget resets. Surfaces it on /key/info, /key/list and the Admin UI Virtual Keys table and key detail view
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Already shaped ProxyException and HTTPException errors passing through the moderations, audio speech, Anthropic Messages, and handle_exception_on_proxy paths now answer with the x-litellm-call-id header the route logged under, without overwriting a header the exception was raised with. The GET /v1/batches failure hook receives the resolved request data so the spend log request_id matches the response header and the error log
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Project-scoped keys never wrote spend to LiteLLM_ProjectTable, so
/project/info stayed at 0 and project budgets could not block. Wire the
PROJECT entity through the spend queue, redis buffer, and db writer,
reserve and increment a spend:project counter, reseed it from the
project row, reset project spend in the budget cascade, and read the
live counter in the project max budget check. Team member budgets keep
gating project-scoped keys alongside the project budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
