The persisted UI settings were read once at startup, so a proxy admin
flipping a runtime flag through PATCH /update/ui_settings only changed the
pod that served the request. Every other pod kept serving the old value
until it restarted.
add_deployment, the reload the scheduler runs every 30s, now re-reads the
row and applies the runtime flags before it takes the model reconcile lock,
so a change made through one pod reaches the rest within one reload
interval. The startup hook and the two settings endpoints share that helper
instead of each repeating the flag copy.
Claude-Session: https://claude.ai/code/session_018PUCupsaarVLJy4iDFx256
Team admins could never reach POST /team/update: the route gate answered
401 before the handler's team-admin branch ran. This moves /team/update
into the self-managed routes, resolves proxy admin, org admin or team
admin inside the handler, and filters team admins through a new
proxy-wide UI setting, team_admin_editable_team_fields. The setting is
an allow-list of team fields. Empty means team admins cannot edit team
settings and get a 403 pointing at the proxy admin, and changing a field
outside the list fails 403 naming that field. Only values that differ
from what is stored count, since the dashboard resends the whole form.
The registry of fields the setting accepts ships empty on purpose. Each
field lands in its own follow-up PR with its value diff and dashboard
wiring. The Admin UI gains a "Team admin editable fields" section under
Settings > UI and a toast on the team page while editing is disabled.
Refs LIT-5722
Claude-Session: https://claude.ai/code/session_01A6SkwJdfZUmkzfUkrEkqX8
Keeps the original CustomStreamWrapper so response headers and the
correlation-context cleanup in __del__ are untouched when a deployment
hook rewrites the converted response. Covers the early-return branches
for real provider streams and unmapped call types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The token exchange mint keeps reading the user row from the database, since JWT auth caches the user it creates before adding it to the JWT's team and a mint off that cached row refused the first exchange for a new user. Introspection and the refresh revalidation go back to the cache read, so a resource server calling /introspect per request pays no database read.
The filter kept an extra as its original object whenever the plain and
scrubbed safe_dumps renderings matched, but safe_dumps skips what it cannot
render (non-string dict keys, anything past its depth limit, fields a repr
hides), so a secret in those places rode the untouched object past the
litellm_redacted stamp. The filter now walks JSON-native shapes itself
(strings, scalars, string-keyed dicts, lists and tuples) and keeps the
original only when every value it holds comes back unchanged from the
redactor; anything else is handed to safe_dumps and the record carries the
scrubbed JSON shape the formatter would have rendered
Each route now has litellm/rust_bridge/<route>/{entrypoints,callbacks}.py and a
public dispatch module (litellm/chat_completions/dispatch.py,
litellm/responses/dispatch.py, litellm/messages/dispatch.py) that binds the
public call to the legacy Python signature, builds a frozen request, and asks
the runtime to pick Rust or Python from the catalog. The legacy implementations
stay in litellm/main.py, litellm/responses/main.py and the anthropic messages
handler, and litellm/__init__.py re-exports the dispatch names over them the
same way it already does for ocr
The per-handler shims in rust_bridge/chat_completions/native.py and
rust_bridge/messages/native.py are removed along with their call sites in the
anthropic and bedrock chat handlers and the http handler. The exception
mapping that every callbacks module repeated moves to rust_bridge/failures.py
and the signature binding helpers to rust_bridge/public_call.py
Builds 232 and 233 held the Bedrock hit rate at 9% with the Claude Code
driver already sending byte-identical requests and headers, so something
between the proxy's ingress and the upstream still moves per build and
the flat key cannot say what.
Emit a digest per key component next to the counters: the test id, the
method, the URL, each keyed header, the whole body, and one digest per
top-level JSON body field. Values are digested, so no payload or
credential reaches the artifact. Diffing two builds' artifacts names the
field that moved.
Diagnostic, to be removed once it has answered.
A pre-call deployment hook can turn a requested stream into a non-streaming provider call and the result is wrapped back into a fake stream. The async client wrapper treated that wrapper like a caller-requested stream and returned before async_post_call_success_deployment_hook, so SDK callers lost post-call deployment processing (including CustomGuardrail post_call enforcement) on converted streams. Run the hook on the complete ModelResponse behind the wrapper and rewrap a modified response so it reaches the emitted chunks.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The selector read raw cost-map dicts as `Mapping[str, Any]`. It now validates
each together_ai entry into a frozen Pydantic model and takes the two
capabilities as keyword booleans, so nothing in the helper is coarsely typed
or stringly addressed.
CI runs these lanes as `python -m pytest` from the repo root, so the root is
already on sys.path and `tests._live_test_helpers` imports without help. The
insert only tripped the TQ003 test-quality budget.
`together_ai/openai/gpt-oss-20b` was hardcoded in two live tests and is no
longer served, so both failed on a vendor catalog change rather than on
anything litellm did.
Both call sites now resolve the cheapest non-deprecated together_ai chat
entry at runtime, filtered on the capabilities the tests actually exercise,
mirroring what tests/e2e/llm_translation/test_together_ai_e2e.py already
does. The selector lives in tests/_live_test_helpers.py so both lanes share
one implementation.
A non-string extra was scrubbed by a safe_dumps round trip, which handed
every user-attached handler a JSON-shaped copy even when nothing in it
was redacted. The record now keeps the original object whenever the plain
and the scrubbed renderings compare equal, so only an extra that carried
a secret comes back as its scrubbed shape
A budget bypass that ships off by default stays open for every deployment
that does not know to look for the flag, so `enforce_fallback_budget` now
defaults to true and `general_settings.enforce_fallback_budget: false` is
the opt-out for anyone who wants the old unguarded behaviour back.
BREAKING CHANGE: a paid fallback target is now refused for callers who are
over their key or user `max_budget`. Deployments relying on fallbacks to
keep serving over-budget callers must set enforce_fallback_budget: false.
JWT auth caches the user it creates before it adds that user to the JWT's
team, and adding a team member never evicts the cached user row, so the mint
read a row with no teams and refused the very first token exchange for a
never-seen user as not a member. The loader now reads the row from the
database and leaves the fresh row in the cache for the requests the
credential makes next
Delete litellm/ocr/input.py and the native _ocr_file_document, _ocr_upload_document
and _ocr_mime_type helpers. File documents now project to a typed OcrDocumentInput
and the core lifecycle reads local paths, encodes bytes and asks the host to read
file-like objects through a ReadDocument operation before the provider request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
tests/e2e/llm_translation/test_together_ai_e2e.py registers its model with the
full registry key, so a slashed together_ai/<vendor>/<model> goes through the
prefix strip on every e2e run and an over-strip would fail against the real
API. The unit assertion was a second copy of that. The roles check stays, since
nothing in e2e exercises it.
The failure hook received data without the resolved id when body parsing or
add_litellm_data_to_request raised, so proxy-only spend logging minted a fresh id
that did not match the error log or the x-litellm-call-id header. The id is now
part of the request data from the start and merged over the parsed body, which
also removes the post-hoc in-place assignment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The generic API logger batches whatever is queued when it flushes, so records
from tests in other files in the same job land in the same request. Two tests
assumed otherwise: one read actual_request[0], the other counted NDJSON lines,
and both broke whenever another file logged first. Select by the messages each
test sent instead, which keeps the format assertions and stops the order from
deciding the outcome.
The exchange refused the very user JWT auth upserts (its row has no
user_role) as "no longer active". The credential now carries the role the
proxy already enforces for that user on every request, internal_user when
the row has none, the same rule _get_user_role applies on the data plane.
A rejected subject_token no longer echoes JWT auth's wording on the public
/token endpoint: the response is a fixed invalid_request and the reason
goes to the proxy log, since that wording can name the JWKS URL or relay
the IdP's reply.
The exchange grant is listed on /register, /.well-known/litellm-cli-auth,
and the aggregate authorization-server metadata only when JWT auth is on,
backed by a database, and licensed, so a client never selects a grant the
gateway would then refuse.
Build 232 took two compat cells red with a FileNotFoundError renaming
`.claude.json.197` onto `.claude.json`. `run_claude_models_parallel`
drives several models from one process, so a pid-suffixed staged name is
shared between threads: one thread renamed the file the other was still
writing, and the loser died on a path that no longer existed.
mkstemp in the same directory gives a name that is unique per thread as
well as per process, and the rename stays atomic.
