Interrupted Anthropic streams that die before message_delta were billed at
the message_start placeholder (any value above 1 was trusted) or at 0 when
the partial response was reasoning-only, because the token_counter fallback
only looked at visible text. Reset the placeholder whenever no finish_reason
or second usage event arrived, fold the already-counted reasoning tokens into
the fallback estimate, and drop the stale completion_tokens_details so cost
is computed from the recovered count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_PROXY_MaxBudgetLimiter re-checked spend:user:{id} against user_max_budget in
async_pre_call_hook without the zero-cost model exemption that
_user_max_budget_check applies in auth, so free models were rejected with
"Max budget limit reached." once a user was over budget. Auth already owns
this check, so the hook is deleted rather than taught the exemption again
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
update_team loaded the org without its budget row, so the org max_budget,
tpm_limit and rpm_limit checks silently passed. It now loads the budget the
same way /team/new does
tpm_limit is the first field in the team-admin allow-list registry. The team
settings tab gives a team admin a form with only the enabled fields and sends
only those on save, and UI Settings labels the checkbox the same way
The dashboard gated the team settings form on a role it guessed from the
is_* props, the members list and the org list. The org list is premium
gated and empty while loading, so a team admin who is also an org admin
was told team admins cannot edit, although /team/update accepts them as
an org admin
/team/info now returns caller_edit_access, resolved by the same helper
/team/update uses, and TeamInfo keys the form and the toast off that
field. The org list is only read for the organization dropdown now, and
general_settings is read through one validated accessor in both handlers
assert_never landed in typing in 3.11, so importing it from typing broke the
3.10 import smoke job and the py310 typing gate. Take it from typing_extensions
like the rest of the proxy does.
Also drop /team/update from test_neighbouring_team_routes_stay_closed. The route
is self-managed now, so the coarse gate admits the caller and update_team decides,
which the sibling test in the same file already asserts.
Claude-Session: https://claude.ai/code/session_018PUCupsaarVLJy4iDFx256
The persisted UI settings were read once at startup, so a proxy admin
flipping a runtime flag through PATCH /update/ui_settings only changed the
pod that served the request. Every other pod kept serving the old value
until it restarted.
add_deployment, the reload the scheduler runs every 30s, now re-reads the
row and applies the runtime flags before it takes the model reconcile lock,
so a change made through one pod reaches the rest within one reload
interval. The startup hook and the two settings endpoints share that helper
instead of each repeating the flag copy.
Claude-Session: https://claude.ai/code/session_018PUCupsaarVLJy4iDFx256
Team admins could never reach POST /team/update: the route gate answered
401 before the handler's team-admin branch ran. This moves /team/update
into the self-managed routes, resolves proxy admin, org admin or team
admin inside the handler, and filters team admins through a new
proxy-wide UI setting, team_admin_editable_team_fields. The setting is
an allow-list of team fields. Empty means team admins cannot edit team
settings and get a 403 pointing at the proxy admin, and changing a field
outside the list fails 403 naming that field. Only values that differ
from what is stored count, since the dashboard resends the whole form.
The registry of fields the setting accepts ships empty on purpose. Each
field lands in its own follow-up PR with its value diff and dashboard
wiring. The Admin UI gains a "Team admin editable fields" section under
Settings > UI and a toast on the team page while editing is disabled.
Refs LIT-5722
Claude-Session: https://claude.ai/code/session_01A6SkwJdfZUmkzfUkrEkqX8
Keeps the original CustomStreamWrapper so response headers and the
correlation-context cleanup in __del__ are untouched when a deployment
hook rewrites the converted response. Covers the early-return branches
for real provider streams and unmapped call types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The token exchange mint keeps reading the user row from the database, since JWT auth caches the user it creates before adding it to the JWT's team and a mint off that cached row refused the first exchange for a new user. Introspection and the refresh revalidation go back to the cache read, so a resource server calling /introspect per request pays no database read.
The filter kept an extra as its original object whenever the plain and
scrubbed safe_dumps renderings matched, but safe_dumps skips what it cannot
render (non-string dict keys, anything past its depth limit, fields a repr
hides), so a secret in those places rode the untouched object past the
litellm_redacted stamp. The filter now walks JSON-native shapes itself
(strings, scalars, string-keyed dicts, lists and tuples) and keeps the
original only when every value it holds comes back unchanged from the
redactor; anything else is handed to safe_dumps and the record carries the
scrubbed JSON shape the formatter would have rendered
Each route now has litellm/rust_bridge/<route>/{entrypoints,callbacks}.py and a
public dispatch module (litellm/chat_completions/dispatch.py,
litellm/responses/dispatch.py, litellm/messages/dispatch.py) that binds the
public call to the legacy Python signature, builds a frozen request, and asks
the runtime to pick Rust or Python from the catalog. The legacy implementations
stay in litellm/main.py, litellm/responses/main.py and the anthropic messages
handler, and litellm/__init__.py re-exports the dispatch names over them the
same way it already does for ocr
The per-handler shims in rust_bridge/chat_completions/native.py and
rust_bridge/messages/native.py are removed along with their call sites in the
anthropic and bedrock chat handlers and the http handler. The exception
mapping that every callbacks module repeated moves to rust_bridge/failures.py
and the signature binding helpers to rust_bridge/public_call.py
Builds 232 and 233 held the Bedrock hit rate at 9% with the Claude Code
driver already sending byte-identical requests and headers, so something
between the proxy's ingress and the upstream still moves per build and
the flat key cannot say what.
Emit a digest per key component next to the counters: the test id, the
method, the URL, each keyed header, the whole body, and one digest per
top-level JSON body field. Values are digested, so no payload or
credential reaches the artifact. Diffing two builds' artifacts names the
field that moved.
Diagnostic, to be removed once it has answered.
A pre-call deployment hook can turn a requested stream into a non-streaming provider call and the result is wrapped back into a fake stream. The async client wrapper treated that wrapper like a caller-requested stream and returned before async_post_call_success_deployment_hook, so SDK callers lost post-call deployment processing (including CustomGuardrail post_call enforcement) on converted streams. Run the hook on the complete ModelResponse behind the wrapper and rewrap a modified response so it reaches the emitted chunks.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The selector read raw cost-map dicts as `Mapping[str, Any]`. It now validates
each together_ai entry into a frozen Pydantic model and takes the two
capabilities as keyword booleans, so nothing in the helper is coarsely typed
or stringly addressed.
CI runs these lanes as `python -m pytest` from the repo root, so the root is
already on sys.path and `tests._live_test_helpers` imports without help. The
insert only tripped the TQ003 test-quality budget.
`together_ai/openai/gpt-oss-20b` was hardcoded in two live tests and is no
longer served, so both failed on a vendor catalog change rather than on
anything litellm did.
Both call sites now resolve the cheapest non-deprecated together_ai chat
entry at runtime, filtered on the capabilities the tests actually exercise,
mirroring what tests/e2e/llm_translation/test_together_ai_e2e.py already
does. The selector lives in tests/_live_test_helpers.py so both lanes share
one implementation.
A non-string extra was scrubbed by a safe_dumps round trip, which handed
every user-attached handler a JSON-shaped copy even when nothing in it
was redacted. The record now keeps the original object whenever the plain
and the scrubbed renderings compare equal, so only an extra that carried
a secret comes back as its scrubbed shape
