Streaming chat relays on Azure and azure_ai deployments rebuild the response from
the SSE chunks through the OpenAI passthrough assembler, so the spend log carries
usage. The router relays keep the JSON body when the Content-Type carries a
charset, return the upstream status and body instead of a 500 when the deployment
rejects the call, and fall back to the caller's api-version when the deployment
sets none. Lint budgets ratcheted to the measured totals
A shadow eval job could only be scoped by identity, so "this user's traffic on model X
across every key they own" was not expressible and a models field on the start body was
silently dropped. The job now carries a models list that every target is narrowed to,
matched on the requested model group with model_group_alias resolved on both sides. An
unresolvable name is a 400 at start. Empty means every model, which is what every existing
row reads as. The dashboard start form gains an "Only on models" picker and the job
headline shows the scope.
The reset job evicts the cached end-user object only from its own worker's
in-memory cache (plus Redis), so every other uvicorn worker and replica keeps
the pre-reset spend for up to user_api_key_cache_ttl (60s by default). Those
workers pass that stale spend as fallback_spend, and since the authoritative
floor read returned None for spend:end_user: keys, get_current_spend handed
the stale value straight back and the end user kept getting 429 after the
rollover on every worker but the one that ran the reset.
The floor read now consults LiteLLM_EndUserTable.spend for end-user counters,
the same way keys, teams, users, and orgs already read their rows. It runs only
when the shared counter sits below the cached spend (a reset or a Redis
restart) and stays behind the existing 5s in-process marker, so the normal
request path still does no DB read. Cold end-user counters keep seeding from
the cached object rather than the row, so from_db is unchanged for them.
* fix(headroom): bound the /v1/compress and /v1/retrieve calls with a timeout
The headroom guardrail builds its client with get_async_httpx_client(GuardrailCallback)
and no params, and passes no timeout on either outbound call. That client's read, write
and pool legs are 600s (litellm.request_timeout when set explicitly, default 6000s), so
an unreachable or stalled compression service holds the caller's pre-call request open
for the whole window before unreachable_fallback ever runs. Because the client is shared
with every other no-params guardrail, each stalled call also pins a pooled connection for
the same window, so a saturated pool makes unrelated requests block on the pool leg.
Bound both calls at 60s by default, honoring litellm_params.timeout when set (the field
already exists and documents itself as the per-guardrail API timeout; headroom accepted
it and ignored it). The connect leg stays at the http_handler default, or the configured
budget when that is shorter, so a dead host still fails fast.
Live on a proxy against a stalled /v1/compress: 600.4s -> 60.2s before the 502, and 5.2s
with timeout: 5 configured.
* fix(headroom): reject non-finite timeouts and trim the timeout commentary
`timeout: .inf` on a Headroom guardrail reached httpx and the aiohttp transport
raised OverflowError, so every request came back as a raw 500 instead of going
through unreachable_fallback. Reject non-finite values the same way as
non-positive ones, and cut the comments and docstrings back to what the code
does not already say.
* feat(cli): sync OpenCode models from /v1/models in lite opencode
lite opencode now fetches the proxy's /v1/models with the resolved key and
hands OpenCode an OPENCODE_CONFIG_CONTENT declaring a litellm provider
(@ai-sdk/openai-compatible, proxy /v1 base URL, {env:OPENAI_API_KEY}) with one
model entry per listed chat model, so the model picker mirrors the proxy
without a hand-maintained opencode.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): sync OpenCode models only after the key check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The access group detail page rendered MCP servers, agents, attached teams and keys as bare ids, so an admin had to look each one up elsewhere to audit a group
Every access group response now also carries access_mcp_servers, access_agents, assigned_teams and assigned_keys as {id, name} pairs. Names come from the DB rows first and fall back to config-declared MCP servers and agents (including legacy agent ids), resolved with one query per table across all groups in a list call. The existing *_ids columns are unchanged
The UI renders the name with the id in a tooltip, links teams and keys to their detail pages, and shows the raw id only when nothing resolves
A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full
The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
* fix(health): probe test_connection with the credential the request names
/health/test_connection matches the request's model string against the
configured deployments and merges the match's litellm_params underneath the
request. A request that named a stored credential but no key of its own
still satisfied the "request sets no connection fields" test, so it inherited
the matched deployment's api_key and api_base, and load_credentials_from_list
then skipped the named credential because api_key was already set.
A wildcard route covering the model is enough to match, so the Add Model
page's Test Connect probed with an unrelated deployment's key while echoing
back the credential that was selected.
Naming a credential the configuration does not name now withholds the
configuration's credential fields, the same set already withheld from a
request that supplies its own endpoint. Naming no credential still inherits
them, as documented.
* test(health): drop test docstrings that restate their own names
* test(health): assert the credential probe on the wire, not on the call args
The connection-test regressions patched litellm.ahealth_check and read the
params handed to it. Driving the endpoint through the app with respx faking
the upstream instead lets the real credential resolution run, so the tests
assert the key and host that actually go out, which is what the bug was about.
It also drops three of the five patched proxy internals; the two that are left
are proxy-global wiring with no injection seam, the same ones the image_edit
connection test already has to reach for.
* chore(ui): regenerate schema.d.ts for the test_connection docs change
/key/regenerate carries the JWT-to-key mapping to the new token via FK
cascade, but the jwt_key_mapping cache entry kept resolving the old
(now invalid) token for up to virtual_key_mapping_cache_ttl. Snapshot
the key's mapping cache keys before the token update and evict them
with evict_and_broadcast so every worker drops the stale entry.
Also share the cache-key format through jwt_key_mapping_cache_key and
upgrade the /jwt/key/mapping CRUD endpoints from local-only deletes to
evict_and_broadcast, closing the same cross-worker staleness there.
A row's cost sums only the daily rows that carry a tracked cost, so it
silently under-reports whenever some rows are NULL (pre-migration days,
old pods mid-rollout, an unpriced counter). Both usage endpoints now
return the per-counter units behind those NULL rows next to the cost
(untrackedUsageUnits / totalUntrackedUsageUnits on the overview,
untracked_usage_units on the detail), so a partial cost is never mistaken
for a complete one and the reader can see exactly what it excludes
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
* feat(team): report per-user spend within a team for JWT traffic
Add GET /team/spend/by_user, which groups raw spend logs by (team_id, user)
so JWT/SSO requests with no virtual key are attributed to the user inside
each selected team. Team admins see every member, plain members see only
their own row. The Team Usage page gets a Spend Per User Within Team card
with CSV export backed by the same endpoint.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(team): cover /team/spend/by_user in behavior suite, tf audit allowlist and EntityUsage unit test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(team): drop explanatory docstrings from /team/spend/by_user and regen schema.d.ts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_numeric_form_type only peeled a single ReadOnly layer, so a field still
wrapped in Required/NotRequired was read as non-numeric and dropped from the
mapping. Which qualifiers survive get_type_hints varies by interpreter version
and by include_extras, so on Python 3.10 NotRequired[ReadOnly[int]] reached the
check intact and the field was silently skipped, which is what turns the mapped
test red on the 3.10 leg only.
Peel Required/NotRequired/ReadOnly/Annotated in any order and nesting instead.
The one production caller feeds a schema with no qualifiers, so the resulting
mapping is unchanged on every interpreter in the matrix, but a field written the
house-convention way stops being dropped.
