Cron job locks now go through the shared Redis transport behind TransportPodLockStore, which owns the lock keys, the owner codec and the compare-and-delete release script. PodLockManager.is_held replaces the three direct lock reads in the budget reset, PTU rollup and spend capture-rate jobs and owns the rule that an unreadable lock counts as unheld. Old and new pods respect each other's locks
Background response polling state now goes through the shared Redis transport behind TransportResponsePollingStore, which owns the keys, codec and expiry. The wire format is kept byte for byte so old and new pods read each other's state
The transport now has every data command Python sends, request-scoped and
explicit batches on the one pipeline engine RedisBatch already used,
MULTI/EXEC batches, pub/sub, and the admin and health commands. RedisBatch
declares onto the same engine, so moved and unmoved callers in one request
still share one round trip. Cluster mget goes through the cache's
mget_nonatomic path instead of a cross-slot MGET.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rework RedisTransport into plain get, mget, set, delete and eval methods,
split the breaker step (run_under_breaker over the existing
_run_under_circuit_breaker) from the events step (RedisServiceEvents.observe,
which skips the stack walk and event when no service_callback is set), and
add eval with EVALSHA plus EVAL fallback. Fold in the store layer and the
response polling store, and replace the redis-access skill references with
the migration plan and PR sketches.
* fix(cost): bill the margin once on rebuilt streams and streamed Responses through the chat bridge
The last usage chunk and the Responses terminal event carry litellm's bill in usage.cost, and two readers took that bill for the upstream's own charge and margined it again: completion_cost on a stream_chunk_builder result, and stamp_responses_usage_cost on the chat bridge. The stamp now keeps the upstream's usage.cost under the hidden upstream_usage_cost key, the chunk builder carries it onto the rebuilt response, and the Responses stamp prices that saved figure
* fix(cost): hand the Responses stamp the upstream charge through upstream_usage_cost only, never the response-cost header
response_cost_calculator returns the response-cost header as the final bill without the margin, so a priced copy carrying the upstream's raw usage.cost there billed the client the unmargined figure. The hidden upstream_usage_cost key already feeds billed_usage_cost, which margins it once. The Responses test now prices through a real Logging object instead of a mocked calculator
* fix(cost): leave stamp_responses_usage_cost as on main
The Responses chat bridge copies the inner stream's final usage chunk, so the wrapper stamp keeping upstream_usage_cost already prices it once. Changing the Responses stamp was not needed for that and re-priced xai MCP rounds whose usage.cost is already litellm's bill (0.00512 for 0.00312 in test_xai_stream_auto_execute_bills_each_round_at_its_own_price_with_one_fixed_margin)
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: bedrock claude prompt caching on /v1/messages reports and charges cache tokens
* ci: re-run after ci-coverage and assert-shard-coverage were cancelled by the GitHub Actions incident
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): normalize empty function_call_output to an empty tool result
* test(responses): preserve empty tool outputs through guardrails
---------
Co-authored-by: Tin <tin@berri.ai>
* feat(ui): show ai&'s name and logo in the dashboard
The Add Model form already lists ai& from its provider metadata, but the dashboard had no entry for it, so the provider showed no logo and model tables printed the raw aiand slug. Adds ai& to the dashboard provider list with its official mark, plated in dark mode so the red and black mark keep their colours, and the aiand/openai/gpt-oss-120b model placeholder
* fix(aiand): retire deepseek-v4-flash, glm-5.2 and kimi-k2.7-code
ai& retired these three models on 2026-10-27 and https://api.aiand.com/v1/models now lists ten. The rows keep their prices with a deprecation_date, so earlier spend still prices, and aiand/* stops listing them, the same way baseten handles retired rows. The other ten rows already match the live prices, context windows and output limits
* feat(aiand): add deepseek-v4.1-flash-1017
ai& launches deepseek-ai/deepseek-v4.1-flash-1017 on 2026-11-02. It is served on https://staging-api.aiand.com/v1/models with the same price, context window, output limit and capabilities as deepseek-v4.1-flash, so its row copies that one
* fix(aiand): retire deepseek-v4.1-flash
ai& retires deepseek-ai/deepseek-v4.1-flash on 2026-10-30, now that deepseek-v4.1-flash-1017 is live in production on https://api.aiand.com/v1/models with the same price, limits and capabilities. Its row keeps its prices with a deprecation_date, like the other three retired ai& rows
* fix(aiand): retire deepseek-v4.1-flash on 2026-11-06
https://api.aiand.com/v1/models now gives deepseek-v4.1-flash a deprecation_date of 2026-11-05T15:00Z, which is 2026-11-06 00:00 in Japan, so its row uses 2026-11-06 instead of 2026-10-30 and aiand/* keeps listing it until then. The wildcard test now leaves a model out only once its retirement date has passed
* test(aiand): derive retired models from the rows instead of pinning dates
The wildcard test now reads each row's deprecation_date, so it holds whichever dates ai& sets, and the test that only restated the JSON dates is gone
* fix(aiand): retire models on their UTC date while the proxy runs, and add muse-spark-open
The wildcard listing now checks each row's deprecation_date when it expands, against the UTC date the field
documents, so a model drops on its day without a restart. deepseek-v4.1-flash now carries 2026-11-05, the
UTC day ai& retires it, and meta/muse-spark-open gets its price row from ai&'s live /v1/models
* fix(aiand): hide retired models only from bundled listings, including the wildcard health check
* fix(aiand): share the UTC retirement check with Together AI and the health check sort
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): evict a deleted key from every worker's auth cache
/key/delete dropped the key object only from the in-memory cache of the worker
that served the call. Auth serves key objects cache-first, so every other
worker, and the Redis copy when enable_redis_auth_cache is on, kept
authenticating the deleted key until user_api_key_cache_ttl expired. /key/update
already evicts and broadcasts through evict_and_broadcast, delete now does the
same for the tokens it deletes
* fix(proxy): evict only the deleted rows and broadcast them in one batch
Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker
* refactor(proxy): drop the comment above the delete eviction call
* fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait
* test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests
* test(proxy): wait for the peer to apply the broadcast instead of polling its cache
* test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer
* fix(key_management): evict every requested key on delete, including keys whose row is already gone
A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone
* test(key_management): split the gone-row eviction assertion into one comprehension per worker
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(streaming): stop raising AttributeErrors inside pydantic on every streamed chunk
Every later plain chunk on openai and azure streams raised and caught 12 AttributeErrors inside pydantic. Assigning an undeclared extra goes through a class lookup that raises in the model metaclass and is never memoized, and the hidden_params getter went through the private attribute fallback
set_extra_attribute writes the extra and fields_set directly when a cached per-class check says pydantic would do the same, and falls back to setattr otherwise. The hidden_params getter reads the private storage directly. Emitted chunk JSON is unchanged
* fix(streaming): match pydantic's own extra-field bookkeeping on every supported version
Take the direct extra write only when the installed pydantic itself adds an assigned extra to
model_fields_set (2.13 and later) and the class defines no attribute of that name, so pydantic
2.11, 2.12 and 2.14 see the same model state as a plain setattr. A deleted _hidden_params raises
AttributeError again, and the mutable test models carry frozen-ok reasons.
* fix(streaming): keep the normal write for models that do not allow extra fields
openai's construct gives every model an extras dict, so a forbid model built that way still needs
setattr's ValueError instead of a silent extra write.
* test(streaming): cover a ClassVar declared without a value in the extra-field parity test
* fix(streaming): type the streamed choice's logprobs as the parsed value main now stores
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(model_prices): add EmpirioLabs model rows from the provider catalog
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): align EmpirioLabs rows with the registry endpoint vocabulary and schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): absorb Gemini long-prompt batch rates and Azure GPT-6 Astra long-context flex rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): drop EmpirioLabs rows from the registry audit
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures
Pass-through failures on Gemini and Vertex routes carry the model only in the
URL, so the failure spend row and the Prometheus failure counter logged an
empty model. The failure payload now falls back to the model parsed from the
URL on those routes, and the Prometheus provider label honors the provider the
request declares before inferring one from the model name
* fix(passthrough): keep model and provider on mid-stream and Vertex failure logs
Build the failure payload from the URL at both streamed relay sites so a
200 stream that dies mid-body still logs the model, derive vertex_ai for
Vertex routes that never declare a provider, and have the Vertex logging
handler reuse get_vertex_model_id_from_url instead of its own regex
* fix(passthrough): bound the Prometheus provider label and name Vertex failures like the success handler
Declared providers only reach the api_provider label when they name a
registered provider, otherwise the label falls back to inference.
Vertex failure rows now use the success handler's model naming
(vertex_ai/ prefix on rawPredict and streamRawPredict, vertex_ai/search_api
on :search) and its host-based provider detection, and the detected-stream
failure sites are covered by tests.
* fix(passthrough): name Google failures from the URL on untracked methods too
A Gemini or Vertex pass-through failure on a method the success handler does
not track (countTokens, for one) still logged model "" and no provider,
although the model sits in the URL. The failure payload now falls back to the
Google AI host for the provider and reads /models/<name> from the URL, so
countTokens 400s name their model and provider like generateContent ones do.
Non-Google hosts keep the tracked-route rule and still log nothing extra.
* fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases
The failure path only named the Google provider from the URL when the
method was one the success handler tracks or the host was a Google API
host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE
(or a Vertex one behind a custom base) still logged an empty model. A
route that declares gemini or vertex_ai now wins outright, and a URL
whose path carries /projects/<p>/locations/<l>/ counts as Vertex on any
host, while a tracked method on a non-Google host no longer passes as
Vertex.
* fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model
* test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback
* fix(passthrough): scope declared provider to pass-through failures and match Google hosts only
Prometheus trusts the resolved provider as before and reads the request-declared
provider only on pass-through failures. Vertex attribution matches Google hosts,
strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the
URL path only, covers Vertex Live WebSocket failures, and labels a router alias
failure with the alias the client requested
* fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels
A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler.
* fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures
A pre-call guardrail block on a pass-through route now builds its failure
payload the same way upstream failures do, so the model and provider read
from the URL reach spend logs and metrics. Vertex Live websocket failures
(connect errors, upstream rejections, and the credential failure) now carry
the model from the ?model= query parameter under the vertex_ai provider
* fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls
Prometheus reads a failure's declared provider only on pass-through calls, so
without call_type it fell back to guessing from the model name. That labeled a
guardrail block on the Gemini route as vertex_ai and the Live credential
failure as None
* fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row
* fix(pass-through): keep URL model attribution off routes that name another provider
A built-in route that names a non-Google provider, like /openai pointed at
Gemini's OpenAI-compatible host, no longer takes its failure model from the
Google URL. A guardrail block response now returns the x-litellm-call-id of
its spend row.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
