* chore(hooks): drop Conventional Branches pre-push hook in favor of litellm_ prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(makefile): describe install-hooks as Conventional Commits only
* docs(contributing): scope the litellm_ branch prefix to internal contributors
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Savings and the savings baseline priced deployments through Router.get_deployment_model_info, which deep-copies the cost map entries on every call. The router already keeps an lru_cache of the same lookup that is cleared whenever the model list or cost map changes, and budget reservation reads from it, so the savings path now reads from it too
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* fix(speech): correctly propagate spend logging and usage for Gemini TTS bridge
* fix(speech): call Gemini once per TTS request and log the bridge's token usage
aspeech re-ran speech whenever the first result was not a coroutine, so the
speech-to-completion bridge sent every Gemini TTS request upstream twice. It
now returns the sync result.
The bridge's inner completion runs as an internal sub call, so only the
speech call fires success callbacks, and the completion's usage rides on the
audio response so the spend log and usage_object carry real token counts.
Replaces the separate inner Logging object and the cost_calculator fallback,
which never reached the spend row.
* fix(speech): flag a cached Gemini TTS reply as a cache hit so it logs at no charge
* fix(speech): price a cached Gemini TTS reply's savings from its token usage and keep its cache key
* test(integration): cover Gemini TTS speech billing, logging, cache hits and deployment TPM
* test(integration): start the speech TPM cell in a minute's first 10 s so the two requests never cross a minute
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): refresh cached team on /team/block and /team/unblock
Both endpoints wrote `blocked` to Postgres and returned without touching the
management-object cache. Auth resolves teams through `team_id:{team_id}`
before reading the DB and `common_checks` enforces `blocked` off that cached
object, so a team whose traffic had already warmed the cache kept serving
requests after being blocked, and kept getting rejected after being
unblocked, until the entry expired.
`_refresh_cached_team` already exists for exactly this and its docstring says
every endpoint mutating `litellm_teamtable` must call it; these two were the
only mutating endpoints that didn't. The update now also asks for
`object_permission` so the refreshed cache entry doesn't drop the relation,
matching `team_model_add` and `team_model_delete`.
* fix(proxy): keep a team block from being undone by in-flight spend writes
update_cache read the cached team when a request finished and wrote that
whole copy back in a background task, so a team blocked or invalidated
while requests were in flight came back unblocked on every busy worker
until its 60 s cache entry expired. The team spend now goes onto the team
held in this worker's memory at write time, and a missing entry is left
missing for the next auth read to load.
Block and unblock responses keep litellm_model_table and object_permission
null, as before this PR, while the cache still gets both relations.
* test(proxy): type the team spend cache test parameters
* test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock
---------
Co-authored-by: Hamza Shah <hamzasepal@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The override test slept a fixed 10 s after writing responses_websocket_session_limit_seconds and then
opened sockets on both workers. A worker whose config resync ran later than that kept the old 7200 s
cap, so its socket never closed and the test failed after 75 s
A test-only /health/integration-general-setting route returns the value a worker holds along with its
pid. The test polls it over fresh connections until every worker reports the new cap, and after the
delete until every worker reports it gone
`lite claude --subscription` sends the LiteLLM key in the x-litellm-api-key header through
ANTHROPIC_CUSTOM_HEADERS and leaves ANTHROPIC_AUTH_TOKEN and ANTHROPIC_API_KEY unset, so Claude Code
stays on its claude.ai login and its connectors keep working. The pre-launch key check sends the key
the same way.
`claude_subscription` and `pkce` are now `lite config` keys that default the new
`--subscription/--no-subscription` flag on `lite claude` and the `--pkce/--no-pkce` flag on
`lite login`. The flags still win for a single run.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cooldown cases in test_hosted_vllm_stream_stall_wire.py boot an owned proxy, which writes to a
private database since #47213. The file was written on a base without #47213 and read its rows from
the worker database through read_rows, so every cooldown case that waits for a failure row timed out
on main with State did not converge: []
Every row read now goes through Gateway.rows on the gateway that served the request
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
#46662 made streamed /v1/responses lifecycle events report the requested model in the nested response.model, matching the non-streamed body. The arize v2 caller-stream helper still expected the upstream deployment name there, so f1 and a6 failed on every main run since bbd446b476
The SSO removal test configured Google sign-in, which fetches its discovery
document from accounts.google.com before redirecting. CircleCI blocks that
egress, so the test failed on every main run since #46709 added it. Microsoft
sign-in builds its endpoints locally and goes through the same SSO environment
variable handling, so the test now uses it
* test(proxy): cover OpenAI pass-through billing for each /responses sub-route
* fix(proxy): bill only the OpenAI pass-through Responses calls OpenAI bills
* fix(proxy): fail closed on unknown /responses sub-routes and label unbilled spend rows
Bill every OpenAI pass-through /responses call except input_tokens, input_items,
delete, and a non-background retrieve or cancel, so an encoded or new sub-route
is never free. Unbilled calls keep the response's model on their spend row, and
a resumed background stream takes its model from the stream's final event.
* fix(proxy): give billed background Responses reads their own spend log row
A billed retrieve or cancel of a finished background response reused the
create's resp_ id, so the spend log insert dropped the row that carried the
cost. It now takes the native path's _background_cost suffix. Adds Azure
path coverage and types the test recorder.
* fix(proxy): bill a background Responses read only once it carries usage
An in-progress poll took the _background_cost spend log id with $0, so the
finished read's row that carried the cost was dropped again. Polls with no
usage are now unbilled like any other read.
* fix(proxy): keep OpenAI's response id for callbacks on a billed background read
* fix(openai): give a streamed background create the settlement spend row id
* fix(proxy): keep a request body from giving its spend row the background settlement id
* test(proxy): give the forged settlement marker test a fixed time
* fix(passthrough): move the background settlement marker into the pass-through types module
A test that reloads spend_tracking_utils gave it a new enum class, so the
handler's marker no longer matched by identity and the settlement row lost
its _background_cost id. The types module is never reloaded.
* fix(managed_files): send provider file ids when creating a fine-tuning job from managed files
Fine-tuning job creation with a managed training file sent the gateway's managed id and the provider-prefixed deployment model to OpenAI, which rejected the job. The pre-call hook now builds the file id mapping for training_file and validation_file, the deployment hook swaps both, the SDK strips the provider prefix from the model, and the create response returns the caller's own file ids
* fix(managed_files): route managed fine-tuning creates only to deployments holding every file
Fine-tuning creates now narrow routing the same way managed batches do, so a
deployment added after the upload is never picked. Adds a two-worker chaos
integration test and runs tests/unit/fine_tuning in the endpoints shard.
* fix(managed_files): keep fine-tuning creates on the model group when no deployment holds the files
A managed file used on a model group it was never uploaded for kept failing
with the provider's 400 before the routing filter. The filter turned that into
a 429 no-deployments error, so fine-tuning now falls back to the whole group in
that case while batch input files keep routing only to deployments that hold
them.
* test(fine_tuning): cover the provider prefix strip for OpenAI and Azure creates through the proxy
* test(fine_tuning): start each multi-deployment case from an empty upstream log
* test(fine_tuning): boot both proxies of the late deployment cell on one scratch database
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer 400 for a malformed tool_result text on the Responses bridge and count_tokens
A tool_result text block whose text is null, a number, or missing made
/v1/messages and /v1/messages/count_tokens answer 500 on OpenAI and Azure
models. Both routes now answer 400 naming the block, and drop the block
when drop_params is on.
* fix(anthropic): keep a missing tool_result text as empty text, bound the error, honor body drop_params on count_tokens
* fix(anthropic): apply the tool_result text check on the chat/completions bridge and every count route
The chat/completions bridge used for Azure, Gemini and other providers now answers the same 400 for a
null or numeric tool_result text and drops the block under drop_params, so a model group that mixes
OpenAI with another provider answers the same way on every deployment. Gemini and Vertex count_tokens
apply the same check, count_tokens turns on drop_params for Claude Code and Codex the way /v1/messages
does, and a tool_result without a tool_use_id is checked too
* fix(proxy): answer 400 for a malformed tool_result text on count_tokens models without a count API
Providers with no token counting API fall back to the local tokenizer, which counted a null or numeric tool_result text as fine while /v1/messages on the same model answers 400. The fallback now runs the same check, unless drop_params applies to the deployment or the request
* fix(proxy): honor request drop_params on the local count fallback when no deployment resolves
* test(integration): cover non-string tool_result text on both /v1/messages bridges and count routes
* fix(anthropic): keep a dropped tool_result on Mistral and reject it before a guardrail rewrite
Under drop_params a tool_result whose only text block was null or a number became an
empty tool message, and Mistral's empty-message filter removed it, so Mistral answered
400 for a missing tool reply. The filter now only removes empty assistant messages,
which is what Mistral rejects.
A guardrail that rewrites structured_messages wrote the dropped text back as "", so a
null or numeric text was silently blanked even without drop_params. The rewrite path now
answers the same 400 as the bridges unless the request or global drop_params is set.
* test(integration): cover the Mistral empty tool reply and the guardrail rewrite tool_result text check
* fix(anthropic): keep a malformed tool_result text through a guardrail rewrite and keep Mistral's null-message drop
A guardrail that rewrites structured_messages now writes the caller's null or numeric
tool_result text back in place, so the route's bridge and the deployment's drop_params
decide after routing, the same as with no guardrail. The rewrite path no longer answers
its own 400 before routing.
Mistral drops a null-content system or user message again, as main does, and only a tool
reply is exempt from that filter so a dropped tool_result keeps its tool call.
* test(mistral): expect empty system and user turns dropped and the empty tool reply kept
* perf(anthropic): restore kept tool_result text parts in one linear pass
* test(integration): cover the tool_result text restore across tool_results, the chat bridge, and a batch upload behind a rewriting guardrail
* fix(anthropic): let guardrails scan malformed tool_result text and honor drop_params on native Anthropic
A rewriting guardrail now sees a non-string tool_result text as its JSON text and the write-back
restores the caller's value by the sanitized tool_use_id when the guardrail left it unchanged.
Native Anthropic drops the bad block under drop_params on /v1/messages and count_tokens, and
Mistral keeps an empty tool reply only when it answers a tool call, sent as empty text.
* fix(anthropic): compare a kept tool_result text to its scanned form once and align the wire tests with drop_params
* test(integration): cover native Anthropic drop_params resolution, guardrail scanning of numeric tool_result text, and the Mistral orphan tool message
* fix(count_tokens): honor the router's default drop_params when counting a malformed tool_result text
* test(count_tokens): send the tool_result count tests through a real request body instead of a mocked parser
* test(count_tokens): send the native Claude count test through a real request body
* fix(anthropic): keep each non-string tool_result text when tool_use_ids sanitize to the same id
* fix(anthropic): keep non-ASCII characters in the guardrail's view of a non-string tool_result text
* test(gemini): mock the count_tokens upstream with respx instead of patching the client factory
* fix(mistral): keep an empty tool reply only for an open call, and forward non-object message items on the native drop
An empty tool reply re-answering a call the conversation already moved past made Mistral reject the request, and a tool call id that is not a string crashed the filter. Native Anthropic under drop_params assumed every message item was an object.
* fix(mistral): find answering tool replies in one pass instead of copying the open call set per reply
* fix(mistral): ignore dropped empty placeholders when tracking open tool calls
* fix(mistral): drop an empty tool reply when another reply to the same call carries the result
* refactor(anthropic): share tool_result text helpers to shrink the diff
* fix(anthropic): answer 400 for a non-string tool_result text on the OpenAI Responses bridge
* fix(prometheus): label deployment failures with the deployment's provider instead of None
* test(prometheus): drive every deployment failure surface through the provider label
Streaming chat, responses and messages, dropped and stalled upstreams,
JSON-configured, custom-provider-map, prompt-management and unprefixed
deployments, forged request metadata, an unauthenticated request, the
OpenAI and Anthropic SDKs sync and async, and a concurrent burst counted
exactly once per deployment
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(logging): report the cost-map entry billing used in model_map_information
Billing resolved the deployment's own cost-map entry and logging ran a second,
name-only lookup, so callbacks and spend logs showed the catalog rates for a
custom-priced deployment. completion_cost now stores the entry it billed from on
the logging object and the standard logging payload reads it, falling back to the
old lookup only when billing never ran.
* fix(logging): pass the deployment id into the logging cost-map lookup
Drop the logging-object plumbing and keep the lookup where it was:
get_standard_logging_object_payload now hands the deployment id to
get_model_cost_information as router_model_id, the same id billing uses, so
the deployment's own cost-map entry is reported instead of the catalog entry.
* test(logging): mirror the proxy metadata in the deployment-price regression test
* fix(logging): keep the catalog model_map_key, report only the deployment's own rates
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure): cache the remaining per-request Entra credentials and keep token refresh off the event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): await the now async azure health-check auth header helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure): await async azure_ad_token_provider results in the threaded wrapper
* fix(azure): keep transport params when validating async Azure client params on pydantic 2.11
* test(azure): assert the token and validation thread hops by thread identity instead of sleeps
* fix(azure): give the OIDC token exchange its own short-lived HTTP client
The exchange now runs from worker threads, so it no longer shares litellm.module_level_client's connection pool with other threads
* test(azure): check the OIDC exchange closes its client through an in-process transport
* refactor(responses): merge caller extra_headers into a new dict instead of updating validated headers in place
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(search): add DeepSeek as a /v1/search provider
Adds DeepSeekSearchConfig for POST https://api.deepseek.com/v1/tools/search, registers it under search_provider=deepseek, prices deepseek/search at 0.00075 per query and marks search support in provider_endpoints_support.json
* fix(search): reject DeepSeek queries over the 128 character limit before the call
* test(search): escape regex metacharacters in DeepSeek query limit match
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
68db022d57: `AssertionError: (('success',),)`).
The test now reads the spend row first, which is written once the proxy has settled the disconnect, and
releases the upstream after it. The proxy's behavior is unchanged; only the test's ordering is.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): record cooled deployments on the all-cooled failure row
A RouterRateLimitError now carries the cooled deployments of the route the
request resolves to for the caller's team (model group, alias, wildcard or
team route), each with its masked api_base. get_error_information copies them
into error_information.error_cooldown_deployments so the SpendLogs failure row
and every logging callback can tell which deployments were cooling
* fix(router): name cooled deployments from the router's own resolved route
Use the deployments the router resolved for the request, before health and
cooldown filtering, instead of re-deriving the route. Alias chains, the
provider-model fallback, wildcards and team filtering now match routing by
construction. A routing-strategy refusal records null.
* fix(router): name only the cooled deployments tag and region routing allow
The refusal's cooled-deployment list now leaves out deployments the
request's tags or the end user's allowed region keep it from, so a
tenant's refusal row no longer shows another tenant's api_base
* fix(router): keep failure logging working for refusal subclasses that skip init
A RouterRateLimitError subclass that never calls its parent init now logs a null cooled-deployment list instead of breaking the failure payload
* test(router): call the refusal routing helpers directly
The router coverage check needs each router function called by name from a unit test
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
anyio below 4.14.0 can leave a cancelled waiter queued on an asyncio Lock or
Semaphore, so the next task to acquire waits forever on a lock nobody holds
(agronholm/anyio#1145, fixed in 4.14.0). httpcore takes these locks for every
httpx request, including the prisma query-engine client
litellm imports anyio directly but never declared it, so pip and uv kept an
already installed anyio 4.13.0 when installing litellm and accepted an
explicit anyio==4.13.0 next to it. uv.lock already resolves 4.14.2, so Docker
and CI installs do not change. Both the litellm and litellm-core manifests get
the floor, and the lock only records the new direct dependency
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(openai): drop unsupported reasoning effort for gpt-6.1-sol
gpt-6.1-sol accepts low, medium, high, xhigh, and max. none and minimal were still forwarded, because chat completions ignored an explicit supports_none_reasoning_effort=false and the Responses mapper never applied that contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openai): mark the stripped reasoning object as outgoing JSON
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Azure responses reading the OpenAI effort map
gpt-6-astra on Azure accepts reasoning.effort none. The responses gate was checking the OpenAI entry, which rejects it.
Signed-off-by: Paul El Khoury <82139421+pOpO1-9@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openai): gate every reasoning_effort level from the cost map on chat and Responses
One gate reads the model's cost-map row for whatever effort level a request
asks for, on Chat Completions and the Responses API, for OpenAI and Azure.
xhigh stays opt-in on chat (Azure also keeps none opt-in); every other level
is refused only when its supports_<level>_reasoning_effort flag is false. A
refused level is dropped under drop_params or answered with a local 400, and
Responses keeps the rest of the reasoning object when it drops effort. The
guards that follow judge temperature by the level the provider will see.
Tests move into the mapped test files and derive the expected outcome for
each level from the map row, so a new GPT-6 model needs only its row.
* test(openai): pin gpt-6.1-sol's refused effort levels and prove the row flags alone drive the gate
* fix(cost-map): azure gpt-6-astra refuses none and honors max on Responses, as live calls show
* test(reasoning_effort): pin the astra max and none flips through the adapter and Foundry paths
* fix(reasoning_effort): keep max out of the route gates and correct the gpt-5.3-codex rows
The same model honors max on Responses and refuses it on Chat Completions, so one row flag
cannot gate it on either route. The gates now skip max, the Azure gpt-6-astra rows keep max
off so the /v1/messages adapter degrades to xhigh and /model_group/info stops short of it as
on main, and chat still forwards max for the Responses bridge.
The gpt-5.3-codex rows said none and xhigh off and minimal on; the live model answers the
other way round (2026-10-01), and an opt-out gate would have dropped the levels Codex CLI sends.
* fix(reasoning_effort): bridge function tools to Responses when the chat gate drops none
* docs(reasoning_effort): correct the max advertisement note
* fix(azure_ai): read the Foundry row on the Responses effort gate and turn none off for Azure gpt-6.1-sol
* fix(azure): gate a prefixed Foundry deployment's Responses effort off its row
litellm.responses(model="azure_ai/<deployment>", api_base=<Foundry host>) is
re-routed to the azure provider with the azure_ai/ prefix intact, so the Azure
Responses config saw a prefixed name, skipped it as another provider's
pass-through, and forwarded none and minimal to Azure even under drop_params.
The Azure gpt-5 resolver also read no row at all for a prefixed name.
The Azure Responses gate now detects the family the way the Azure chat gate
does, and an azure_ai/ name on the Azure OpenAI configs reads its azure_ai/ row
when the map has one, else the azure/ twin, the rule the Foundry chat config
already applies to the same deployment.
* fix(cost-map): turn none off on azure/eu/gpt-6-astra like its sibling rows
* docs(reasoning_effort): say how each gate reads none and shorten the module docstring
* fix(azure_ai): call the public effort-level check from the Foundry Responses gate
* test(reasoning_effort): fold the Foundry Responses tests into the mapped file and type the row helpers
---------
Signed-off-by: Paul El Khoury <82139421+pOpO1-9@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(proxy): opt-in self-serve budget edits for key owners on /key/update
Adds general_settings.self_serve_budget_policy (disabled by default) so a
non-admin can change max_budget or budget_limits on a key they created and
still own. lower_only allows adding or lowering limits. ceiling also allows
raising or removing them when the owner could create that key with the new
budget via /key/generate (same member permissions and delegation ceiling).
soft_budget and spend stay admin-only
* fix(proxy): sync dashboard API types and cover the UI session cap on update
* test(proxy): cover team role restriction for owner budget raises
Drop two unreachable branches from the non-raising key generation check.
* fix(proxy): close self-serve budget bypasses for linked budgets, null caps and retained limits
Keys with a linked budget_id need an admin to change max_budget, owners cannot
leave a key uncapped when an upperbound max_budget is configured, and the
ceiling check now runs on the budget the key would carry after the update.
* fix(proxy): keep open key budget windows and the admin path on self-serve budget edits
A resubmitted budget_limits window now keeps its stored reset_at while that period is still
running, so adding or editing one window no longer restarts the others. Team and org admins
answer False on a ceiling loosen so they fall through to _check_key_admin_access as before.
The self-serve tests feed the team through the cache and prisma mocks instead of patching
get_team_object, and every helper is typed
* perf(proxy): group key budget windows by duration in one pass
The per-duration minimum used to rescan the whole window list once per distinct
duration, so a request with thousands of windows pinned the event loop before any
permission check ran. Sort once and take each group's minimum instead
* refactor(proxy): take budget windows as a Sequence in the delegation ceiling checks
* test(proxy): assert allowed self-serve budget edits return no denial
Also drop the unreachable assert_never arm from the exhaustive budget change match.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* refactor(count_tokens): hold the tool counting wiring for a follow-up commit
* refactor(count_tokens): hold the chat template tool sections for a follow-up commit
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* fix(count_tokens): skip tool_choice and non-function tools for chat template families, leave out Fireworks V4.1 spelling
* fix(count_tokens): count Anthropic tools in the shape the messages bridge sends them
A missing description is no longer rendered as null and strict is counted when set, matching what DeepSeek V4 and Qwen3.5 deployments receive. Adds tests that pin each template's schema rendering and its system turn.
* test(count_tokens): check the prompt caching threshold agrees with token_counter for DeepSeek V4 and Qwen3.5 tools
* fix(count_tokens): skip Anthropic web search tools and count unmapped Anthropic tool keys for chat template families
* test(token_counter): cover chat-template tool counts across count routes, budgets, ITPM and context edits
Integration cells check that DeepSeek V4 and Qwen3.5 tools are counted the way their chat templates render them on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens, strict budget reservation, project ITPM on chat, messages and responses, and the clear_tool_uses trigger, with grok controls keeping OpenAI framing
Show discovered Kubernetes pods on the model page for admins
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Requests with an explicit session id stick to one discovered pod via rendezvous hashing
Requests without a session id keep round robin
Pod routing records are included in logs, spend logs, and traces
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(baseten): list current Model APIs instead of retired 2023 deployment IDs
`baseten/*` wildcard expansion and `/v1/models` read a hardcoded set of
three 2023 deployment IDs (Falcon 7B / WizardLM / MosaicML) that no longer
exist, while the model cost map already carries current Baseten entries.
- Populate `baseten_models` from the model cost map, like fireworks_ai.
- Add `BasetenConfig.get_models()` against the Model API `/v1/models`
endpoint and register it in `ProviderConfigManager.get_provider_model_info`,
so `check_provider_endpoint: true` returns the live catalog.
- Remove the retired deployment IDs from `constants.py`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(baseten): mark retired Model APIs deprecated and skip them in listings
15 Baseten entries in the model cost map are no longer servable: the Model
API returns 410 "model version ... has been deprecated" (DeepSeek-V3.1
returns 400 deactivated), and none appear in `inference.baseten.co/v1/models`.
- Set `deprecation_date` on those entries (kept for spend tracking).
- Skip entries past their `deprecation_date` when building `baseten_models`,
so `baseten/*` expansion only lists models that still serve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(baseten): assert the custom api_base model listing on the returned models
The custom api_base test only inspected the patched client's call args, which trips the
TQ002 mock-echo gate. Serve the custom URL with respx and assert on the models that come
back instead, so a wrong host or a broken parse both fail.
* fix(baseten): drop newly retired models on cost map reload and fake discovery at the HTTP boundary
A cost map refresh only ever added to the Baseten set, so a model that gained a past
deprecation_date after boot stayed listed under baseten/* until restart. Discard it on reload.
Tests now serve the catalog with respx instead of patching our own HTTP client, the
isinstance registration test is gone since the get_valid_models test already proves
discovery end to end, and new test locals carry Final.
---------
Co-authored-by: AlexKer <AlexKer@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
