mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-27 06:13:07 +00:00
875 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
alberto
|
f768fb32ef
|
fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
e333ccabdb
|
fix(scim)!: delete a group through the team delete's bulk path instead of one member at a time (#47544)
* fix(scim): write a group's roster in one locked transaction instead of one member at a time POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the membership rows and the users' team tags in one transaction under the team's advisory lock, so a 500-member push runs a fixed number of statements instead of thousands. PUT now answers with the roster it just wrote * test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers * fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock * fix(scim): evict the cached team on a group rename and read a no-op roster from the writer A rename-only PUT or PATCH wrote the new alias and left the team cached under its id and old alias on every worker, so keys kept logging spend against the old team alias until the TTL expired. The group write now evicts the cached team (and broadcasts the eviction) right after the alias or metadata update. sync_team_roster reads the team from the writer when the plan names nobody, so a replica cannot hand back the pre-rename row; a plan already matching the roster returns inside the lock without writing, evicting, or emitting the members metric; the response is built from the written row, so lastModified reflects the roster write; and detaching a removed member bumps the user's updated_at like every other user write. * fix(scim): bump detached members' updated_at in UTC The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts into Prisma's naive timestamp column in the session timezone, so on a database whose session is not UTC the removed member's row landed hours away from the UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and have the bulk roster cell assert the bumped row sits between two reads of the database's own UTC clock taken around the replace. * fix(scim): delete a group through the team delete's bulk path instead of one member at a time * fix(scim): chunk the group write's member lookups under the bind cap Every Prisma "in" filter the group write issues bound one parameter per member with nothing bounding the list, which check_unbounded_in_lists reports as 11 new findings. The single-field lookups now go through litellm.repositories.chunked_in (caller_transaction inside the roster transaction, per_chunk_ok for the role recompute), and the one cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two slices, and a member named only in the last slice still resolves. * fix(scim): demote a deleted admin group's members before the team delete so a failed delete stays retryable * fix(scim): recompute member roles again after the team delete so a concurrent group write cannot regrant admin * fix(proxy): send bulk cache evictions in slices so the broadcaster drops none Veria pointed out that the invalidation publisher caps its pending tasks at 1,024 and drops the excess with a warning, and that a group push now detaches every removed member's keys in one call instead of one member at a time, so a removal of more than roughly 340 members (three cache keys each) silently left stale key objects on the other replicas. The merge base never hit the cap because each member was a separate request. Bulk evictions (team, user, organization, and session key deletes, plus the roster settle) now publish in slices of 256 and wait, bounded at two seconds, for the backlog already handed to the event loop before each slice. A Redis that never answers costs a bulk caller at most that wait per slice instead of wedging the request, and single-key callers are untouched. Regression tests push 1,500 keys and a 500-member removal through a slow publisher and assert every eviction reaches Redis; both fail against the previous tip. * fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts Greptile pointed out that pacing the broadcasts in slices also paced the local eviction: a bulk delete waited for the publish backlog before dropping the next slice from this worker's own memory, so on a Redis that accepts connections but never answers a key deleted from the database kept authenticating on the handling worker for up to two seconds per slice. The merge base cleared every key locally at once. Bulk evictions now drop every affected entry from this worker's memory first, through the cache's own partition lookup since key objects live apart from the other auth objects, and only then publish in paced slices. The roster settle does the same for the deleted tokens, the JWT mapping keys and every changed member's user and membership entries before any step that can wait. Regression tests run both paths against a publisher that never answers and check that nothing is still cached on this worker before the first bound elapses; both fail with the local eviction removed. * fix(scim): write recomputed member roles only where the membership still backs them * fix(team): report the users a team delete detaches so SCIM demotes late-added members * test(scim): audit cells for the bulk group delete Move the SCIM group helpers into tests/integration/_support/scim.py and add the delete cells: a member's other group survives, the admin group's members are demoted and the audit row names the SCIM caller, a member a concurrent PUT adds while the delete waits on the team lock is demoted too, unknown, repeated, non-admin and hostile ids, an empty group, the peer proxy, a 20-delete burst across both proxies, and a delete after a worker was SIGKILLed * fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts * docs(proxy): drop the module docstring that repeated the PR description * fix(team): sweep deleted team references only under the lock so a late-added member is demoted The unlocked sweep that ran before the team's advisory lock stripped a member a concurrent group write added after the roster was read, so neither the roster snapshot nor the locked sweep's RETURNING named them and the SCIM recompute left them proxy_admin after the admin group was gone. The locked sweep does the same strip and reports whom it stripped, so it is the only sweep now. The integration cell holds the delete on a member's user row so a PATCH lands between the roster read and the lock, then checks the late member is demoted. Both waiter queries in the SCIM support module return their count as text so the first poll parses. * chore(team): carry the unbounded-IN baseline entries to delete_validated_teams The delete pass moved from delete_team and the unlocked sweep into delete_validated_teams, so the checker's baseline names the function the two team_id IN lists now live in * test(scim): expect the master key alias in the audit row and send post-kill requests on a fresh connection * fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps Bulk key evictions now drop entries from memory, delete them from Redis in one chunked DEL, and broadcast every key in one pipelined publish, so a large roster removal no longer overflows the publisher's pending cap or costs a Redis round trip per member. A push waits for in-flight membership loads before evicting. User rows are locked in byte order to match the spend writer, a bulk user delete takes every team lock before its first user write, and a POST whose roster write fails deletes the empty team so the IdP retry creates it again. * test(team): observe delete broadcasts at the Redis layer * fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently * fix(team): chunk the email lookup behind the member row lock * fix(scim): create a member deleted mid-push again instead of failing the group write * fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch A member deleted mid-push is created again only when scim_upsert_user allows it, under the verb's own created_via. New group members are checked against the license seat limit as one batch before any is created. Auth cache publish drops log the key count instead of the key. * fix(scim): lock member rows while recomputing admin roles after a group push * fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member * fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member * fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time. The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team. A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup. * fix(user-delete): keep the team alias label on the team members gauge after removals * test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them * fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404 * fix(scim): answer 404 and roll back when the team is deleted during a roster write * test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast * test(scim): cover deleting a SCIM group through the team delete's bulk path * fix(scim): evict a deleted group's member caches in bulk and drop cached roles after a recompute * fix(scim): answer 404 to a group delete that lost a race to another delete * fix(scim): chunk the locked team lookup and delete in the team delete transaction * fix(scim): keep service-account keys and team models on a group delete and write its records once under the lock * fix(scim): demote a deleted group's members inside the delete transaction The admin-group role write now runs on the team delete's transaction, over the roster read under the team lock, so a failed delete leaves roles untouched and a member added while the delete waited is demoted with the rest. The delete transaction also waits for a free connection as long as it may run, so a burst of deletes queues instead of failing after Prisma's two-second default. Every member in the 5-vs-500 statement test now holds a key. * fix(audit): write deleted keys' audit rows in one insert A delete that removes many keys wrote one audit insert task per key, and those tasks held pool connections the delete's follow-up writes waited on. The race test now waits on any lock wait behind the delete. |
||
|
jesus-berri
|
c8f7e0de7d
|
feat(model hub): show admin-defined custom tags and metadata from model_info (#46064)
* feat(model hub): show admin-defined custom tags and metadata from model_info Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): move model hub custom field merging into router_utils Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(model hub): share the custom tag badge list between the column and details Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(model hub): cover public hub, patch, null, hostile, forwarding, and killed-worker paths for custom fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model hub): drop custom_metadata entries with non-string keys or non-finite numbers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(model hub): use isinstance narrowing instead of TypeGuard for custom_metadata values Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(model hub): build custom tags and metadata in the proxy instead of the router Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model hub): serve custom tags and metadata on model_group_alias rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * remove docstring from model_group_custom_fields.py * style(model hub): fix lint and format in custom metadata tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(model hub): drop leading blank line in custom fields helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
480ee653c5
|
fix(responses): send the billed background response cost to the creating key's and team's loggers (#47506)
* test(responses): the billed background response cost reaches the creating key's and team's loggers * fix(responses): send the billed background response cost to the creating key's and team's loggers * fix(responses): bill an unreadable or rejected creator through proxy loggers, redact and honor disabled callbacks on the billed event * fix(responses): follow the creator's message logging setting on the billed event and read team logging for non-key creators The billed event no longer forces redaction. It follows the creating key's or team's turn_off_message_logging, else the proxy's, and redacts when the creator cannot be read or its settings are rejected. A creator without a virtual key hash bills through its team's settings. function_setup keeps its original callback split, and the billed path uses a separate non-mutating helper. * fix(responses): bill rotated, deleted, and custom auth creators through their own logging and retry failed creator lookups * fix(responses): follow a key's whole grace-period rotation chain to the live key for its billed charge's logging * fix(responses): bill a background response whose creator cannot be read, without the creator's settings * docs(enterprise): say the background response row is claimed before its creator is read * fix(responses): keep the loaded creator rows' budgets and aliases when one creator lookup fails |
||
|
joshua-berri
|
4aec80f5ee
|
fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap (#47034)
* fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap MCP SDK 2.x reads Streamable HTTP responses through httpx2's SSE parser, which rejects any single event over 1 MiB, so a large tools/call result failed with 'SSE stream ended without a response'. mcp 2.3.0 exposes max_sse_event_size, and MCPClient now passes a 64 MiB default, configurable with LITELLM_MCP_MAX_SSE_EVENT_SIZE_BYTES Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): fall back for invalid SSE event limits * test(mcp): verify configured SSE limits and recovery * chore(mcp): remove redundant dependency comment --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
dd511a2b22
|
fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)
* fix(logging): fire one CustomLogger hook per async call outside the async flag list aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are missing from is_sync_litellm_request's flag list, so the sync success and failure handlers called a CustomLogger's sync hook on top of the async hook the async handler had already called for the same request. The sync handlers now skip a CustomLogger's log_success_event and log_failure_event when that run is paired with an async handler run (for_async_request) and the async handler gives the logger its own async hook. The async client wrapper, dispatch_failure_handlers, the proxy-only failure dispatch and the native bridge's sync failure step for an async call mark their sync failure run as paired. Resolves LIT-9486 * fix(logging): keep the sync CustomLogger hook when no async success runs The async client wrapper skips async success logging for calls carrying a fallbacks kwarg, and the native bridge skips it for internal and fallbacks calls, yet the paired sync success run still dropped a both-hooks logger's sync hook, so that logger got no success event at all. The sync success run now suppresses the sync hook only when the async success handler was actually scheduled for the call: the client wrapper passes async_handler_runs=not is_completion_with_fallbacks, and the native host records per call whether it enqueued or deferred the async success. Tests that pinned the doubled failure hook on native aocr, messages and responses calls and on proxy-rejected requests now expect the async hook alone. The integration test adds /v1/messages and /v1/responses cells and runs the key-auth rejection on every route. * fix(logging): type dispatch_success_handlers kwargs The new async_handler_runs keyword on success_handler made the untyped **kwargs that dispatch_success_handlers forwards an unknown argument for it, which grew the reportUnknownArgumentType count past its merge-base ceiling. Typing the forwarded kwargs as Any, as _success_handler_body already does, removes those diagnostics without changing what is forwarded. * test(integration): cover CustomLogger hook counts across modes, chaos and router strategies Moves the hook count rig into tests/integration/_support/hook_counts.py and extends the cells it drives: - streamed chat, Messages and Responses requests, success and failure - loggers the async handler never reaches (sync-only overrides, sync event lists only) keep one sync hook per request on every surface outside the async flag list, key-auth rejections included - a concurrent burst while the logger's destination stalls and then fails, and a burst while a worker is killed, never give a request two hooks - least-busy keeps counting a request in flight while async speech requests finish around it, and usage-based routing v2 counts an async image request's tokens once * test(integration): Datadog and LangSmith deliver one record per async request Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local sink and checks every async surface outside the flag list: one Datadog log entry and one LangSmith run per request (two on the merge base), and never two OpenTelemetry spans, which its own emit-once guard already ensured. * fix(logging): give a both-hooks logger its sync hook when the async run never reaches it The sync hook was skipped whenever the logger sat in the async callback list, so a logger the async run never reached got no hook at all: a sibling stalled past the logging worker timeout, a batch poll that skips default logging, or an async failure run cut off. The sync run now holds the hook and the async run hands it back unless it reached the logger * fix(logging): run the held sync success hooks when the proxy drops a blocked response's success A post-call guardrail block drops the parked async success, which left the sync success hook held for it unrun. Base gave a both-hooks logger that hook on these call types, so the drop now ends the pairing and the held hook runs as before * fix(logging): run held sync hooks only after every sync run of the request has closed A held hook released by the async run could run on another thread while the sync run was still walking its other callbacks. The pairing now waits until the async run has ended and every sync run has closed, and the last sync run to close runs the held hooks itself * fix(logging): end the async run when its coroutine is dropped before it starts A logging queue that drops the async success coroutine, or a deferred success that fails to schedule, left the sync hooks held for it unrun. The enqueued async run now ends when its coroutine is discarded, and the proxy ends it when firing the deferred success raises * fix(logging): leave a still-queued async run to the exit flush The discard finalizer ran at interpreter exit before the logging worker's exit flush, so a queued request could get its held sync hook and then its async hook * test: keep the router and file-stream tests in step with main Main gives each owned proxy a private Redis by default, so the router cells share the worker's Redis to read its counters, and the file-stream stub takes the hook pairing the stream now passes * fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up A during_call guardrail that raised after the LLM call parked its deferred success skipped the proxy's flush, and async runs the exit flush never started were never ended, so a both-hooks logger lost its sync hook in both cases * fix(logging): skip the sync failure hook when the async failure run logs to the logger An async call outside is_sync_litellm_request's flag list ran both failure handlers, so a CustomLogger with both failure hooks got both. The sync run of an async request now skips log_failure_event for a logger that overrides async_log_failure_event and is registered for async failures. The four sync runs of async failures (dispatch_failure_handlers, the @client async wrapper, the proxy-only failure thread and the Rust host) pass for_async_request=True. Drops the HookPairing fallback. Successes are left to #47192, which already gives such a logger only async_log_success_event. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
e6065b9f4f
|
fix(bedrock)!: apply the deployment guardrailConfig on /bedrock pass-through routes (#47429)
* fix(bedrock): apply the deployment guardrailConfig on the /bedrock pass-through routes The router-model pass-through dropped a Bedrock deployment's guardrailConfig, so converse, converse-stream, invoke and invoke-with-response-stream reached Bedrock unguarded while /v1/chat/completions on the same deployment was blocked. The signer now sends it the way Bedrock reads it, in the body for Converse and as the X-Amzn-Bedrock-Guardrail* headers for Invoke, replacing any guardrail the caller sent * test(bedrock): reject version-less guardrails in the scripted Bedrock and cover the 400 The mantle and fallback fixtures named a guardrail with no guardrailVersion, which real Bedrock refuses, yet the scripted Bedrock answered them as guarded successes. The fake now answers a version-less guardrail with a 400 ValidationException, the fixtures carry versions, and a new case checks a version-less deployment guardrail reaches Bedrock and the caller gets its rejection on all four actions --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
78a6eeb212
|
fix(anthropic): keep tool names over 64 chars when a guardrail rewrites a /v1/messages request (#47434)
* fix(anthropic): keep tool names over 64 chars when a guardrail rewrites a /v1/messages request The guardrail translation shortens long tool names for the OpenAI-shaped guardrail input and wrote the shortened names back into tools and tool_use blocks, while tool_reference blocks and tool_choice kept the original names, so Bedrock rejected the request * fix(anthropic): restore declared tool names only from the request's own tool list A history tool_use name whose shortened form matched a declared short tool renamed that declared tool, so a forced tool_choice no longer matched and a tool allowlist saw a different name than the one forwarded * fix(anthropic): keep a tool_use name the caller already used as written A history tool_use named exactly like the shortened form of another long name was rewritten to that long name; such an ambiguous alias is now left alone --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Emerson Gomes
|
74230b4d50
|
fix(logging)!: redact Responses instructions and callback output (#43315)
* fix(logging): redact Responses instructions and callback output * fix(logging): avoid copying Responses output twice * fix(logging): return redacted Responses output without mutation * fix(logging): type Responses redaction payloads * fix(logging): redact streamed Responses instructions * fix(logging): redact request-side Responses instructions in logged params and spend log bodies * fix(redaction): keep per-callback opt-out off shared instructions and skip redacted replays A callback that sets message_logging=False no longer strips instructions from sibling callbacks, the spend log, or the client's response. Dict logging surfaces keep the reasoning config, and previous_response_id no longer sends the redacted placeholder upstream as the system prompt. * fix(redaction): redact instructions in an opted-out callback's own copy A callback with message_logging=False now gets instructions redacted in its kwargs copy and response object, while sibling callbacks and the spend log keep them * fix(redaction): give a message_logging false callback its redacted copy on sync calls The sync log_success_event and log_stream_event branch now hands an opted-out callback the same copy as the async path. * fix(redaction): redact a message_logging false callback's sync copy and keep failure estimates whole Give an opted-out callback a copy with messages, input, prompt, the sync stream's complete_response and its response object redacted, and stop the proxy's failed request estimate from counting the redaction marker in place of real instructions * test(redaction): wait on callback events instead of sleep polls The new redaction tests now wait for each callback to log through an event it sets, so they no longer poll on a timer * test(redaction): expect redacted follow-ups to replay no instructions This PR stops bridged previous_response_id follow-ups from reusing instructions the spend log stored redacted, so the integration test from #45923 that asserted the replay now asserts that neither the original instructions nor the redaction sentinel reach the provider * test(proxy): type the redacted-instructions failure estimate test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c4360bbb59
|
fix(harness): set PWD to the working directory of local sandbox commands (#47530)
* fix(harness): set PWD to the working directory of local sandbox commands LocalSandbox started CLIs in the sandbox workdir but passed the parent's PWD through. OpenCode reads PWD as its project directory, so a litellm.agent() turn worked in the Python process cwd instead of the sandbox. * fix(harness): re-home resumed OpenCode sessions into the sandbox folder OpenCode files each session under the folder it ran in and only streams a resumed session's events from that folder, so a session saved before PWD pointed at the sandbox, or resumed in another sandbox, hung. Resume now runs opencode export and import in the sandbox first. Also types the CLI handler's config so its member accesses are known. * fix(harness): fail an OpenCode resume that cannot move its session, with a timeout The export and import step now has a 30 s bound, and a failure or timeout raises HarnessError instead of leaving the resumed turn waiting forever. A restart of a session the handler already ran skips the step. * fix(harness): resume an OpenCode session elsewhere as a copy and fail the turn when it cannot Importing the exported session moved it, so the original stopped working where it was saved. Resume now imports a copy under fresh ids, filed under the sandbox folder, and continues the copy. A failed or timed-out export or import ends the next turn with runtime_error, as base does for a missing session, instead of raising from agent_resume. * fix(harness): retry a failed OpenCode resume on the next turn and keep resumed copies stable Resume now only records the saved id, and the first turn runs the OpenCode export and import under the caller's timeout. A failed export fails that turn and the next turn tries again. The local sandbox kills a command when its run is cancelled, and the copy names the sandbox folder so OpenCode 1.15.5 and older keep it there. * test(harness): keep subprocess tests out of tests/unit The PWD regression test moves to tests/integration/sdk with an isolated child interpreter, and the cancel test checks collect_output with an in-process process instead of a real one. collect_output takes the sandbox Process protocol. * fix(harness): rewrite only id fields when copying an OpenCode session for resume |
||
|
Mateo Wang
|
81d8c9af38
|
fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508)
* test: bill each completed request once and price unpriced returned decision models at the requested model Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(logging,decisions): bill each request once and price returned decision models only when they carry rates Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(decisions): leave the double-billing fix to #47510 and keep the decisions pricing and missing-answers fixes * fix(decisions): name malformed answer entries and pick the Arize cache-hit span by attribute * test(decisions): cover a returned-model row with metadata but no rates * fix(decisions): keep upstream reply text out of the parse error so it stays a 500 * fix(decisions): answer every malformed upstream reply with a 500, whatever text it carries A reply that fails the decisions response schema now raises DecisionsResponseValidationError, which the decisions call maps straight to InternalServerError instead of passing it through exception_type. Its text matching read an answer key like rate_limit, or a value like "Request timed out", as a 429 or a 408, so the router cooled down a deployment for a reply that fails the same way on every retry. The message also drops the empty location for a reply that is not an object, the Arize family test loses the cache_hit entries main already added, and the Bedrock Mantle count-tokens wire test calls _metadata_deployment, since main renamed the _deployment helper under it. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
e78f7b4602
|
test(integration): repair seven stale integration tests on main (#47600)
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply |
||
|
devin-ai-integration[bot]
|
2773ef0084
|
test(integration): remove BUG skip for 1 test that passes on main (#47603)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
7e8be61fa0
|
test: fix flaky team delete, refusal sink, loop-blocking, Moyai tamper and model-cost leak tests (#47135)
* test(integration): stabilize auth and team delete checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): make loop-blocking and tampered-signature checks deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): isolate model-cost state and synchronize loop progress Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(unit): fail loop-progress handshakes when the blocking call never starts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
b245cc0d26
|
feat(proxy): size-triggered spend log cleanup, oldest rows first (#47450)
Adds general_settings.maximum_spend_logs_size, maximum_spend_logs_cleanup_target_size and maximum_spend_logs_size_check_interval, the spend_log_size_check job, maximum_spend_logs_cleanup_cron precedence, oldest-first bounded deletes down to the target, storage fields in the run summary and the litellm_spend_log_storage_bytes gauge, with a Postgres integration test covering the post-VACUUM no-refire case Resolves LIT-9831 Fixes #44589 Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Max Azatian
|
208cac49f0
|
fix(vertex_ai)!: send integer enums to Gemini instead of dropping them (#44908)
* fix(vertex_ai): send integer enums to Gemini instead of dropping them
Gemini rejects enum values that are not strings, so _fix_enum_types removed
every non-string enum. Integer enums are now sent as type INTEGER with
format "enum" and string values, which Gemini accepts and enforces.
* fix(vertex_ai): keep converted integer enums when a tool is reused
The schema is edited in place, so a tool definition passed again already
carries the converted enum. Keep an integer enum that is already in the
format "enum" form instead of dropping it, and type the new locals.
* fix(vertex_ai): use Sequence at runtime in the enum cast
* fix(vertex_ai): treat integer-string enum values as an integer enum
A reused schema already holds its integer enum as strings. Instead of
reading `format` to recognise that, accept integer strings as well as
integers on an integer-typed enum and send both in Gemini's form. This
drops the untyped `schema.get("format")` read.
* fix(vertex_ai): map Gemini tools from a copy so a fallback keeps the caller's schema
Gemini tool mapping rewrote the caller's tools in place, so a router fallback to another provider received the Gemini-converted integer enum (string values with format enum) and returned the integer as a string
* fix(vertex_ai): send integer enums only to Gemini 3 and newer, and only ones Gemini accepts
gemini-2.5-flash stalls for minutes on a larger integer enum, so Gemini 2.x
keeps main's behavior and drops it. An enum outside int64 or longer than 1000
values is dropped too, since Gemini answers 400 for those. A nullable integer
enum written as a type list now keeps its values like the anyOf form does
* fix(vertex_ai): drop integer enum strings longer than int64 instead of failing the request
* fix(vertex_ai): drop integer enums when a request's enums pass Gemini's size limit, and send int enum members as numbers
Gemini limits the total size of all enums in one request, so the per-enum 1000-value cap still let several large integer enums through to a 400. Integer enums are now sent only while every enum value in the request adds up to 2000 characters or fewer, and an (int, Enum) member goes out as its number instead of its name
* fix(vertex_ai): send only integer enums Gemini returns exactly, and count them in Gemini token counts
* fix(vertex_ai): send Gemini integer enums without format enum, which made gemini-3.8-flash think much longer
Also skip the response cache in the repeated-request integration test, since copying tools makes the second identical request a cache hit
* fix(vertex_ai): keep dropping integer enums in Vertex batch rows, which Vertex batch cannot parse
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
2484f5dd6d
|
test(integration): call the renamed metadata deployment helper in the Mantle count_tokens wire tests (#47592) | ||
|
devin-ai-integration[bot]
|
52d5a51fcb
|
test(cost): run azure_ai claude bridge pairs, add anthropic usage speed and vertex gemini audio priority price (#47582)
* test(cost): run the azure_ai claude bridge cost parity pairs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): add anthropic usage speed and vertex gemini audio priority price Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b55024658c
|
fix(proxy)!: stop update_cache from resurrecting evicted auth cache entries (#46997)
* fix(proxy): stop update_cache from resurrecting evicted auth cache entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update update_cache pins for synchronous write-back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): wrap long assert in update_cache tag test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover refill race and pin TTL with injected clock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count global proxy spend for every request with a user, cached on this worker or not * fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter The global proxy spend bump read the total, awaited, and wrote old + cost back from a background task, so a budget reset that deleted the key in between got the pre-reset total written back, and workers overwrote each other's totals. RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key exists, keeping its TTL, and this worker's copy is then set from the shared total with no await between its read and write. The bump now runs whenever a proxy max_budget is set, matching the DB writer, and the armed post-call read and the batch-read declaration it used are removed. * fix(proxy): report the global spend Redis increment to service metrics and request traces The global proxy spend increment now emits Redis service success and failure events under the spend_counters target and passes the request span through, so it shows as redis.incr spend_counters like the write it replaced. The unit tests run the real RedisCache increment against fakeredis instead of mocks, and a new test checks the span lands under the request span on success and on a Redis failure. * test(proxy): keep the spend counter Redis fake off the network * test(proxy): count increment_if_present as a shared-cache call in the key-family guard * fix(proxy): keep the newer global spend total when Redis replies arrive out of order --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
f925c067d2
|
fix(tests): read owned proxy rows from the owned proxy's database (#47333)
* fix(tests): read the stream stall cooldown spend rows from the owned proxy's database The cooldown cells boot an owned proxy, which writes its spend logs to its own scratch database since owned proxies went private, while the cells polled the worker's database and never saw a row. Read each row through the gateway that served the request, so the cells check the proxy they called. * fix(tests): read passthrough spend rows from the owned proxy's database The vertex regional uplift and passthrough stream pricing spend files boot private owned proxies, then polled the worker database for spend rows that only reach the proxy's own database. Read them through the owned proxy's gateway instead. Also hold the stream stall file's hold_headers peer past the longest row wait, so a slow spend row cannot fail the peer teardown. * fix(tests): check agent deletion on the owned proxy's database The agent least privilege test deletes its agent through an owned proxy, then checked the worker database for the row, which is always empty there. Read it through the owned proxy's gateway so a delete that leaves the row fails the test. * fix(tests): read turn admission spend rows from the owned proxy's database |
||
|
Kerry Lu
|
5ab0130abc
|
test(integration): call _fallback_deployment in the count_tokens wire tests #47363 added (#47593)
#47525 renamed _deployment to _fallback_deployment; #47363 was written against the old name and merged after it, leaving three F821 undefined-name errors that fail the python lint job on main Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
047cb4a8e2
|
test(integration): expect the Exa search envelope's provider_metadata after #47423 (#47572)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
0ede0e10e7
|
fix(batch): preserve recoverable costs and prevent poll starvation (#47190)
* fix(batch): preserve recoverable costs and rotate polling pages * test(batch): type retirement failure regression * fix(batch): alternate new arrivals with retry polling * test(batch): defer polling row annotation on Python 3.12 * test(batch): reconcile fixtures with atomic metadata claims * refactor(batch): simplify reconciliation to one bounded scan * fix(batch): restore arrival priority during retry scans * fix(batch): preserve recoverable jobs on unrelated resource errors * fix: revisit recent batches before old retry backlog --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
91724bb0a7
|
fix(responses)!: count input tokens the way inference sends the request (#47363)
* fix(count-tokens): translate chat tool use before Claude token counts Responses and chat tool use (function tools, function_call items, tool messages) is now converted the way Claude chat inference converts it before the Anthropic, Azure AI and Vertex partner counters call count_tokens. Translation errors answer 400 before any provider call. Anthropic-shaped requests and /v1/messages/count_tokens keep their body as sent. * test(count-tokens): compare Claude count bodies with inference for Responses tool use * fix(responses): forward the caller's Responses body to OpenAI token counts OpenAI deployments now get the caller's /v1/responses/input_tokens body as sent, minus LiteLLM-only fields, so OpenAI's own count and errors come back. previous_response_id is checked for ownership like inference and routes to the deployment that served it * fix(responses)!: count Claude Responses bodies the way inference sends them Non-native Responses counts now run the body through the inference bridge, so text.format, tool_choice, reasoning, built-in tools, and file_search count the way inference sends them on Anthropic, Azure AI, Vertex, and Bedrock. The OpenAI native count body uses the drop_params-resolved tool fields. * fix(responses): honor drop_params and replay previous turns in Claude counts Claude input_tokens counts now read the same drop_params setting inference merges from the deployment, router defaults, and request, so a forced tool_choice downgrades to auto under drop_params. previous_response_id replays the stored turns through inference's session loader, and chat messages carrying fields beyond role and content go through the provider's chat translation * fix(responses): keep proxy-only request keys out of OpenAI counts and map chat tools in Bedrock counts * fix(count-tokens): import on Python 3.11 and keep stored prompts in OpenAI counts under drop_params * fix(responses): leave tool_choice out of counts when drop_params drops every tool * fix(count-tokens): count Mantle options, drop cache marks, and reject input inference cannot send The Bedrock Mantle fallback now counts thinking, tool_choice and output_config like the runtime count. Claude Responses counts leave out cache_control marks again, so more than four marks count instead of failing. Counts answer 400 before any provider call for user blocks chat inference refuses and for Responses tools with no Chat Completions equivalent, instead of dropping the block or counting a placeholder tool. Adds the wire tests for these paths * fix(count-tokens): count Responses tools the way the inference bridge sends them A Claude count now hoists additional_tools input items into tools, strips a malformed tool flag under drop_params and rejects it without, and leaves tools out when the deployment lists tools in additional_drop_params, so the count matches the body Claude inference sends for the same request * fix(count-tokens): keep Bedrock runtime counts option-free and send OpenAI the provider file id * test(count-tokens): capture the Anthropic count request at the HTTP layer instead of patching the handler * fix(count-tokens): ignore a caller-sent file id mapping and keep non-string file ids as sent |
||
|
devin-ai-integration[bot]
|
4963ba4d99
|
test(translation): add sonnet 5.5 anthropic direct cross-model mid row (#47566)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Max Azatian
|
5145250d83
|
fix(langfuse_otel)!: send the completion start time (#45597)
* fix(langfuse_otel): send the completion start time langfuse_otel never set langfuse.observation.completion_start_time, so Langfuse showed no time to first token for any call, streamed or not. The legacy langfuse callback already sends kwargs["completion_start_time"]. Send the same value as an ISO timestamp with an explicit offset. * fix(langfuse_otel): type kwargs and test against a real span Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any], the declared type of model_call_details, so the new completion start time read adds no unknown-type errors. The tests now check a real SDK span instead of patching safe_set_attribute, and the parameter is typed. * fix(otel): send the completion start time from the v2 Langfuse mapper With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper, which had no completion start time either. Carry the payload's completionStartTime on LLMCallSpanData for streamed calls, the same calls that get time_to_first_chunk_seconds, and map it to langfuse.observation.completion_start_time. * fix(langfuse_otel): take the logging kwargs as a read-only Mapping The new kwargs annotation tripped LIT001. Both helpers only read the logging kwargs, so type them Mapping[str, Any] instead of dict. * fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter * fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON The native Gemini, Interactions and A2A streams never recorded a first chunk, so with completion_start_time now sent both emitters showed TTFT equal to latency on them. They now record it, and Gemini and Interactions take the call's start time like /v1/messages. Both emitters send the completion start time as a JSON string, the form the Langfuse SDK and the langfuse callback send, through one serializer. * fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event A2A streams took their start time when the iterator was built, so their latency left out the work before it. They now use the call's start time like every other route. Cached /v1/messages stream replays recorded no first event, so their time to first token fell back to the end time. The replay now records it on the first event it yields. The tests use the real Interactions config and typed logger hooks. * fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free * test(a2a): type the card fetch test and stop rebinding the seed helper's recorder * test(otel): expect completion_start_time and cache_hit in the B3 span keys --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d776277647
|
fix(responses): count billed background responses toward the creator's project and per-model budgets (#47504)
* test(responses): billed background responses count toward the creator's project and model budgets
* fix(responses): carry the creator's project and per-model budgets on the billed background response
* fix(responses): count a key's budget tier toward background per-model budgets
* fix(responses): bill a background response's project only while the key stays on its team
* revert(responses): keep billing the creating key's project after a team move
A key can only keep or detach its project, so the billed project always pairs with the team it was created under. This reverts commit
|
||
|
Mateo Wang
|
f5d6e89ad8
|
test(integration): follow #47441 metadata-routed counting in count_tokens wire tests (#47525)
* test(integration): follow #47441 metadata-routed Mantle counting in count_tokens wire tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): build the runtime count body without mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
8d2e7dca26
|
fix(sagemaker)!: parse every streamed record however SageMaker cuts the payload parts (#47495)
* fix(sagemaker): parse every streamed record however SageMaker cuts the payload parts SageMaker cuts a container's response stream into PayloadParts at arbitrary byte offsets, so one part can hold several SSE events or JSON lines, or only part of one. The decoder treated each part as one record, so vLLM's [DONE] was logged at ERROR on every stream and a split at an unlucky byte dropped the rest of the answer. Peel records off a running buffer with JSONDecoder.raw_decode, skipping SSE framing and [DONE] between them. * test(sagemaker): cover streamed records cut or merged across payload parts * fix(sagemaker): decode UTF-8 across parts, skip SSE comment lines, and log container error records A UTF-8 character split across two PayloadParts now decodes whole through one incremental decoder per stream. SSE comment lines (TGI's keep-alive) and event, id and retry field lines are skipped as framing. A container error record is logged at ERROR with its message and skipped, and any other record that is not a chunk is logged and skipped, the same way on the sync and async paths. * fix(sagemaker): raise a container's error record and ModelStreamError on both stream paths A record with a non-null error now raises a 500 SagemakerError carrying the record after the chunks before it, on sync and async. Async no longer swallows a non-200 event, so a ModelStreamError raises as it does on sync. A chunk with a null error field is delivered, and any other record that is not a chunk is logged and skipped on both paths. * fix(sagemaker): answer a streamed error with the container's status and fail a stream cut mid-record A container's error record now raises with the status the container gave it (vLLM's error.code, the older flat vLLM shape, TGI's http_status_code or error_type) instead of a fixed 500 that the router retried as a 503. A ModelStreamError raises a SagemakerError naming its ErrorCode and Message, a stream that ends inside a record raises instead of ending as a stop, and a skipped record is logged by its keys only * fix(sagemaker): trust only 422 and 429 from a streamed error record and fail LMI generation errors * test(sagemaker): drive an LMI failed-generation stream through the proxy on a sagemaker/ deployment * test(sagemaker): fall back to the next model when a stream fails before any content |
||
|
Mateo Wang
|
dfac770e2b
|
feat(proxy): map baggage values to budget tags (#47358)
* feat(proxy): map baggage values to budget tags * fix(proxy): preserve mapped tags in the parsed request cache * fix(proxy): preserve mapper accounting and tag policy * fix(proxy): preserve baggage tags across native transports * fix(proxy): keep mapper rejection and passthrough logging safe * fix(proxy): scope baggage tag mappings to billed inference and keep them out of tag routing Mapped tags now apply only to billed LLM API routes, skipping token counting, model listing, MCP and A2A. They classify spend and budgets but never select deployments unless a key, team or client set the same tag. WebSocket inference returns 403 only while a required rule exists. A tag created with /tag/new resets its spend counter so its budget counts from creation. The General settings editor keeps a new mapping unsaved until every required field is filled in. * fix(proxy): keep client-tag rejection identical with baggage, map A2A calls, skip request previews * fix(proxy): keep mapped tags out of requester metadata, answer 503 on tag lookup errors, map custom pass-through routes, check the key before the WebSocket 403 * fix(proxy): keep caller-sent tags routable, keep mapped tags out of policy and guardrail matching, honor allow_requests_on_db_unavailable, reject control characters in baggage values * test(proxy): cover baggage tag mappings end to end Adds integration cells for request_tag_mappings across chat, messages, responses, A2A and custom pass-through spend, required and budgeted rules, the WebSocket refusal, unbilled routes, reject_clientside_metadata_tags, tag routing and policy isolation, a dropped tag read with and without allow_requests_on_db_unavailable, and the /tag/new spend counter reset * fix(proxy): keep base tag merges for requests that map nothing, accept JSON-string metadata tags, count litellm_metadata caller tags Requests with no mapped baggage now reuse the cached body exactly as before, so header and key tag merges still normalize metadata for routing. The client body copy is made only when a tag maps, and the base merges are mirrored onto it. A request that maps nothing no longer gets a 400 for non-object metadata. caller_tags now counts tags sent in both metadata and litellm_metadata. Optional mappings no longer reject JSON-string metadata tags under reject_clientside_metadata_tags. * fix(router): keep baggage-mapped tags from selecting tag-scoped pre-routing strategies Auto, complexity, adaptive and quality router selection, the consumed-tags stamp and compression policy lookup now read request tags without baggage-mapped tags, unless the caller sent the tag or the key, team or project carries it, matching deployment tag filtering. * feat(proxy): parse baggage headers into request tag mappings * fix(proxy): match plus-sign baggage keys and keep deployment tags in guardrail modes * fix(proxy): keep baggage-mapped tags behind key, team and deployment tags and serve non-object or invalid DB mapping settings like unset Mapped tags are added after the key, team, deployment and caller tags, so a tag-mode guardrail picks the same mode with or without baggage. A non-object metadata field maps like empty metadata, and an invalid request_tag_mappings value loaded from the DB is ignored with an ERROR log instead of failing every request * fix(router): merge deployment tags in linear time and keep malformed client tags as sent under mapped baggage Merge serving deployment tags into the request tags in one pass, so a large client tag list no longer blocks the event loop. Leave a non-list metadata.tags untouched when baggage maps a tag, so the request gets the same answer as without baggage. Add the C17 integration cell for the team tag's guardrail mode under a mapped deployment tag. * test(spend): cover string metadata.tags under mapped baggage * fix(proxy): check mapped tag budgets when metadata.tags is a non-list or holds non-strings Mapped baggage tags are applied again after the header and key tag merges, so a header or key tag that replaces a string metadata.tags no longer hides the mapped tag from the tag budget check. An empty or false metadata.tags is read like a missing one, and a tags list with non-string items is kept as sent with the mapped tags added after it, instead of failing auth with a 401 |
||
|
Mateo Wang
|
270d108ace
|
fix(anthropic): json_schema envelope edge cases on the Anthropic json tool path (#47478)
* test(anthropic): cover json_schema envelope edge cases on the Anthropic json tool path * fix(anthropic): unwrap required-only keys, walk past tool-named envelopes, keep non-ASCII text, flush cut streams, check Vertex output_format * fix(anthropic): send held json tool text on cut Bedrock Invoke streams and drop a non-object output_format under drop_params A Bedrock Invoke stream that ends before the json tool block closes now sends the held text and closing brace, as the direct Anthropic stream already did. A non-object output_format on the Anthropic messages route and its providers is dropped under drop_params and otherwise forwarded, so the provider answers its own 400. Vertex Claude names the proxy setting in its 400 and reads drop_params through the shared helper. Adds integration cases for the json_schema envelope and the cut stream. * fix(anthropic): drop a non-object output_format on chat, unwrap a whole envelope on a cut stream, escape lone surrogates Chat completions to Anthropic, Azure AI, Bedrock Invoke and Vertex now drop a non-object output_format under drop_params, matching /v1/messages. A stream cut after a complete held envelope is unwrapped like a normal end. A json_schema answer holding a lone surrogate keeps its \u escape so the response can still be encoded * fix(anthropic): join a split answer when the stream ends in its last block A cut during the second json tool block now runs the held block through the same flush as a block stop, so it joins the streamed keys or the earlier split blocks instead of sending two objects back to back. The Vertex output_format fold drops its unreachable drop_params branch, since both parent configs drop a non-object value first. * test(vertex): expect the LiteLLM 400 for a non-object output_format on /v1/messages |
||
|
Mateo Wang
|
d698106ed7
|
fix(router)!: schedule default_priority requests once and reject a malformed priority (#47498)
* fix(router): schedule default_priority requests once and reject a non-integer priority A router configured with default_priority failed every request that carried no priority with a 500 naming a missing 'priority' argument, and an explicit priority re-entered acompletion and hit the same error. acompletion now resolves the priority once (the request's integer, else the default, with 0 kept as a real priority) and schedules through _schedule_factory directly, while schedule_acompletion delegates to acompletion so the two paths share one queue pass. A non-integer priority answers 400 naming the param, or is dropped under drop_params, instead of leaking to the provider or crashing the scheduler. * fix(router): honor router-level drop_params and schedule prompt-management models under a priority * fix(router): drop a non-integer priority when the model's deployments set drop_params * fix(router): resolve the drop_params deployment scope for team routes and deployment ids * fix(router): read the request and router drop_params flags without a loop-assigned Final * refactor(router): type the prompt-management wrapper's return and mock the provider in its test * perf(router): consult drop_params for the priority only when it is not an integer * test(integration): assert the new priority contract for non-integer and zero priorities * docs(router): describe default_priority as the priority every request without one takes * fix(router): parse default_priority once and apply it to text completions and the queue route A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig. * fix(router): type the priority dispatch so the lint budgets hold * fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order * fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses * fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority A request whose model is a deployment id now counts that deployment as healthy in the scheduler, since routing by id already ignores cooldown. Before, the scheduler found no healthy deployment for an id, so the request waited behind every entry ahead of it. A boolean default_priority (true, or YAML off) is ignored at startup with a warning, like any other value that is not a whole number. The scheduler integration tests now share worker state for the defaulted pair, cool a group through /v1/responses so no stale in-memory queue is left behind, and use "high" instead of "1" for the non-integer cases. The concurrent burst test no longer asserts that the queue drains, matching the explicit-priority burst test, since the scheduler's read-modify-write queue can race across workers. * test(router): keep the queue drain check as a BUG-skipped integration test * fix(router): queue default_priority chat requests once instead of failing them * test(router): open with the default_priority regression tests alone * fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name |
||
|
yucheng-berri
|
7ae9a1c350
|
fix(router)!: only prefixed caller tags route once tag_routing_prefix is set (#47428)
* fix(router): scope tag routing to prefixed tags once tag_routing_prefix is set With router_settings.tag_routing_prefix configured, an unprefixed caller tag (for example a spend tag such as user_id:234) no longer drives tag routing, so it can no longer refuse a model group whose deployments are untagged. Key/team/project tags (metadata.inherited_tags) keep applying with or without the prefix. Rows of batches the proxy runs itself now carry those tags as inherited_tags too, so they keep routing on them once the prefix is set. Takes over #39903. Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com> * test(integration): pin that proxy-run batch rows keep key/team tag protection * refactor(router): filter routing tags from the raw request tags; deterministic tests and key-tag cells * test(integration): audit cells for tag_routing_prefix: project tags, forged inherited_tags, odd tag values, outage, fallback, user-agent routing, concurrent burst * test(integration): proxy-run batch rows keep key and project tags too --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com> |
||
|
yucheng-berri
|
f03f98ac22
|
fix(search): price Tavily advanced and Linkup deep searches at their own cost map entries (#47411)
* fix(search): price Tavily and Linkup searches at their per-mode cost map entries * test(search): read the reported cost through the public hidden_params property * fix(search): keep the type gate flat on the cost map membership check * test(search): drop vendor price guards and a redundant case, type helper params as mappings * fix(search): keep a registered search model's own price, cover every interception endpoint, hostile modes and a failing burst * fix(search): keep a runtime-registered base search price for modes that have no runtime price of their own --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
ekpog200
|
ba61ecdf57
|
fix(videos)!: keep model_id in proxy video ids so per-model keys resolve on status/content (#44614)
* fix(videos): keep model_id in proxy video ids so per-model keys resolve on status/content
Provider transforms encode returned video ids with an empty model_id, so
GET /v1/videos/{id} and /content cannot resolve the deployment (and its
per-model api_key) and fail with invalid_api_key.
Re-stamp the returned VideoObject id with the resolved model_id in
video_generation, video_status, video_remix, video_edit and
video_extension.
Fixes #33423
* fix(videos): return a copy instead of mutating the response, read hidden params via helper
* test(videos): fully type the new video id re-stamping tests
Annotate every parameter and return added by this PR (hidden_params,
harness, response, -> None), drop Any from the test helper, mark locals
Final, and narrow the endpoint/helper results with isinstance(VideoObject)
instead of reading .id off an untyped value. Route the mocked processor
response through a typed Harness.respond_with, read NotRequired
DecodedVideoId keys with .get, and name the rule and reason on the
_hidden_params suppressions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): fill only an empty model in returned video ids
Keep an id that already names a model as is, so the asked-for name,
routing group and served group in it still drive access checks and
routing. Fill an empty model from the model the caller asked for
before the routed deployment id, and read the request model through a
typed view so the type gate does not grow
* fix(videos): route video calls to the deployment that created the video
Video ids now carry the id of the deployment that created them. Status,
content, remix, edit and extension calls go to that deployment when it
belongs to the requested group, so a group spread over two provider
accounts no longer polls the wrong one. A provider model shared by
several groups resolves to the creating deployment's group, and the
asked-for model is written into an id only when the router can route it.
* fix(videos): pin video calls only to deployments the caller may use
Pin a video id's creating deployment at deployment selection, and only when the caller may use that
deployment in the requested group, so aliases, routing groups and access groups still apply and the
spend label and fallbacks keep the requested group.
A status poll keeps the deployment its id names and takes the deployment that answered when the id
names none. An alias that routes to the creating deployment's group is kept on status and content.
* test(router): reset module fallback globals between unit tests
* fix(videos): check and route polls by the name the router serves
A poll on a video created through a provider wildcard is checked against the name the wildcard serves, a team model is checked by its public name, a poll stamps the deployment that answered, and the sync Router pins video calls the same way the async path does
* fix(videos): pin polls only among the deployments routing allows
Video calls route by group again and the creating deployment is picked only from the deployments left after access, team, health, cooldown, budget and tag filters, so a pinned poll no longer skips them. Wildcard groups now pin too. Adds tests for cooldown, wildcard pinning and character lookups through a provider wildcard, and return types on the auth tests
* fix(videos): scope the video pin to video calls, keep fallback-served videos reachable, skip cooldown on video 404s
* fix(videos): stamp the requested group only when one of its router fallbacks served the create
* fix(videos): follow chained router fallbacks for fallback-served videos, test with a real Router
* test(router): list the video pin helper with its sibling in the router coverage exemptions
* refactor(auth): walk chained router fallbacks without recursion
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Lai Wei
|
bda4e60f9b
|
fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions (#44349)
* fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt Read chunk fields as typed attributes instead of untyped subscripts, which clears the reportUnknownArgumentType breach the lint gate flagged * fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): rebuild dict-form text completion chunks again stream_chunk_builder routes dict chunks holding TextChoices to the text completion builder, which now reads attributes. Convert them to TextCompletionResponse at the routing point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage * fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild * fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions * fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null * fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`. The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text` next to `delta`, so streamed /v1/responses calls to Together ended without response.completed. Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about the same as rebuilding an n=1 stream of the same size. * fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals * fix(streaming): rebuild text completion choices without an index at index 0 Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base * fix(streaming): rebuild text completion choices with a string index under that int A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string * test(streaming): drop a docstring that repeats its test * fix(streaming): fall back to choice 0 for an infinite text completion index A float infinity index made int() raise OverflowError, which the index coercion did not catch, so the rebuild failed where the merge base returned choice 0. --------- Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
da80552897
|
test(translation): add sonnet 5.5 converse and gemini direct cross-model mid rows (#47328)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c4d24ec2f1
|
test(integration): remove BUG skips for 3 tests that pass on main (#47364)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
daa133da5c
|
fix(mcp)!: bill both model calls of a chat completions MCP auto-execute on one spend row (#47422)
* test(mcp): cover one spend row billing both rounds of a chat completions MCP auto-execute * fix(mcp): bill both model calls of a chat completions MCP auto-execute on one spend row * fix(mcp): keep provider usage fields when summing both chat rounds of an MCP auto-execute * fix(mcp): log each auto-executed MCP tool call under its own spend-log id A chat MCP auto-execute logged its tool calls under the request's call id, which is also the id of the request's failure row. When the follow-up call failed, the database kept the tool row and dropped the failure row that bills the paid first call * test(mcp): cover the margin, cache, failure row and parallel tool rows of MCP auto-execute billing |
||
|
Mateo Wang
|
96f1bc02d8
|
feat(router)!: apply each deployment's reasoning_effort_levels on every provider and route (#46601)
* feat(router): apply each deployment's reasoning_effort_levels on every provider and route A deployment's declared reasoning_effort_levels now decide the effort sent upstream on /v1/chat/completions, /v1/messages and /v1/responses for every provider. Requests round up by default, litellm.reasoning_effort_rounding and model_info.reasoning_effort_rounding switch to down, a declaration lets providers like azure_ai accept reasoning_effort, and each remap logs one INFO line with structured fields. * fix(router): round reasoning effort only on a deployment's own declared levels Cost map levels no longer round requests or force-allow reasoning_effort, so undeclared sibling deployments and catalog-only models behave as on main. A declaring deployment lets azure_ai accept any level the resolver sends, and a hosted_vllm chat thinking budget is rounded too. * fix(router): drop undeclared reasoning effort under drop_params and round native /v1/messages effort Under drop_params a level outside the deployment's reasoning_effort_levels is dropped instead of sent. Providers that take /v1/messages natively now round output_config.effort onto the deployment's declaration. The rounding setting ignores case and surrounding spaces. * fix(router): strip line breaks from every field in the reasoning effort remap log * fix(router): round thinking-budget efforts onto a declaring deployment on every provider * fix(router): keep the caller's thinking display when rounding a thinking budget on adaptive Claude Warn on a reasoning_effort_rounding value other than up or down instead of silently rounding up * test(integration): cover deployment reasoning_effort_levels rounding on chat, messages and responses Wire cells assert the exact body each provider receives when a deployment declares reasoning_effort_levels, plus undeclared controls, malformed input, drop_params, rounding settings, fallbacks, model groups, live updates and the response cache. Chaos cells cover a killed worker, a proxy restart, an upstream outage, a slow stream and the INFO remap log. * test(integration): cut held streams when the upstream stops in the reasoning effort chaos cell The upstream-stop cell waited for the held streams before stopping the upstream, so they only ended on a client or proxy timeout and the result depended on which fired first. The upstream now stops while the streams are held, and each cut stream must carry an error and no answer * fix(router): round thinking budgets only on declaring deployments and round Converse output_config An undeclared hosted_vllm deployment serving a Claude model name now maps a thinking budget the same as any other undeclared hosted_vllm deployment. A model_info that echoes a cost map key no longer borrows that entry's levels. A non-numeric thinking.budget_tokens answers 400, or is dropped under drop_params, where main answered 500. A declaring Bedrock Converse Claude deployment now rounds output_config.effort, which /v1/messages sends for Claude Code * test(integration): cover output_config rounding and non-numeric thinking budgets in the reasoning effort cells * test(integration): keep an explicit reasoning_effort winning over a malformed thinking budget on hosted_vllm * test(integration): cover output_config rounding on openai, azure and azure_ai chat * fix(hosted_vllm): let an explicit effort win over a malformed budget and reject one on /v1/messages An explicit reasoning_effort beside a non-numeric thinking.budget_tokens keeps main's behavior again. A declared hosted_vllm deployment with a Claude model name maps a budget through the generic table instead of the adaptive Claude one. /v1/messages to a non-Claude target hands a malformed budget to completion(), which answers 400 or drops it under drop_params, where it answered 500 * fix(router): drop thinking beside an explicit effort when a declaring deployment's provider cannot read it * test(integration): cover malformed thinking budgets on /v1/messages to non-Claude targets * fix(anthropic): answer 400 for a malformed thinking budget on /v1/messages to any non-Claude model, or drop it under drop_params * test(integration): cover an explicit effort beside thinking on azure_ai and drop_params on the /v1/messages bridges * fix(anthropic): answer 400 for a malformed thinking budget through adapter_completion without drop_params * fix(anthropic): type the adapter_completion request so the drop_params read adds no unknown argument * fix(router): a default effort beside a thinking budget sends no effort on chat Chat now leaves the budget-derived effort out when the caller's effort resolves to none or asks output_config for one, matching /v1/messages. The chat thinking decision moves into resolve_chat_reasoning so completion() stays under basedpyright's complexity limit. * test: type respx_mock and mark the adapter result Final in the reasoning effort tests * fix(router): keep the caller's thinking beside a kept effort and the budget effort beside a dropped one A non-default output_config.effort beside a thinking budget now leaves the thinking as sent, so Bedrock Invoke keeps display updates. A level drop_params drops counts as never sent, so the budget still sets the effort. output_config.effort replaces a thinking budget the provider cannot read, and a non-numeric string budget on adaptive Claude answers 400 in place of 500, or drops under drop_params. * fix(router): send a kept output_config effort as reasoning_effort where the provider cannot read thinking On a declaring hosted_vllm or azure_ai deployment, a thinking budget goes out as reasoning_effort. An output_config.effort beside it now takes that slot, rounded onto the declared levels, instead of leaving only output_config, which those backends ignore. * fix(reasoning-effort): honor global drop_params for a non-numeric budget and keep the budget effort beside a dropped /v1/messages level * fix(reasoning-effort): send a level a gpt-5 deployment declares even where the cost map does not list it * fix(reasoning): apply declared levels on azure_ai gpt-5 and reject a bool thinking budget on adaptive Claude * fix(router): drop a thinking budget's effort that a deployment declaring no ranked levels excludes A deployment declaring [] or ["none"] skipped the budget check, so adaptive Claude still sent the budget-derived output_config.effort under drop_params on chat, Bedrock Converse, and native /v1/messages. The adaptive thinking now goes out without that effort * fix(anthropic): reject a false, null or empty thinking budget on adaptive Claude |
||
|
Mateo Wang
|
2d181e6633
|
fix(token_counter): count DeepSeek V4 text with DeepSeek's own tokenizer (#47353)
Local token counts for DeepSeek V3 and V4 model names now use DeepSeek's own bundled tokenizer instead of cl100k_base, and tool definitions on V4 names count with V4's chat-template tool framing |
||
|
Yassin Kortam
|
ce4a6cbf5b
|
perf(streaming): cut per-chunk pydantic round trips on openai-compatible streams (#47030)
* perf(streaming): skip full-chunk dump and delta rebuild on openai-compatible chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(streaming): reuse the built choice and delta on plain openai-compatible chunks Co-authored-by: yassin <yassin@berri.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): copy upstream chunk attributes in upstream key order The scalar fast path iterated a frozenset of keys, so the SSE key order of chunk extras like service_tier and obfuscation changed with each worker's hash seed * test(streaming): count chunk_creator calls with a profiler instead of patching classes * test(streaming): check service_tier still passes through chunk_creator * fix(streaming): reuse the built choice for chunks with upstream choice extras * test(streaming): cover ModelResponseStream chunks on the choice reuse path * fix(streaming): keep main's delta rebuild for deltas with nested values Clearing role in place on a delta that holds tool calls left its nested fields unset, so chunk.model_dump() dropped keys like a custom tool call's name on later chunks. Deltas holding only scalars keep the in-place path. * perf(streaming): build each plain OpenAI chunk's delta once and skip pydantic attribute misses Later plain ChatCompletionChunks now build one Delta and one StreamingChoices, with no full-chunk model_dump, and read stored values instead of attribute probes that raise and catch AttributeError. Nested-value chunks no longer pay the dict(model) misses that made them slower than the merge base. * fix(streaming): read hidden params a chunk carries as a pydantic extra * test(streaming): pin logprobs, Azure filter extras, and audio and image deltas on later OpenAI chunks * test(streaming): cover role, reasoning, choice extras and billing on plain OpenAI SDK chunks --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
edf4097baf
|
fix(databricks): stream Claude json_schema answers as text on every streaming path (#47424)
* fix(databricks): stream Claude json_schema answers as text on every streaming path
Async and proxy streams on Databricks Claude json_schema returned the
internal json_tool_call tool call with empty text, because the async
stream factory dropped json_mode. Databricks now passes it through, the
json tool's argument fragments stream verbatim as content (a lone {}
fragment was blanked before, corrupting nested empty objects), and the
stream ends with finish_reason stop like the non-stream answer.
* test(databricks): keep the json_schema stream regression tests ahead of the fix
Holds the Databricks transformation at the merge base so the first approval covers the regression tests; the fix commit follows.
* fix(databricks): stream Claude json_schema answers as text on every streaming path
* fix(databricks): keep finish_reason tool_calls when a json_schema stream also calls a user tool
* fix(databricks): drop text beside the json tool and renumber user tools in Claude json_schema streams
With reasoning_effort set, Claude can write prose before json_tool_call or call it next to a user tool.
Streams now hold text until the end and drop it when the json tool was called, renumber user tool
deltas past the dropped json tool so indexes start at 0, and non-stream returns the json tool's
arguments as content next to the user tools.
* fix(databricks): stream JSON-looking text at once and keep prose ahead of user tools in Claude json_schema streams
* fix(databricks): hold a user tool call behind held prose so a later json tool answer does not glue to it
A json_schema stream with thinking on that wrote prose, called a user tool, then called the json tool sent the prose at the user tool call and appended the JSON answer to it, so content did not parse. User tool deltas behind held prose are now held with it and sent merged per index at the json tool or the finish, and a json tool after text already sent is dropped so the answer is not repeated.
* test(databricks): cover json_schema answers on every endpoint in the integration wire cells
|
||
|
devin-ai-integration[bot]
|
1660c8ba1f
|
feat(key_management)!: apply default_key_generate_params.allowed_routes on key generation (#41236)
* feat(key_management): apply default_key_generate_params.allowed_routes on key generation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(key_management): avoid mutating default routes payload Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): apply default allowed_routes to key_type presets it covers * fix(proxy): keep management and read_only presets, skip service account keys, and show the default routes to custom_key_policy * fix(proxy): keep Full Access keys out of the default allowed_routes, let owners re-pick a key type, and show custom_key_policy the member key's routes * fix(proxy): keep owners from changing routes a key got from default_key_generate_params * docs(proxy): describe key_type when default_key_generate_params.allowed_routes is set, and test custom_key_policy input with the default off * test(proxy): type the default_key_generate_params allowed_routes tests * fix(proxy): let key owners send unchanged allowed_routes back on /key/update * fix(proxy): ignore a route default with a non-string entry and match resent routes exactly A default_key_generate_params.allowed_routes list with a non-string entry made /key/generate fail at the database, so it is now ignored like a non-list. An owner's resent allowed_routes on /key/update skip the route check only when they equal the stored list in order, so reordered or duplicated lists are checked as a change. Adds integration cells for the default and the owner update paths. * test(proxy): cover a non-string allowed_routes default in the key integration tests * test(proxy): read key rows from the owned proxy's database in the default routes integration tests * fix(proxy): leave allowed_routes out of a key update that resends them unchanged --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
MFJG
|
783cf06f31
|
fix(sdk): decide MCP gateway routing without importing the proxy handler (#45961)
* fix(mcp): keep the gateway handler off the SDK path when no tool is an MCP tool completion(), responses() and anthropic messages imported the MCP gateway handler whenever tools were passed, and responses() did so on every call. Since the handler began importing the MCP catalog at module level in #42568, that import needs the optional mcp package, so a base SDK install failed with ModuleNotFoundError on any function-tool call and on every responses() call A new import-safe pre-check skips the handler unless some tool is a dict of type mcp, which is the only shape the handler's gateway decision can accept. MCP gateway tools still reach the handler as before Adds in-process regression tests for the three entry points and a function-tool completion to the base SDK install check * fix(mcp): leave one-shot tool iterables unread in the gateway pre-check responses() declares tools as an Iterable and the handler scans the same object a second time, so a pre-check that read a generator would hand the handler one tool fewer. The pre-check now scans only collections and reports anything else as possibly naming the gateway, which keeps the previous path The Anthropic messages regression test now mocks the httpx layer with respx and turns off the aiohttp transport, instead of replacing AsyncHTTPHandler * fix(mcp): scan only lists and tuples in the gateway pre-check The pre-check read any Collection, but a Collection that yields its items only once would still be exhausted by the handler's second scan. Only the list and tuple builtins are guaranteed to iterate the same way twice, so they are the only inputs read now. Everything else keeps the previous path and is reported as possibly naming the gateway The unread-input test now covers a generator, an iterator, a one-shot collection and a deque * fix(mcp): scan only exact list and tuple types in the gateway pre-check isinstance accepted list subclasses, and a subclass that iterates only once would still be consumed by the handler's second scan. The fast path now reads only objects whose type is exactly list or tuple. Every other input, subclasses included, keeps the previous path and is reported as possibly naming the gateway The unread-input test gains a one-shot list subclass and a tuple subclass * fix(sdk): decide MCP gateway routing without importing the proxy handler The pre-check now holds the handler's exact gateway test, so remote MCP tools, mcp 1.x installs and any tools iterable stay off the handler, and the base SDK check calls responses() with a remote MCP tool. * fix(sdk): send /mcp/<name> remote MCP URLs to the provider when the gateway registry cannot load A path-only /mcp/<name> match now counts as a gateway tool only when the MCP server registry imports, so SDK installs without the proxy extras pass remote MCP servers like Zapier to the provider. Also drops a dead test patch. * test(sdk): send a /mcp/<name> remote MCP tool in the base SDK install check * style(sdk): drop docstrings from the MCP gateway pre-check and its tests * test(mcp): cover /mcp/<name> gateway routing on the proxy and Messages tools on a base SDK install --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
6c4c134f23
|
fix(secret_managers)!: resolve oidc/file and oidc/env refs in admin-configured api_key per request (#47341)
* fix(secret_managers): resolve oidc/file and oidc/env refs in admin-configured api_key per request An api_key set to oidc/file/<path> or oidc/env/<VAR> in a credential_list entry or a deployment's litellm_params was sent upstream as literal text, so every call answered 401. The SDK client wrapper now reads the referenced file or env var on each call and never writes the token back, so a rotated token is used on the next call. The existing server-owned federation gates treat an oidc/ api_key as server-owned by value, so a request body, a non-admin credential or model write, user_config, fallback targets and routing tier params can never carry or name one. Other oidc/ kinds answer 400 * fix(secret_managers): strip only the leading oidc/ from a secret reference A token file under a directory named oidc resolved to the wrong path, because every oidc/ in the reference was removed * fix(auth): refuse oidc/ api_key refs in advisor tools and scope the route gates to them The advisor tool's own api_key now answers a 400 when it holds an oidc/ reference, since its sub-call runs through the client wrapper that resolves one. Stored fallback targets, auto-router tiers and user_config refuse only an oidc/ api_key, inline or through the credential they name, so their workload identity federation handling stays as it was. Provider model discovery for a wildcard deployment resolves the reference before calling the provider. * fix(auth): allow admin oidc api_key on deployment writes, resolve refs on discovery, batch probe and realtime health, refuse client api_base on oidc deployments * fix(utils): resolve oidc api_key refs only from admin-configured sources Mark an oidc/ api_key read from a proxy-level deployment or a stored credential, and resolve only that marked value. Team deployments, prompt templates, nested configs and a direct SDK api_key send the text as written. An unsupported reference kind raises a 401 so the router fails over to the group's next deployment. * fix(utils): send a credential's oidc token only to its own api_base host A stored credential's oidc/ api_key now resolves only for the scheme, host and port of the credential's own api_base or base_url, so a team model or vector store that names the credential with another api_base gets a 401 and that host gets nothing. Saving a team model with an inline oidc/ api_key answers 400, since team rows never resolve one. * fix(proxy): send an admin oidc api_key only to the hosts configured beside it Every admin oidc/ reference now carries the origins of the host params set beside it, checked for api_base, base_url and the Bedrock, S3, SageMaker and deployment_url params. Token counting reads the reference only when it calls the provider, answering 401 when it cannot, and pass-through 401s redact internal paths. * fix(proxy): redact the token path from token count auth errors * test(security): cover token count, Bedrock endpoint, api_base update and pass-through path redaction for oidc api_key refs * fix(proxy): redact the token path from secret reference errors where they are raised * fix(proxy): keep the secret reference's path out of the error a caller sees The 401 for an unreadable oidc api_key or pass-through credential now names only the reference kind and a fixed reason, so a token path under /run/secrets or a custom credential dir never reaches the caller, while the server log keeps the full reference for the operator * fix(secret_managers): strip line breaks from the logged secret reference A reference holding a newline can no longer forge a server log line * fix(secret_managers): label an unknown oidc reference kind instead of echoing it An api_key such as oidc/env_MY_VAR (a typo for oidc/env/MY_VAR) came back in full in the caller's 401 and in the spend log, since the kind was whatever text followed oidc/. A kind get_secret does not know now reads oidc/<unknown kind>. The relative-path hint uses oidc/file//<absolute path>, which the client-message redaction on chat routes leaves whole. * test(health): run the real model auth check in the oidc test connection test * test(secret_managers): read an oidc/file token stored under a directory named oidc * fix(secret_managers): count a URL in the model name as an upstream host for oidc api_key references A huggingface/* wildcard call could name its own endpoint URL as the model and receive the resolved token. The origin check now treats a URL in the model as a host the reference must allow. The integration contract also reads the proxy's own database instead of the pytest worker's. * fix(secret_managers): treat a default port as explicit when matching an oidc api_key origin A credential on https://api.mistral.ai refused a call to https://api.mistral.ai:443, though both reach the same server. The origin check now compares scheme, host and effective port, as assert_same_origin does, and an unparseable api_base is refused instead of raising. The advisor test sets the real proxy flag instead of replacing our own helpers. * test: type the oidc request-body helpers and drive the tier test through mock_response * fix(secret_managers): refuse an oidc api_key reference whose call drops the configured host |
||
|
yuneng-jiang
|
923b7d1784
|
test(integration): read router cooldown failure rows from the proxy that served them (#47457)
test_router_cooldown_failure_row.py boots an owned proxy, which writes to a private database since #47213. The file was written on a base without #47213 and read its rows from the worker database through read_rows, so all 13 cases timed out with State did not converge: [] Every row read now goes through Gateway.rows on the owned proxy's gateway, the same fix #47410 made for the stream stall cooldown cases |
||
|
Joshua Wang
|
2754da5391
|
fix(friendliai): drop litellm-internal message fields before sending (#45092)
* fix(friendliai): drop litellm-internal assistant fields before sending Claude Code replays thinking blocks on every turn after the first. The Anthropic adapter turns them into thinking_blocks plus reasoning_content on the chat assistant message, and FriendliAI rejects any message key it does not know, so multi-turn Claude Code sessions on friendliai/ models failed with a 500 after the first tool call FriendliAI accepts reasoning_content and GLM reads it back, so only thinking_blocks and provider_specific_fields are dropped. together_ai already had this exact helper as a private function, so it moves to common_utils next to strip_litellm_internal_message_fields and both providers share it Related to #32188 * test(friendliai): cover a replayed tool call turn The regression fixture is now the shape Claude Code replays after a tool call: an assistant turn with thinking_blocks, tool_calls and no content, followed by the matching tool result. The test asserts both survive the sync and async request paths unchanged * ci: retrigger checks * ci: retrigger benchmarks after runner timeout * chore(friendliai): mark inherited list signatures as mutable-ok The type-discipline gate on main now caps LIT001 at the merge-base count, so the _transform_messages overloads need the same inherited-contract marker together_ai uses for its identical override * ci(test-unit): add tests/unit/llms/friendliai to the llms-providers shard * test(friendliai): check the outbound FriendliAI body through the proxy for chat and messages replays Both replay shapes go through a real proxy to a scripted FriendliAI upstream, and the test reads the request body FriendliAI received. It fails at the merge base, where thinking_blocks and provider_specific_fields reach the upstream, and passes with the field stripping in place. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
93b79d2587
|
fix(anthropic): apply tool_config cache-control points on chat and /v1/messages (#47394)
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages A cache_control_injection_points entry with location tool_config was spread into the Anthropic request body on /v1/chat/completions, which Anthropic rejected with 400 Extra inputs are not permitted, and the native /v1/messages path dropped the point silently. Both paths now put the point's control on the last tool Anthropic caches and never forward the key. Co-authored-by: Tan Nguyen <tan@pressingly.net> * refactor(anthropic): validate tool_config inputs instead of casting them Keeps the lint gates flat: the helper takes untyped tools and points and narrows them with the existing object validators, the chat transform builds the body with the marked tools instead of writing into optional_params, and the handler reads a typed view of kwargs. * fix(openai): keep tool_config cache points out of extra_body, add integration test The cache hook hands a tool_config point on for the provider step, and for providers that call through the OpenAI SDK every non-OpenAI param rides in extra_body, so OpenAI answered 400 Unrecognized request argument supplied. The new integration file drives a live proxy against scripted Anthropic, Bedrock Converse and OpenAI upstreams for /v1/chat/completions and /v1/messages. * test(integration): cover tool_config cache points on bridged /v1/messages Three new cases on /v1/messages for models without a native Anthropic messages path: OpenAI through the Responses bridge and Gemini through the chat bridge never see cache_control_injection_points in the body, and Bedrock Nova through the chat bridge gets the cachePoint on toolConfig --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Tan Nguyen <tan@pressingly.net> |
||
|
Vansh Singh
|
f4347d4f50
|
fix(speech)!: correctly propagate spend logging and usage for Gemini TTS bridge (#44551)
* fix(speech): correctly propagate spend logging and usage for Gemini TTS bridge * fix(speech): call Gemini once per TTS request and log the bridge's token usage aspeech re-ran speech whenever the first result was not a coroutine, so the speech-to-completion bridge sent every Gemini TTS request upstream twice. It now returns the sync result. The bridge's inner completion runs as an internal sub call, so only the speech call fires success callbacks, and the completion's usage rides on the audio response so the spend log and usage_object carry real token counts. Replaces the separate inner Logging object and the cost_calculator fallback, which never reached the spend row. * fix(speech): flag a cached Gemini TTS reply as a cache hit so it logs at no charge * fix(speech): price a cached Gemini TTS reply's savings from its token usage and keep its cache key * test(integration): cover Gemini TTS speech billing, logging, cache hits and deployment TPM * test(integration): start the speech TPM cell in a minute's first 10 s so the two requests never cross a minute --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |