mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-15 09:12:18 +00:00
190 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
580203cce6
|
fix(otel): send the Langfuse environment, release and version on OTel v2 (#51385)
* fix(otel): send the Langfuse environment, release and version on OTel v2 * fix(otel): keep the Langfuse environment on the Langfuse exporter only * fix(otel): annotate the exporter resource attributes as a read-only mapping * fix(otel): shed the Langfuse release and version first at the attribute limit * fix(otel): keep base attributes when the Langfuse release and version do not fit - the model span sheds release and version before the identity fields - the request root takes release and version only when they fit under its attribute limit - an exporter routed to a tenant's credentials drops the proxy's Langfuse environment - a langfuse.environment the proxy's resource already names wins over LANGFUSE_TRACING_ENVIRONMENT * test(otel): annotate the new attribute-limit and routing tests |
||
|
paveln83
|
9fba44d300
|
fix(openai): cap and bill web search rounds on Responses-pinned chat completions (#51802)
* fix(openai): cap web search agentic loops with max_agentic_loops on OpenAI chat completions The OpenAI chat dispatcher ran every custom logger's chat agentic loop without a depth check, and the web search follow-up never carried the loop depth, so max_agentic_loops was ignored on openai/ deployments. The dispatcher now skips loggers the shared chat completion loop in litellm.acompletion already runs, so those go through its depth, cap, and repeated-call checks once. Loggers that only implement the chat-specific hooks still run in the OpenAI dispatcher * fix(openai): honor max_agentic_loops when chat completions bridge to Responses A gpt-5.4+ chat completion with function tools on api.openai.com is sent to /v1/responses by the Responses bridge. The Responses handler then ran its own agentic loop on that call, which never saw the request's max_agentic_loops or depth, and the shared chat completion loop in litellm.acompletion ran again on what it returned, so every chat follow-up restarted the Responses loop at depth 0. With max_agentic_loops=1, web search made 8 Responses calls and 7 searches The async bridge now marks its Responses call as part of a chat completion turn, and the Responses handler skips its own loop for such calls. The shared chat loop owns the whole turn, so its depth, cap, and repeated-call checks run once * fix(responses): keep search follow-ups on the Responses API for responses/ pinned models When an async chat completion crosses the Responses bridge, the shared chat loop now owns the turn and builds each web search follow-up from the model recorded on the logging object. responses() overwrote that record with the stripped model, so a model pinned with the responses/ prefix (for example openai/responses/<model> on a Responses-only gateway) sent its follow-up to chat completions, and the turn ended on the unexecuted tool call responses() now leaves the record alone when the chat loop owns the turn, the same way completion() skips its own record when the reverse bridge owns it * fix(agentic-loop): bill every round of a non-streamed Responses-bridged chat loop once A bridged chat request logs its first round before the chat loop runs, so a follow-up run as an internal sub-call was never billed. Each follow-up of a non-streamed bridged parent now writes its own spend row, and a failed follow-up no longer re-bills rounds that already wrote one * fix(agentic-loop): keep streamed Responses-bridged web search from billing its final round twice * fix(openai): bill every bridged search round in one row after guardrails On the proxy a non-streamed Responses-pinned chat request now parks one spend row that sums every round the loop served, so the row and the x-litellm-response-cost header show the whole request and nothing is written before post-call guardrails pass. A streamed request keeps round one's row and folds every later round into the replayed final row. The bridge now skips only Responses hooks that run their own chat loop, so hooks that only handle the Responses API still run. * fix(openai): price each bridged search round at its own size Each round of a bridged chat request is now priced on its own usage and the combined proxy row adds those prices, so summed usage never crosses a long-context price tier. A streamed replay keeps that total and its cost breakdown * fix(agentic-loop): run the chat hook of bridged callbacks that define both agentic hooks The bridge skipped the Responses hook of a callback that runs its own chat loop, and the shared loop only asked the general gate, so a custom callback with both hooks stopped intercepting bridged chat. Ask its chat gate after the shared loop. Price tier tests now use test-owned gpt-5.6 prices. * fix(agentic-loop): route bridged chat to the chat hook only for callbacks that opt in A custom callback that defined both agentic hooks lost its Responses hook on chat completions bridged to the Responses API, so a chat hook that declined left the request unhandled, and one that claimed it changed which hook answered compared with main. Callbacks now opt in with wants_chat_completion_hooks_on_bridged_chat. Web search interception sets it, so its loop still runs once on the chat side. Every other callback keeps main's routing through its Responses hook. * fix(websearch_interception): keep main's Responses loop for a bridged web_search_options call --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
giladk1221
|
9d827093c5
|
feat(github_copilot): support GitHub App authentication with PEM upload (#51427)
* feat(github_copilot): support GitHub App installation authentication * Add GitHub App authentication to the Copilot credential form * Explain secure private-key setup beside Copilot App fields * Store uploaded Copilot App keys through encrypted named credentials * fix(github_copilot): handle App validation errors and credential replacement * fix(ui): make GitHub App field help accessible * fix(github_copilot): preserve saved App key references on edit * fix(github_copilot): trust litellm's CA settings when minting App tokens and hide App keys from /health * fix(github_copilot): keep device-login headers, mint App tokens off the shared thread pool, cache App responses Device-login requests keep the merge base header behavior, so caller headers still replace the Copilot defaults. Only App requests merge and pin their identity headers. App token refreshes are single-flight per cache key with a shared future. Only the refreshing caller uses a thread, so waiting callers no longer fill the default executor that every provider call shares. A cached App token is served before the key secret is read. Uploaded keys still rotate the token when their contents change. App deployments are deployment-owned, so they share the response cache like shared device-login deployments. App credentials are validated even when credential_info names no provider. * fix(github_copilot): finish an App token refresh even when the caller that started it is cancelled The caller that owns a refresh used to await asyncio.to_thread, so cancelling it while the job was still queued cancelled the job, left the in-flight future unfinished, and every later request for that App waited on it. The refresh now runs in the executor without being tied to the caller, and every caller, the owner included, awaits the shared future * fix(github_copilot): refuse deployments naming an unloaded credential, keep App key references, accept numeric App IDs A Copilot deployment whose litellm_credential_name is not loaded now answers 400 instead of falling back to the shared device login. The router registry keeps github_copilot_app_private_key_ref values as references, a YAML App ID can be an integer, the server-owned field error names Copilot auth for Copilot fields, and the credential edit modal shows the stored provider again * fix(github_copilot): report a deployment naming an unloaded credential as unhealthy The /health check now answers unhealthy with the same 400 the request path returns for a Copilot deployment whose litellm_credential_name is not loaded, instead of running it through the shared device login * fix(github_copilot): keep the base fallback for an unloaded credential and report the App host as api_base A Copilot deployment that names an unloaded credential uses the shared device login again, as on base. A deployment that sets github_copilot_auth_type: github_app still refuses before any GitHub call. get_api_base now returns the Copilot host an App or per-user session calls instead of an admin api_base it ignores. * fix(github_copilot): count a GitHub rejection of the App installation token as a Prometheus deployment failure * fix(github_copilot): keep per-user session api_base reporting as on base and name Copilot in the non-admin 403 Only an App deployment reports its App host as api_base before a configured api_base. Per-user OAuth sessions report the same api_base they did on base. The 403 for a non-admin attaching App credentials now names GitHub Copilot authentication. The token-dir fixture in the App api_base test is applied with usefixtures. * fix(github_copilot): report the configured api_base instead of raising when GitHub App settings are incomplete * test(github_copilot): type the incomplete App settings test parameter --------- Co-authored-by: Gilad Katz <29776916+giladk1221@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
3ad164382b
|
fix(responses): send cache_control as a prompt_cache_breakpoint on the chat-to-Responses bridge (#51783)
* fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode caching never wrote. The bridge now renames the marks before converting the messages when the served deployment reads breakpoints, using the same gate the Responses config uses for native /v1/responses input. Deployments that do not read breakpoints keep today's behavior. The rename also leaves unmarked content parts untouched instead of validating them, so a part with non-string keys no longer raises on either path. * fix(responses): keep leading system messages in client order once one goes to input A marked first system string became a list and moved to input while the next unmarked string stayed in instructions, so the model read them in reverse. The leading run that joins instructions now stops at the first system message whose content is not a string * fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge Bedrock Mantle already caches the growing prefix on its own, so turning cache_control into breakpoints there added write charges. The client-order rule for leading system messages now applies only when a mark turns a system string into a list, so unmarked chats send the same request as before. * test(responses): cover a marked user turn in the unmarked system split test A cache_control mark on the user message must not switch unmarked leading system messages to the client-order rule. The split test now runs with the user turn plain and marked, so dropping the role check from the system string test fails it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
quietjules
|
9ba216d521
|
fix(vertex_ai): reuse one Google token across RAG Engine searches and cold bursts (#51239)
* fix(vertex_ai): reuse one Google token across vector store searches * fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst * test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy * test(vertex_ai): type the fixtures and token callback in the concurrent search test * fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects A shared RAG Engine config made one store's slow token fetch hold up every other store, and kept a rotated key file's deleted key until a restart. First loads and expired refreshes now take a lock per credentials and project, refreshes take a lock per credentials object, and any RefreshError reloads the credentials once. * fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls * test(vertex_ai): check async callers sharing an expired credential refresh it once * fix(vertex_ai): reload Google credentials only when a refresh is rejected for good A retryable refresh error such as a 503 now raises without reloading the key, so an outage does not double google-auth's retries. Credentials that cannot be weak-referenced share the config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase and holds the token response until the other callers have started * fix(vertex_ai): reload Google credentials only when Google answers invalid_grant google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did * fix(vertex_ai): pick up a swapped Google key file on the next call The credential cache key now carries the key file's inode, mtime and size, for a vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of after Google rejects the old key or the proxy restarts * fix(vertex_ai): keep the earlier key while a replaced key file does not load A key file caught halfway through a non-atomic write, or replaced by one that does not load, now keeps the credentials loaded from its earlier version in use and logs a warning, as the cache did before key files were re-read. The next complete file is loaded on the next call. The pub/sub test no longer writes a key file it only reaches through the seeded cache * fix(vertex_ai): try a replaced key file again when its first token fetch fails The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one * fix(vertex): raise when an async reload keeps an earlier key Google rejected --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
cd2e4b6b60
|
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) * feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope * fix(custom-logger-v2): add field projections and keep the request scope across copies * fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line * fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists _safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones. * fix(custom-logger-v2): contain a failing v2 callback on logging hooks A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged. * fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path. * fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes. * fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected. * fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged. * refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow. * fix(custom_logger_v2): await a single-entry v2 stage in the caller's context A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather. * fix(custom-logger-v2): raise a native callback panic as a runtime error * fix(custom-logger-v2): run a synchronous v2 line on a private event loop * fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler * fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure * fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope * fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once * fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure * fix(callbacks): reject a selection name or alias that is not lowercase * fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks * fix(router): type the fallback event kwargs handed to the v2 dispatch * feat(custom_logger_v2): type the contract fields as records and mark derived-only fields Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy * feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers * feat(callbacks-python): hand the logging payload to native callbacks by reference NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy * feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity Typed members keep the bool, int, float or str python holds through a Scalar type on both sides, so ids, limits and priorities no longer lose values that lax pydantic refused. A member the contract cannot carry is named in rejected so a port can reproduce what python does with it. Logging hooks get logged_key_identity, built from the payload metadata the python loggers read on every route, while key_identity stays on request hooks and keeps reading the auth object. * fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways budget_reservation read the dict budget_reservation.py builds with a KeyError on callback_bound. Undeclared keys are now ignored, callback_bound is a member, entries keep a non-mapping item as null so capacity's model access group check stays exact, and costs read as the float budgets computes with float(x or 0.0). Float members cross as the raw json marker when they are not finite, and a native verdict returns through json_to_py, so an infinite or NaN cost survives a read and a write instead of turning into null. * feat(callbacks): carry a python value as its str() text and truthiness in scalar members Scalar gains a Stringified variant with the str() text and bool() of a value that is not a bool, int within i128, float or str, so a port can reproduce the label, path or log text python formats and the falsiness of an empty container without confusing it with a real string. Budget reservations say whether python held the entries as a list. * feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary A Scalar member now carries a list, dict, set, bytes, object or an int past i128 as its str() text and truthiness instead of naming it in rejected, which stays for values whose str() or bool() raises. request_tags carries any iterable as the items prometheus iterates, and budget reservation entries carry what budgets iterates with entries_is_list for capacity's list check, never written back to the shared dict. * feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int A Stringified scalar now carries hashable, from trying hash() at the boundary, so a port can raise the TypeError capacity's priority lookup and the limiter's descriptor identities raise for a list, dict or set and not for a tuple or object. It also carries integer, so an int past i128 is told apart from any other value whose text is digits. * fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does Prometheus treats any guardrail_mode that is not a list as a single mode, so a tuple mode holds no str mode and a missing mode is one None mode. The derivation unpacked tuples and rejected other non-str modes instead. * feat(callbacks): let a native handler await typed host calls the python caller serves * test(callbacks-python): drive a native handler's host calls through the python dispatcher * test(custom_logger_v2): route a native logger with a host through the execution driver * fix(custom_logger_v2): type the wrapped guardrail modes for pyright * feat(custom_logger_v2): carry the python type name and hash error text of a stringified value * feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it * feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none * feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs * feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one * fix(callbacks-python): render null keys like a python dict merge, adding a missing key last * feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar * feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload * feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end * feat(custom_logger_v2): carry the start and end times prometheus reads from the call details * feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call * fix(callbacks): export the limiter request record * test(custom_logger_v2): give the type name cases stable ids and sort the record exports * fix(custom_logger_v2): type the new derivations and tests for pyright * fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads * style(callbacks): rustfmt the record exports * refactor(custom_logger_v2): rename native callbacks to rust callbacks native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle, NativeState and HostedNative renamed to their rust forms. The pyo3 classes NativeCallback and NativeRequestState become RustCallback and RustRequestState, and the tests, fixtures and messages follow. * refactor(custom_logger_v2): rename slots to field sources slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A table that only derives is named DERIVED_*_FIELDS, and a table that can write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The include_v2 comment moves down to the functions that take it. * refactor(litellm_logging): place the v2 members of Logging beside what they serve v2_scope, v2_auth and v2_failure_delivered join the other per-request class defaults, v2_logging_scope and its opt-out helper follow should_run_callback, v2_call_facts sits by the standard logging payload builder, and claim_v2_failure sits above failure_handler. bind_v2_request_scope checks for a Logging object directly instead of two runtime protocols. * refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate * refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field A logged payload that is present but not a mapping now looks the same to a rust callback as a missing one, a parity gap accepted for a case that does not occur. metadata_labels, input_sequence_length and limiter_request move down to bits 44-46, on both sides of the contract. * feat(callbacks): type a rust callback's request state Callback gains an associated State type: request_state returns it and handle and handle_hosted receive Option<&State> beside the input, which no longer carries the state. ErasedCallback is the object-safe form the python wrapper stores; it carries the state type-erased and downcasts it once, returning CallbackError::StateType rather than panicking on a mismatch. handle_hosted's default names its unused host _host. The record sample count follows the dropped logged_payload_not_a_mapping field. * docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are * refactor(callbacks-python): render payloads with python-compat's json writers pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr. python-compat's json module now exposes write_string, generic over a String or byte sink, and float_text, and the payload renderer uses them. * docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet * docs(custom_logger_v2): add a README for the package * fix(custom_logger_v2): import the field sources tests from field_sources * test(callbacks-python): stub the litellm package in each payload test * fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check |
||
|
quietjules
|
f5ef14308b
|
fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved (#50516)
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved Converse used to drop any service_tier outside default, flex, and priority with no error, rewrite auto to default, and drop reserved. Now reserved and standard_only map to Bedrock tiers, auto sends no tier, and any other value answers 400 unless drop_params is on. Nova batch rows honor the row's drop_params. * fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows * test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs * fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params * fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params * fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase * fix(bedrock): mark the service tier block field read-only * fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params * test(prometheus): sort the service tier label test imports * fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400 * fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova * fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one * test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it * refactor(bedrock): drop the passthrough's own service tier header reader that main re-added The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Nate Armstrong
|
20b6e855bd
|
feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0) (#51214) | ||
|
ishaan-j-berri
|
46716fad56
|
fix(proxy): alert on customer org, customer team, and customer member budgets (#51579)
* fix(proxy): alert on customer org, customer team, and customer member budgets Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): scope customer alert dispatch and Redis tracing Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(alerts): escape customer identities in Slack budgets Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): dispatch customer budget events through custom loggers Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): read committed customer budget spend from writer Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): retain customer budget claims for the full period Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
joshua-berri
|
e2dfd3dad1
|
fix(langfuse): reserve span capacity for failure usage attributes (#51538)
* fix(langfuse): reserve span capacity for failure usage attributes * fix(langfuse): type pending failure attribute keys --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yujonglee
|
2be96aed13
|
refactor(redis): delete the Python RedisBatch now that every request op is native (#51383)
The request scope keeps only the native batches. RedisBatch, its ops, the cluster fallback, the Python post-call deadline and the DualCache post-call helpers had no production caller left, and the pre-call SET, DEL and miss tracking that UserApiKeyCache never reached go with them. The mixed-families pipeline metadata had no producer left |
||
|
tin-berri
|
9633be87ea
|
feat(ui): show customer organization usage and budget utilization (#51483)
* feat(ui): show customer organization usage by date range * feat(customers): align usage reads with daily spend contract * fix(customers): combine spend display and resolve CI failures * test(vector-stores): isolate search warnings from SDK verbosity * fix(ui): cap customer usage date ranges at 400 days |
||
|
ishaan-j-berri
|
df8b3b498e
|
feat(proxy): add customer hierarchy usage analytics (#51495) | ||
|
joshua-berri
|
a7052ea8a8
|
fix(cost): preserve Bedrock cross-region pricing for Messages (#51479)
* fix(cost): bill Bedrock cross-region profiles at the profile rate on /v1/messages Bedrock reports the region-stripped model id for a cross-region inference profile, and the cost selector preferred that served id, so streaming and non-streaming Messages spend (input, output, cache write and cache read) used the global row. A served id carrying a different cross-region prefix keeps its own price. Fixes #46286 * test(langfuse): derive the Bedrock profile export cost from the profile's model info The regional us. deployment is now billed at its profile rate, so the expected Langfuse cost comes from that profile's rates and the mock usage instead of a hardcoded global price. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
7130a5b4fd
|
feat(dynamic_rate_limiter_v3): prometheus share, borrowed and throttle metrics per team and model (#51417)
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits the request, a router fallback's settlement publishes the fallback target's own share under its model group, the gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests restore the process registry, and the integration test polls for the settled totals. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prometheus): type the gauge setter and the label factory so the type gate holds _set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the label test tolerates the now visible None values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(prometheus): keep the label test diff to the two None-tolerant assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
dc1284d3d2
|
fix(otel): distinguish missing usage from prompt estimates on chat failures (#51443)
* fix(otel): distinguish unknown timeout usage from prompt estimates * fix(otel): handle expected tokenizer failures and cover parent spans * fix(otel): validate failure usage inputs at typed boundaries * fix(otel): type Logging.messages for the failed prompt token estimate Declare Logging.messages as Sequence[object] | None at its single __init__ assignment and drop the list TypeAdapter, so the failure estimate reads a typed value without new unknown-type diagnostics. * fix(otel): skip the failure prompt estimate for prompts with attachments Only estimate the failed prompt when every message is plain text, so files, documents and images never reach PDF parsing or media decoding on the failure path. Those requests still report unknown provider usage without an estimate. * fix(otel): count the failed prompt off the event loop for async requests Async failure logging now awaits the existing offload_token_count limiter for the optional prompt estimate, and the sync half of an async request no longer counts inline, so a slow tokenizer load cannot stall other requests. * fix(otel): only count the failed prompt for loggers that receive failures The estimate check now reads general callbacks, the global failure lists through a typed get_failure_callbacks accessor, and the dynamic failure lists, so a logger registered only for success never triggers failure token counting. * fix(otel): return an immutable failure callback snapshot * fix(otel): keep malformed content blocks from breaking failure logging Only compare string block types against the text block table, and run the text-only check inside the estimate's narrow try, so a malformed prompt leaves usage unknown without an estimate instead of interrupting failure callbacks. * fix(otel): stop the token counter from debug-logging raw prompts The failure estimate counts the original prompt, so the counter's debug line could write prompt text to application logs while message logging was off. Drop that line and assert the prompt stays out of debug logs on both routes. * test(otel): pin text-only eligibility and failure callback categories Cover the public has_only_text_content contract for malformed and non-text messages, and check get_callbacks_by_type puts sync and async failure registrations in the failure categories through the public manager APIs. * fix(otel): keep failure logging when the prompt estimate cannot run Return no estimate when the offloaded count hits a worker or tokenizer RuntimeError or OSError, so failure callbacks still run, while cancellation still propagates. Treat SDK Message objects like their dict form when checking whether a prompt is text only. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
joshua-berri
|
d723545320
|
fix(otel): report the provider model in experimental GenAI traces (#51430)
* fix(otel): report the provider's response model under gen_ai_latest_experimental With OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, gen_ai.response.model now uses the provider_response_model kept on the response hidden params, then the deployment model when the response only echoes the client alias. Legacy mode and litellm.model_group keep the alias. Fixes #51029 * fix(otel): type the semconv response model inputs and attach hidden params in tests Bind the request model once through as_str so the deployment fallback no longer reads untyped kwargs. Non-streamed fixtures now attach provider_response_model with set_hidden_params, since ModelResponse(hidden_params=...) does not keep it on the pydantic private attribute. * fix(otel): read the request model as object before narrowing it Binding kwargs.get("model") to a Final[object] stops basedpyright from reporting an unknown argument passed to as_str. * fix(otel): validate the untyped request model through a TypeAdapter boundary TypeAdapter(object).validate_python accepts the untyped kwargs value and returns a known object, the existing litellm/utils.py and router.py pattern, so as_str no longer receives an Unknown. * test(otel): stream the aliased response through a real Logging object Greptile flagged the MagicMock logging_obj as a double on our own code. The streamed fixture now builds a real Logging with a fixed start time, like test_streaming_handler.py. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
moe-berri
|
d2711ac559
|
feat(alerts): add management controls and opt-in daily summaries (#51418)
* feat(alerts): add alert controls and daily notification summaries * fix(alerts): reserve daily notifications before delivery |
||
|
zoveki
|
22252073e7
|
feat(bedrock): search knowledge bases by image and return image hits (#51063)
* feat(bedrock): search knowledge bases by image and return image hits
vector_stores.search and asearch take a query_image (base64 jpeg, png, gif or webp), and query is optional when it is sent. Bedrock sends it as an IMAGE retrievalQuery, and a caller's own retrievalQuery in extra_body still wins
Bedrock IMAGE hits now come back as image content items instead of being dropped. The media type comes from the source file mime type metadata, then the data URI label, then application/octet-stream
A request with both query and query_image, with neither, or with a malformed image gets a 400. So does query_image on a provider without image search, unless drop_params is on. The proxy accepts query_image in place of query on /v1/vector_stores/{id}/search
Spend logs redact image data when prompts are not stored. Emulated file_search skips image-only hits, RAG rerank maps its indices to the text documents it ranked, and the dashboard tester renders image hits
* fix(vector-stores): keep image hits to image searches and reject a caller retrievalQuery
Text searches on a multimodal Bedrock knowledge base return the same text hits as before, so chat, RAG and logging payloads stay unchanged. Image hits come back only when the caller sent query_image
A caller retrievalQuery now answers 400 pointing at query_image, or is dropped under drop_params, instead of replacing the built query unchecked
A file_search chat tool that carries query_image or rewrite_query searches again instead of failing, and a non-string query next to an image answers 400 instead of 500. query_image is keyword-only on search and asearch
* fix(vector-stores): redact query images and keep store search params out of image checks
* fix(vector-stores): redact images nested in extra_body and keep a store's extra_body retrievalQuery out of searches
* fix(vector_stores): keep a caller's image query through store and router params
A deployment's retrievalQuery no longer reaches a routed search, a caller's query fields survive a store's own extra_body, and an extra_body image stops before the provider. file_search forwards rewrite_query, both retrieval query spellings are redacted, and the cost error stops echoing the response when message logging is off
* fix(cost): leave image search responses out of the cost error text
* fix(vector_stores): accept line-wrapped base64 in query_image data
Bedrock takes inlineContent wrapped with \n or \r\n line breaks, as plain base64 writes it, so the check ignores only those before decoding and the data goes to Bedrock as sent
* fix(vector_stores): drop rewrite_query false from the chat file_search tool
False asks for the default search every store already runs, and MongoDB
rejects the field outright, so forwarding it made a MongoDB chat lose the
store's context for a no-op flag. True is still forwarded.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Yassin Kortam
|
2c44f26288
|
chore: remove cookbook and preserve live assets (#50944) | ||
|
mrinal-berri
|
5e74515c29
|
feat(otel): let a team or key OTel v2 destination add its own request metadata keys (#50813)
* feat(otel): let a team or key OTel v2 destination add its own request metadata keys A team or key callback can now set baggage_metadata_keys in its callback_vars, a comma-separated list of requester_metadata.<key> paths. The tenant fan-out adds them as litellm.metadata.<key> to that destination's spans only, on top of the global LITELLM_OTEL_BAGGAGE_METADATA_KEYS; the operator's exporter and other destinations never see them. Values are bound when each span ends, an attribute the span already carries wins, and names of proxy identity fields are rejected when saved, along with lists over 20 keys. Resolves LIT-9545 * fix(otel): refuse request headers in a destination's baggage_metadata_keys The proxy copies the request headers into requester_metadata.headers with credential headers masked, but a header a gateway in front of it added (an IAP or OIDC assertion, an internal token) stays as sent. A team or key admin listing requester_metadata.headers.<name> could export it to an endpoint the team controls. Entries under requester_metadata.headers are now rejected when saved, so exporting a header stays an operator decision in LITELLM_OTEL_BAGGAGE_METADATA_KEYS. Also drops an unused import. * fix(otel): hold destination metadata to the configured span limits A destination's extra litellm.metadata.* attributes were added to a span view built from a plain mapping, so they skipped the limits the SDK applies in set_attribute: a 1 MiB caller value survived an operator's OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT. Values are now cut to the length limit, and keys past the attribute-count limit are dropped and counted instead of evicting an attribute the span already carries. The per-destination views are built in one Final tuple ahead of the forwarding loop, since a Final cannot be assigned inside the loop. * fix(otel): apply destination metadata after the destination view, within the provider's limits The span limits were checked against the span as it ended, before the destination view strips prompt content and backend-only attributes, so a model-call span full of llm.input_messages.* could drop the team's keys on a no_content destination that had room. The values are now bound when the span ends and added as the last step before export, counted against what that destination actually sends, so their dropped count also survives the view. The limits come from the tracer provider the fan-out is attached to, read from the live span at start, instead of a separate env-built SpanLimits. * refactor(otel): consolidate span limits, destination validators and metadata key parsing Reads a span's limits in one guarded place that the emitter and the fan-out share, replaces the two parallel lists of per-destination callback-var validators with one mapping, splits and strips a baggage_metadata_keys list once, drops a filter the post-view merge already enforces, and folds the repeated fan-out test setup into one helper. No behavior change. * ci: retrigger checks stranded by the GitHub Actions outage |
||
|
yujonglee
|
11af2eb22d
|
refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822)
* refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Joseph Barker
|
2c9068dbab
|
fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id (#49912)
* fix(guardrails/rubrik): moderate text-completion prompts and forward rac_agent_id _moderate_prompt synthesized a user message from texts for /v1/completions requests but built the payload from structured_messages, so the webhook received an empty messages list and the prompt went unmoderated. Pass the resolved messages into _build_prompt_moderation_payload. Forward metadata.rac_agent_id to the before_prompt and after_completion webhooks so agent-scoped policies apply. Only that key is forwarded; the rest of request_data metadata carries non-serializable objects and inbound headers * fix(guardrails/rubrik): also read rac_agent_id from litellm_metadata /v1/messages, /v1/responses and other LITELLM_METADATA_ROUTES keep proxy metadata under litellm_metadata, so a caller on those routes may set rac_agent_id there. Read metadata first, then litellm_metadata, forwarding only a non-empty string * fix(guardrails/rubrik): take rac_agent_id from key or team metadata only The Rubrik webhooks use rac_agent_id to choose which agent-scoped policies apply. Forwarding it from the request body let a caller pick any agent's policies, including one a policy exempts. Read it from the caller's admin-configured key metadata, falling back to team metadata, via CustomGuardrail._get_admin_metadata. The proxy writes those from the authenticated key and strips client-supplied user_api_key_* fields from both metadata dicts, so a caller cannot set or override the value --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> |
||
|
taxfree
|
bc3a69ce35
|
fix(otel): serialize pydantic Message objects in langfuse_otel, weave_otel and langtrace (#46845)
langfuse_otel, weave_otel and langtrace now encode litellm.Message objects, and any value JSON cannot encode, when they write the prompt, so spans keep their token usage and model parameters. Under LITELLM_OTEL_V2 the payload builder dumps Message objects instead of dropping them |
||
|
Mateo Wang
|
9fb29c7e38
|
fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions (#50467)
* fix(tool_choice): accept the Responses API's flat allowed_tools layout on chat completions
An allowed_tools tool_choice in the flat Responses layout ({"type": "allowed_tools", "mode": ..., "tools": [...]})
answered 400 on litellm.completion, acompletion, Router and the proxy's chat and /v1/responses-to-chat routes. One
parser now reads both layouts into the nested Chat layout, and the existing Responses bridge flattens it back for
/v1/responses, so either layout reaches OpenAI in the shape each endpoint expects
* fix(tool_choice): drop an allowed_tools choice under drop_params and refuse conflicting flat entries
An allowed_tools tool_choice a provider can't enforce, or a malformed one, now drops under drop_params and answers 400 without it, the same rule LiteLLM uses for other unsupported values. That covers providers outside OpenAI and Azure, malformed choices on every provider, and Azure api versions older than 2023-12-01
A flat allowed_tools entry that carries both name and a nested function or custom key answers 400, since the two names can disagree, matching what flattening already refuses
* fix(tool-choice): keep the 400 when LiteLLM runs the tools, report auto after a drop
Under drop_params an allowed_tools choice a provider can't enforce was dropped, so on MCP gateway tools and in
run_tool_loop and arun_tool_loop LiteLLM could run a tool the allow list excluded. Those paths now keep the 400 they
answered before the flat layout was accepted, and a malformed choice stays a 400 on MCP gateway tools.
/v1/responses served through chat echoed a choice it had dropped, on sync, async and streamed responses. It now
reports auto, and a choice the provider received is still echoed as sent.
* fix(tool-choice): refuse an excluded tool right before LiteLLM runs it
Checking the provider up front missed a model alias, an old Azure api_version,
a litellm_proxy hop, and SDK fallbacks. The tool loop and the MCP gateway now
check the allowed_tools list right before they run a tool call, so an excluded
call answers 400 and never runs.
Under drop_params the choice now drops only when the caller runs every tool,
so a provider-run MCP server or built-in keeps the 400. The Azure checks read
the request's api_version, which also fixes the /v1/responses echo there.
* fix(tool-choice): drop the choice under drop_params and hand an excluded MCP call back unrun
The drop now applies to every provider that cannot take an allowed_tools
tool_choice, whatever the request's tools hold. LiteLLM never runs a call the
choice excludes: on MCP paths the whole round goes back to the caller in a 200,
the same shape as require_approval, and the tool loop keeps its 400 because it
only returns final content. A malformed choice stays a 400 before any request
on the tool loop and on chat with MCP gateway tools.
* fix(tool-choice): refuse an allowed_tools entry whose name is not a string
An mcp or other non-function entry with a list or dict name passed the
parser and crashed when the allow list was hashed. The parser now treats
it as malformed on both layouts, so it answers the same 400 as an empty
allow list, or drops under drop_params.
* fix(tool-choice): honor server-level and null-name mcp entries and stop hooks running excluded calls
* fix(tool-choice): match named mcp entries by server label and let built-in entries allow their interception call
* fix(tool-choice): leave a malformed allowed_tools choice unrewritten in the interception hooks
* fix(tool-choice): resolve mcp entries like the gateway and keep the choice on every hook path
Under drop_params a malformed allowed_tools choice now acts as if it was never sent, and only a choice that parses is stored and enforced. Without drop_params it answers 400 before any provider request, on /v1/responses with gateway tools before MCP listing
An mcp entry resolves its label the way the gateway picks servers to list (server name in any case, alias, access group, toolset), and a named entry matches the tool's own name or the gateway's prefixed name. The gateway rewrites mcp entries into function entries in the caller's layout and sends "none" when nothing is left
The web search follow-up keeps an allowed_tools choice, compression's litellm_content_retrieve is never excluded and joins the provider-facing choice, and the native /v1/responses hook loop applies the exclusion check and stores the choice
* fix(allowed_tools): stop web search and code interpreter follow-ups forcing another call
A kept allowed_tools choice now goes to the follow-up in mode auto, so a
required choice no longer loops to the agentic loop limit. The MCP label
scope dataclass uses a default_factory so the module imports on Python 3.11
* fix(allowed_tools): stop the Responses web search follow-up forcing another call
The Responses-shaped follow-up after a web search left tool_choice out of its patch, so the caller's required choice went out again through the chat to Responses bridge and looped to the loop limit. It now carries a kept allowed_tools choice in mode auto, like the chat follow-up
* fix(mcp): scope each MCP entry to its own allowed_tools and keep the allowlist on follow-ups
A label now covers only the tools its own allowed_tools lists, so a second label on the same server no longer widens it. Responses MCP follow-ups, streamed or not, keep an allowed_tools choice in auto mode instead of dropping it. The new Responses tests fake only the upstream MCP listing, so LiteLLM's own discovery runs
* fix(mcp): match an entry's allowed_tools against the tool name the gateway lists
A server with a display name override lists the display name, and the listing
filter matches allowed_tools against it, but the label scope only checked the
original name, so no spelling worked. The scope now also accepts the listed
name and its unprefixed part, like the listing filter. Tests now cover the
whole gateway and toolset entries and a display-named tool.
* fix(mcp): match an entry's allowed_tools only against a server's own prefix, not the first hyphen
* fix(mcp): match a tool no server resolves only by its exact listed name in an entry's allowed_tools
* fix(mcp): refuse a tool whose display name only looks like an allowed name, and let the chat follow-up answer
An MCP entry's allowed_tools now matches a tool only by its exact listed name or a spelling routing
accepts for its real name, so a display name that carries the server's own prefix no longer passes for
another tool. The chat MCP follow-up keeps an allowed_tools choice in auto mode, so the model can answer
with text after the tool ran.
* fix(mcp): match a toolset label's tools by exact native name
A toolset label served any tool whose name a routing spelling of a granted name matched, so a toolset granting the literal name "server-tool" also served the real tool "tool" that only another toolset grants. It now serves a tool only when the toolset grants its exact native name, the same check tool_is_granted uses when the gateway lists tools
|
||
|
norkelbyte
|
68e78bc53a
|
fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted (#50730)
* fix(router): count a Responses WebSocket turn's tokens before the next turn is admitted * fix(responses): forward and log a finished websocket turn when its early count fails or is cancelled A usage payload the parser rejects no longer stops the relay before the terminal event reaches the client, and a client that leaves while the turn is being counted still gets the turn logged. A cancelled early count now clears its counted-tokens marker so the logging callback counts the turn in full. * fix(responses): finish a turn's early token count when the client leaves A client that left while a finished turn was being counted cancelled the count halfway. The router's count could be written with no marker, so the logging callback counted the turn a second time. The count now runs to the end before the turn is logged, and the failure path goes back to clearing only on errors * fix(router): count one router request per Responses WebSocket turn A socket whose finished turns each counted their own request in router usage no longer adds one more request when it closes, so N turns count N requests, as N HTTP calls do. A socket with no counted turn still counts 1 * fix(proxy): keep router usage markers out of pass-through client metadata and ClickHouse rows --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yujonglee
|
523b3efea2
|
refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501)
* refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
zoveki
|
abe3b11f7a
|
fix(proxy): show committed spend in the remaining budget gauges (#49199)
* fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
b83a1f0608
|
fix(otel): name the routed model on auto-router LLM spans (#50498)
On an auto-router request the LLM span reported the name the caller asked for (the router or an alias of it) as gen_ai.request.model (OTel v2) and gen_ai.response.model (v1 and v2). Both now name the model group the router picked, read from the request's routing_decision. In v2 the span name and the vendor keys built from the request model (Langfuse, OpenInference, Langtrace, Weave) follow. Requests that were not auto-routed are unchanged. |
||
|
mrinal-berri
|
e3387b6fc5
|
fix(otel): seed request metadata baggage for team-only OTel v2 destinations (#50373)
* fix(otel): seed request metadata baggage for team-only OTel v2 destinations A team newrelic (or other preset) destination exported spans without the operator-allowlisted litellm.metadata.* keys unless a proxy-wide otel callback was listed, because only the OTel logger's async_pre_call_hook seeded requester_metadata into Baggage and that hook runs only for loggers in litellm.callbacks. ProxyLogging.pre_call_hook now re-seeds identity with the request data once requester_metadata exists. Resolves LIT-9783 * fix(otel): seed only request metadata from the proxy pre-call hook The pre-call re-seed also passed the request model, which by then can be an alias target, mcp-tool-call or an internal embedding model, so it rewrote gen_ai.request.model on the server span for V2 deployments without otel listed. Seed only the request metadata and keep the model auth seeded. Adds a unit test for the pre-call seed and the guardrails_only skip, drops the explanatory comments, and requires one server span plus the model span in the integration test. * test(otel): annotate new test variables Final and build test config without mutation Addresses review: the new unit and integration tests now follow AGENTS.md (LIT010 Final on every variable, no in-place mutation, read-only parameter types, one for-clause per comprehension). * test(otel): cover key-level callbacks and assert tenant spans carry no credentials Parametrizes the tenant metadata test over team and key callbacks, and adds a test that sends credential headers and a planted New Relic key with an allowlist naming credential headers, then asserts no exported span carries the master key, the virtual key, the team's New Relic key or any credential header value, and that every export used only the team's own New Relic key. |
||
|
yujonglee
|
2b4485dee5
|
fix(guardrails): check a model-level post_call response once across the proxy and deployment layers (#50300)
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy checked the same response a second time (LIT-10481). The deployment hook now records guardrail + response hash on the request and on the response, and the proxy skips a guardrail only on a matching record. The hash covers every field except ids, usage, model and other metadata set between layers, so a response rewritten between the layers (OCR included) is still checked. Guardrails with their own post_call hook keep their pre-#40154 behavior Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Mateo Wang
|
3f38730d2c
|
refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986)
* refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type |
||
|
JeongCheolLee
|
febbe38326
|
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ishaan-j-berri
|
da2ecbc5bb
|
feat: email affected owners before model deprecations (#50172)
* feat: opt-in model deprecation emails for affected owners * fix: annotate deprecation sender initialization * fix: clarify deprecation emails and use consistent settings toggles * fix: exclude former team admins from deprecation emails * feat: clarify provider deprecations and model migration steps |
||
|
zoveki
|
784754ca48
|
fix(azure)!: answer 401 when Entra rejects the deployment's credential (#49863)
* fix(azure): answer 401 when Entra rejects the deployment's credential A ClientAuthenticationError from the Entra token provider (wrong, expired or rotated client secret) used to surface as a retryable 500 carrying the app's client id and Entra trace ids. Map it to a 401 AuthenticationError that keeps the AADSTS code and first sentence with GUIDs redacted, on both the direct token fetch and the provider handed to the OpenAI SDK client. Network failures stay 500. * fix(azure): keep network failures to Entra a retryable 500 azure-identity wraps a transport failure in ClientAuthenticationError, so the 401 mapping now skips errors whose cause chain holds ServiceRequestError or ServiceResponseError * fix(azure): cover async token providers and Entra 5xx, and scope caller-sent Entra credentials * fix(azure): map only AADSTS-coded Entra rejections to 401 and keep caller credential failures out of cooldown An Entra ClientAuthenticationError now answers 401 only when its message carries an AADSTS code, its HTTP status is not 429 or 5xx, and the code is not the AADSTS50196 throttle, so a chained credential's outage or throttle stays a retryable error. Caller-sent Entra fields no longer change the deployment id; a 401 from a value the caller sent skips the shared deployment's cooldown instead. The Azure AI Foundry agent token fetch maps the same rejection, and the A2A endpoint answers it with HTTP 401. * fix(router): count router default_litellm_params as configured Entra credentials when deciding cooldown * fix(azure): keep transient Entra codes retryable and scope 401 handling to Entra rejections ClientSecretCredential drops Entra's HTTP status, so throttling and transient codes (AADSTS50033, 50196, 90006, 90024, 90033, 90055) now stay retryable instead of mapping to 401. The A2A endpoint answers 401 only for an AuthenticationError caused by an Entra token rejection, and the router skips cooldown for a caller-sent Entra field only on Azure-family deployments. The azure/ Decisions route and the Bing grounding search tool now map an environment-credential rejection to the same 401. * fix(azure): map only known Entra credential rejections to 401 and cover the OIDC token exchange Unlisted AADSTS codes keep the retryable 500, the router skips the 401 cooldown only when an Entra rejection caused it, and a federated token Entra rejects answers the same 401 on Azure chat, Responses and A2A agents * fix(azure): answer 401 for Entra rejections on the /azure passthrough and token counting The fallback hop skips cooldown when Entra rejected a credential the caller sent, Prometheus counts a deployment's own Entra rejection as a deployment failure again, and an OIDC exchange error keeps Entra's headers so Retry-After reaches the router. * fix(azure): keep a no-op caller Entra field from skipping cooldown and redact the passthrough 401 message A caller-sent Entra field now skips the deployment's cooldown only when it differs from the deployment, its named credential, the router defaults, the AZURE_* environment variables and the default scope. An empty value counts as unset, so a deployment whose own secret is wrong still cools down when the caller's field changes nothing about the token request. The /azure passthrough's Entra 401 message goes through the same internal hostname redaction as other client-facing errors. * fix(router): compare a caller's Entra field with the value the deployment uses A caller-sent Entra field skipped the cooldown only when it matched no configured source, so a value equal to an environment default the deployment overrides (a stale AZURE_CLIENT_SECRET, say) counted as the deployment's own and let a caller's rejected credential cool down a working deployment. The check now resolves the value the deployment would use without the caller, in order from the router defaults, the deployment, its named credential, the AZURE_* environment variables and the default scope, and skips the cooldown only when the caller's value differs from it. * fix(azure): follow the settings merge for empty Entra fields and keep non-rejection OIDC exchange errors unchanged * fix(azure): count only the failed attempt's Entra rejection and ignore a non-callable token provider * fix(azure): keep the streaming Entra rejection inside the failed attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
b9ad61d4f6
|
fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227)
* fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call * fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests * fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500 * fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message * fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever. A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry. On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed * fix(agentic loop): bill retried attempts and clean up what failed or capped turns return The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints * fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try * fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run. The failure row bills every round the loop served before the failure, at any depth, and a later attempt that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the response cost headers adding up. The error response's x-litellm-response-cost reports the same amount its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before. A refused tool call now matches by id among the refused ids, or by name among refused calls that carry no id, so a name-only refusal next to an id refusal is no longer kept. * fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one * fix(code_interpreter): delete only the session sandbox that failed, never its replacement * test(code_interpreter): type the session sandbox test helpers * fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors * fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too. A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row. A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main. A code interpreter sandbox that fails while running the code now says the code may have run but its result is lost, instead of saying the code did not run. * test(responses): bill a retried MCP request each attempt on its own row again Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row * fix(spend): name a failed follow-up attempt the success record already bills With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off * fix(spend): count a billed failed follow-up's tokens on the success row * fix(spend): bill a failed follow-up on a fallback served from cache * fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown * fix(spend): keep a failed follow-up's billing state off the request-merged call details * fix(spend): record the cost breakdown of a successful agentic follow-up The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
baddc67041
|
fix(anthropic)!: carry prompt_cache_breakpoint to cache_control on chat completions to Claude (#49783)
* fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude
A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped.
Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it.
* fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap
Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and
keep the earliest under it, counting the client's own message, tool, and
top-level cache_control marks. The ttl and malformed-marker errors name
drop_params for proxy users too and no longer name a single provider.
* fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models
A client tool cache_control or a tool_config injection point becomes a Converse tool
cachePoint, so it now takes one of the 4 slots before a translated marker does.
Converse only translates prompt_cache_breakpoint for a model the cost map says takes
prompt caching. An unmapped model such as an application inference profile ARN opts in
through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a
mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403.
Router prompt-caching affinity now treats a marked block like a cache_control one, and
requests without a marker skip the translation after one pass over their messages.
* fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap
* fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results
* perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast
* fix(anthropic): drop the duplicate drop_params left by the main merge
* fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model
* fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap
Under drop_params the translated marks that fit the cap are now split between
the start and the end of the request, so the stable prefix and the newest turns
both stay cached. The cap count also includes a native tool search tool that
keeps its cache_control on the wire.
Bedrock Converse now translates the marker only for the Claude and Nova
families, which Bedrock documents as taking cache checkpoints, unless the
deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its
cache mark when the request fits the cap.
* refactor(bedrock): decide guarded cache marks with a typed bool helper
* fix(bedrock): validate guarded blocks before reading their cache mark
* fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated
When a deployment sets cache_control_injection_points, the client's
prompt_cache_breakpoint marks are dropped on targets that do not read them, so
the operator's injection points decide every Claude breakpoint.
OpenCode Go models served over the Anthropic messages endpoint translate the
marker only for Claude, and guarded Converse text keeps a cache mark only on a
model known to take cache points.
* fix(anthropic): keep client marks unless a message injection point replaces them on a translating route
A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark.
* fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection
A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect.
* fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params
* fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
|
||
|
tessellark
|
5b546259d5
|
fix(cache_control)!: keep one deployment's prompt-cache opt-in from reaching siblings on the same model (#49685)
* fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry Two deployments that share one model string share one key in the live litellm.model_cost map, and router registration writes each deployment's supports_prompt_cache_breakpoint into that key. The cache control hook read the live map, so a flagless deployment picked up a sibling's opt-in and sent prompt_cache_options and prompt_cache_breakpoint upstream, which the provider rejects with a 400 When a call carries the routed deployment's model_info, the prompt cache checks now use its declared supports_prompt_cache_breakpoint, and fall back to the loaded catalog instead of the live map. This covers the hook on completion, acompletion, Responses and the /v1/messages route, the azure_ai chat transform, the OpenAI Responses transform under drop_params, and the /v1/messages to Responses adapter * fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a provider other than openai, such as azure_ai OpenAI models and bedrock_mantle. Bedrock on every route and providers that build Anthropic messages keep cache_control, which Bedrock Converse turns into cachePoint The openai chat reader now passes the routed model_info and checks base_model. The Bedrock cache point reader takes the deployment's flag first and, for a routed call with no flag, reads the catalog as loaded instead of the live map that sibling deployments write into. model_info is threaded from the converse transform, the factory message and tool builders, and the responses bridge * fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations. * fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways * fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases * test(anthropic): cover auto caching on an unmapped Claude deployment * fix(cache_control): keep the merge base's reads where the per-deployment flag changed them A register_model entry under a provider-prefixed name reaches routed calls again, since the overlay now resolves the bare and prefixed names the way runtime registrations are stored. The hook and the openai chat reader no longer judge base_model, as at the merge base, so a client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint and get a 400. A Bedrock deployment flag that is not a real true reads as false again, as at the merge base. An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again, as at the merge base. Other wildcards still leave both cache flags to the catalog. The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps its own flag and that a flagless unmapped ARN keeps its cache points after the delete. * fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name Routed calls read register_model entries through an overlay that merged the bare name after the provider-prefixed one whatever order they were registered in, so a later opt-out under one name lost to an earlier opt-in under the other. Each registration now moves its key to the end of the kept registrations, and the overlay merges both names oldest first, which matches what SDK calls read from the live model map. Also lists the wildcard cache flag helper in the router coverage script, since the hook's wildcard tests drive it rather than calling it by name. * fix(cache_control): keep register_model entries visible while a cost map reload replays them The previous commit moved a re-registered name to the end of the saved registrations by removing and re-adding it, and the overlay walked that dict to order a bare and a provider-prefixed name. A reload replays every saved entry, so a request on another thread could briefly miss an entry or fail on a dict that changed size mid-walk. Entries are written in place again, a per-name sequence records which registration came last, and both the overlay and the reload replay order by it, so the later registration still wins after a reload. * fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
250af8f027
|
fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493)
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
yuneng-jiang
|
1563776cd8
|
fix(prometheus): retain deployment attribution on proxy failures (#49717) | ||
|
tavren27
|
a0b1edc0a0
|
fix(providers): trim prompt_cache_breakpoint to what each API takes under drop_params (#49064)
* fix(openai): trim a malformed prompt_cache_breakpoint under drop_params on chat completions * fix(openai): leave a non-object content part for OpenAI to reject under drop_params * fix(openai): trim an unsupported prompt_cache_breakpoint ttl on every path and on azure_ai chat under drop_params * fix: trim prompt_cache_breakpoint to what Azure and Mantle accept under drop_params Azure OpenAI chat now gets the same content part trim as OpenAI chat, and the Azure and Foundry gates also read a deployment's declared base_model. Each Responses config keeps only the ttl its API takes: Azure keeps 30m, and Mantle keeps no ttl. A trimmed ttl is logged at debug. * fix(responses): trim prompt_cache_breakpoint only on APIs known to read it Other Responses providers get the marker whole or not at all, as on main. Azure chat now trims for a deployment with no base_model unless the cost map marks the model as not reading breakpoints, and Azure AI Foundry Responses keeps Azure's 30m-only shape. * fix(azure): trim prompt cache breakpoints only for deployments known to read them Azure and Azure AI Responses keep main's whole-or-nothing rule for unmapped deployment names, an opted-in custom OpenAI host trims the ttl and keeps the mode, and Azure chat strips every marker for a known non-reader under drop_params * fix(providers): keep main's marker handling where the target won't trim it A /v1/responses model with unknown breakpoint support on OpenAI's host gets a marker whole or not at all, as on main. The chat to Responses bridge keeps main's whole-marker check when a non-bool global drop_params leaves the target's trim off. A hyphenated name like gpt-5-6-sol now reads as gpt-5.6. * fix(providers): read a non-version hyphen suffix like -32k as part of the model name * fix(providers): keep main's reading of a dotted minor version like gpt-5.6o * fix(providers): keep main's breakpoint shape for unmapped Mantle models and skip the Azure lookup without drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
f7b2cd0000
|
fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 (#49237)
* fix(anthropic): answer 400 naming a non-string role or tool call id instead of 500 A list role or an object or list tool_call_id crashed the chat conversion with "unhashable type". Depending on the step that hit it, callers got a 500 or a 400 that never named the field. A failed conversion now answers 400 naming the first non-string role or tool call id, and the prompt caching hook reads a non-string answer id as matching no tool call. * fix(anthropic): name the field the conversion failed on, and answer 400 for a non-string Responses call_id The chat field check now runs only when the conversion raised a type error, and scans roles first, then assistant tool call ids, then tool answer ids, so an orphan tool answer that modify_params drops is no longer blamed. A tool output call_id that is not a string on /v1/responses answers 400 naming input[N].call_id, and the clear_tool_uses polyfill leaves a non-string tool_use_id for the provider * fix(anthropic): name a non-string role whatever error the conversion raised An int or bool role fails the conversion with an invalid message error, not a type error, so it lost the named 400. Roles are now named for any conversion failure, while tool call ids stay limited to type errors * fix(responses): name a non-string message item role by its input index * fix(responses): name a malformed input field only where litellm reads it, by the caller's index A tool output call_id is named only when it is an object or a list and the conversion fails, so an orphan numeric id still converts and another field's failure is never blamed on it. On the bridge to Claude a non-string role or function_call call_id is named by its input item, counting items before a compaction item. A numeric assistant tool call id on chat is named too. * fix(anthropic): name a tool id only for the error it causes and count Codex tool items in input indexes A numeric tool answer id that modify_params drops was blamed for any later type or attribute error, so the 400 hid the real cause. Tool ids are now named only for the error their value raises, and a hashable answer id never. The /v1/responses bridge and its input token count now read the input the caller sent, with only additional_tools hoisted into tools, so input[N] counts those items too. * fix(anthropic): name a tool id only for the error its own value causes and keep Codex tool item text out of the request * fix(responses): name a tool output call_id only when the items before it convert * fix(responses): name a non-string input field only on inference so token counting and batch records keep their answer * fix(responses): let a client field named caller_input or names_non_string_fields reach the bridge * test(anthropic): build the non-string id histories without editing them in place * fix(anthropic): never name an empty role for another field's failure * fix(anthropic): name an empty list or object role when the conversion fails on it * fix(anthropic): name a list role under the drop_params cache cap and keep the Bedrock batch line prefix --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
3155aacda4
|
fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests (#49468)
* fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): return the server tool call id instead of a TypeGuard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Gemini cache write carries the system prompt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Wiley Kestner
|
da71a03f6e
|
fix(proxy): trace model list requests with the caller's traceparent (#49141)
* fix(proxy): trace model list requests with the caller's traceparent
The proxy creates an OTEL SERVER span at the auth boundary whenever a request
carries traceparent, but /v1/models never closed it. Unended spans are not
exported, so callers saw nothing for the model list while inference requests
showed up in the same trace
Route the endpoint through the management span wrapper so the SERVER span is
closed and parented to the caller's trace. The wrapper gains a variant that
skips request and response payloads, since attaching the full model list would
bloat every poll
* refactor(proxy): type the span wrapper with ParamSpec
The strict ruff rules ban new typing.cast imports, so the wrapper keeps the
wrapped endpoint's exact signature via ParamSpec instead of casting the
wrapped callable back to its type. No behavior change
* fix(proxy): keep request bodies off the model list span and tidy its tests
The quiet span wrapper now skips the request body as well as the
response, so no payload reaches the span on success or failure. The
wrapper passes the endpoint's kwargs through instead of copying them,
and reads the caller's key with an isinstance check. The model list
test drives the real endpoint without swapping helpers, and the new
payload test checks both wrappers against a real request body
* fix(proxy): keep the 5xx status on the model list span and trace model retrieve
The span wrapper ended the server span on every failure, so a 500 or 503
from /v1/models lost its status code. It now leaves that span to the
proxy's exception handler, which knows the answered status, and only ends
it itself for an HTTPException, which no handler closes on these routes.
GET /v1/models/{model_id} (the OpenAI SDK's models.retrieve) gets the same
wrapper, so it joins the caller's trace like the listing does.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
tin-berri
|
ec4f66bb94
|
fix(router): budget classifier input by provider limits (#49306) | ||
|
tavren27
|
7e37c20748
|
fix(responses): run the guardrails a Responses WebSocket frame selects (#48725)
* fix(responses): run the guardrails a Responses WebSocket frame selects
* fix(responses): name the field a malformed frame guardrail selection got wrong
A frame whose guardrails or guardrail_config has the wrong shape now gets
an error event naming the field, instead of pydantic's raw validation dump
* fix(responses): accept every guardrail selection shape a Responses WebSocket frame sends, and run during_call guardrails
A frame can pick guardrails as a list of names or name-to-config dicts, inside metadata or litellm_metadata, with policies validated as a list of strings. during_call guardrails now run before the turn goes upstream, and selection fields sent next to a nested response are rejected instead of dropped
* fix(proxy): keep a key's configured guardrails when policies add guardrails
A policy's guardrails were merged with dict.fromkeys, so a configured guardrail entry ({name: {extra_body}}) in key metadata raised unhashable type: 'dict' and every request naming policies failed with a 500 on HTTP and on the Responses WebSocket. The merge now keeps existing entries and skips policy guardrails already selected by name. Also types the WebSocket test guardrail's hooks.
* fix(responses): run streaming-scoped guardrails on socket turns, keep guardrail fields on blocks, merge guardrail lists by name
* fix(responses): check frame guardrail picks against the team permission, honor picks beside a nested response, prefer a guardrail entry with settings
* style(responses): format the frame guard and give the pass-through guardrail test an enterprise license
* fix(proxy): keep pass-through guardrail settings, gate frame policy picks, skip moderation-only hooks on socket turns
* fix(responses): keep string metadata guardrail values as client metadata on socket turns
* fix(responses): ask no team permission for string metadata named like a guardrail pick on socket turns
* fix(proxy): socket guardrail picks read the key's team metadata when the proxy has no database
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
tessellark
|
a084d13635
|
fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures (#49075)
* fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures Pass-through failures on Gemini and Vertex routes carry the model only in the URL, so the failure spend row and the Prometheus failure counter logged an empty model. The failure payload now falls back to the model parsed from the URL on those routes, and the Prometheus provider label honors the provider the request declares before inferring one from the model name * fix(passthrough): keep model and provider on mid-stream and Vertex failure logs Build the failure payload from the URL at both streamed relay sites so a 200 stream that dies mid-body still logs the model, derive vertex_ai for Vertex routes that never declare a provider, and have the Vertex logging handler reuse get_vertex_model_id_from_url instead of its own regex * fix(passthrough): bound the Prometheus provider label and name Vertex failures like the success handler Declared providers only reach the api_provider label when they name a registered provider, otherwise the label falls back to inference. Vertex failure rows now use the success handler's model naming (vertex_ai/ prefix on rawPredict and streamRawPredict, vertex_ai/search_api on :search) and its host-based provider detection, and the detected-stream failure sites are covered by tests. * fix(passthrough): name Google failures from the URL on untracked methods too A Gemini or Vertex pass-through failure on a method the success handler does not track (countTokens, for one) still logged model "" and no provider, although the model sits in the URL. The failure payload now falls back to the Google AI host for the provider and reads /models/<name> from the URL, so countTokens 400s name their model and provider like generateContent ones do. Non-Google hosts keep the tracked-route rule and still log nothing extra. * fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases The failure path only named the Google provider from the URL when the method was one the success handler tracks or the host was a Google API host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE (or a Vertex one behind a custom base) still logged an empty model. A route that declares gemini or vertex_ai now wins outright, and a URL whose path carries /projects/<p>/locations/<l>/ counts as Vertex on any host, while a tracked method on a non-Google host no longer passes as Vertex. * fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model * test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback * fix(passthrough): scope declared provider to pass-through failures and match Google hosts only Prometheus trusts the resolved provider as before and reads the request-declared provider only on pass-through failures. Vertex attribution matches Google hosts, strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the URL path only, covers Vertex Live WebSocket failures, and labels a router alias failure with the alias the client requested * fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler. * fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures A pre-call guardrail block on a pass-through route now builds its failure payload the same way upstream failures do, so the model and provider read from the URL reach spend logs and metrics. Vertex Live websocket failures (connect errors, upstream rejections, and the credential failure) now carry the model from the ?model= query parameter under the vertex_ai provider * fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls Prometheus reads a failure's declared provider only on pass-through calls, so without call_type it fell back to guessing from the model name. That labeled a guardrail block on the Gemini route as vertex_ai and the Live credential failure as None * fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row * fix(pass-through): keep URL model attribution off routes that name another provider A built-in route that names a non-Google provider, like /openai pointed at Gemini's OpenAI-compatible host, no longer takes its failure model from the Google URL. A guardrail block response now returns the x-litellm-call-id of its spend row. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a44e44b980
|
fix(cache_control): pick the chat-path dialect from the deployment's provider prefix (#45295)
* fix(cache_control): resolve the chat-path dialect with the deployment api_base * test(cache_control): type the chat-path resolver helpers * fix(cache_control): pick the chat-path dialect from the deployment's provider prefix Resolving with the deployment api_base followed the dispatch's azure_ai to azure flip, so chat on a non-Foundry Azure host sent Anthropic cache_control while /v1/messages, told azure_ai, sent the OpenAI breakpoint. Use the model's provider prefix when the caller does not name a provider, so chat matches the told path whatever AZURE_AI_API_BASE holds, and keep name-only resolution everywhere else. * fix(cache_control): keep litellm_proxy routing when the chat path reads the provider prefix With USE_LITELLM_PROXY or litellm.use_litellm_proxy on, every call goes to the upstream proxy, so the provider prefix no longer decides the cache format there and resolution matches main. The plain OpenAI check reads the same provider value as the hosted check, so one function no longer mixes two lookups * fix(cache_control): resolve the chat-path dialect with the deployment api_base, as the call is routed The hosted cache-format check now resolves the provider with the deployment's own api_base when the caller names none, the same provider the call dispatches to and the Responses path passes, so AZURE_AI_API_BASE no longer decides the chat format. The plain OpenAI check keeps main's name-only lookup. * fix(cache_control): send acompletion fallbacks the caller's request, not the primary's hooked kwargs acompletion handed async_completion_with_fallbacks the kwargs the cache seed and prompt hook had already rewritten for the first model, so a Claude fallback behind a Foundry GPT model received prompt_cache_options and got a 400. Fallbacks now get a copy of the caller's model, messages, tools, and kwargs taken before the hooks, as completion already does with its args. Both cache seeds also read base_url before api_base, since completion sends the call to base_url when both are set. * fix(cache_control): keep the caller's tools list intact for acompletion fallbacks The vector-store hook pops recognized file_search tools from the request's tools list in place, so the fallback request built from the pre-hook snapshot lost the retrieved context. Snapshot a copy of the list instead. * fix(cache_control): skip acompletion's prompt hooks when fallbacks will run them Each fallback attempt runs the prompt hooks for its own model, so running them first in the outer call only cost a vector-store search whose result the hand-off threw away. * fix(cache_control): run acompletion's prompt hooks for fallbacks and leave only the cache hook to each attempt --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Filip Sinitsin
|
e81dd348a4
|
fix(proxy)!: log and bill Responses-style bodies sent to /cursor/chat/completions (#30277)
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126) The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests. * test(proxy): pin provider metadata and key billing on /cursor/chat/completions A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land * fix(proxy): pick the Cursor route's metadata field from the body shape A Responses-style body on /cursor/chat/completions keeps proxy metadata in litellm_metadata, the same as /v1/responses, while a Chat Completions body keeps metadata, the same as /v1/chat/completions. Adding the whole route to the internal-metadata list let a client's chat-body tags replace the key's tags in spend logs and skip tag budgets. * fix(proxy): keep get_metadata_variable_name callable with only the request * fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field * fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies * fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string * fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Devon Krisman
|
953750ed41
|
fix(websearch-interception)!: stream native server-tool blocks to clients (#40967)
* fix(websearch-interception): stream native server-tool blocks to clients Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code) never received the server_tool_use and web_search_tool_result blocks the interception layer builds for them, so citation panels stayed empty and result counts read zero. Two gaps in the same flow: FakeAnthropicMessagesStreamIterator._create_content_block_chunks only knew text, thinking, redacted_thinking and tool_use; any other block type emitted a bare content_block_stop with no start and no payload, silently dropping the block from the fake stream. server_tool_use now streams as content_block_start plus input_json_delta and any other server-generated block rides whole in content_block_start, matching Anthropic's native streaming of these types. try_short_circuit_search only recognized a native client by finding a web_search_* tool in the tools list, but the pre-request and deployment hooks convert that tool to the LiteLLM standard shape before the short-circuit runs, recording the conversion in WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so the synthetic response carries the native blocks on the converted path instead of only when hook ordering happens to leave the tool untouched. Fixes #35333 * fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500 * fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply * fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply * test(websearch-interception): build the cost test's logging object with a datetime start time * fix(websearch-interception): send search snippets once in the direct search answer * fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta. * fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite A legacy hook that masked a tool call's arguments on a buffered stream made the whole rewrite undeliverable, text included, so the original stream went out. With web search blocks in the stream that now hit any hook masking the search query. The adapter hands same-count tool-call rewrites to the translation's write-back, as a unified guardrail's already are, and still discards a rewrite that adds or drops a tool call. * fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled ModelResponse, which keeps web_search_tool_result blocks only in provider-specific fields, so a hook masking result titles on a stream never saw them. The hook's view now carries those blocks, its rewrite of them goes back through the ended-stream write-back, and the adapter no longer needs to pass texts outside the hook's view through untouched. * test(guardrails): type the legacy hook stream test helpers * fix(guardrails): scan a streamed web search query as text like the non-streaming reply * test(guardrails): type the masking guardrail test helpers * fix(guardrails): keep the block id when scanning a streamed web search query --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |