mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-02 07:13:02 +00:00
1022 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
4c8b90945f
|
fix(realtime)!: send the Gemini Live setup at the first content so session.update settings apply (#39864)
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its default audio setup at connect, so a client's session.update (GA output_modalities or beta modalities) was dropped and every answer came back as audio. The setup now goes out on the client's first message: a session.update becomes that setup, anything else gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a provider hook. Text setups label deltas response.output_text.delta, and downgrading TEXT to AUDIO on an audio-only model logs a warning * fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup * fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup Gemini and Vertex Live accept one setup per connection, so the bridge used to turn only the first session.update into that setup and drop the rest without telling the client. Now every session.update that arrives before the first content frame is held, deep-merged (later values win per field), acknowledged with a session.updated built from the setup it would produce, and the merged session becomes the one setup sent with the first conversation item, audio frame, or tool result. The session events also surface the systemInstruction text as instructions, which the old string check never matched because the setup carries it as a Content object. * fix(realtime): cap the session.update merge depth so the recursive detector accepts it _merge_session_fields recurses once per nested mapping level, so CI's recursive_detector flagged it. Bound the descent at eight levels (the OpenAI session object nests four deep at most), replace anything deeper wholesale with the later session.update's value, and list the helper with the other depth-capped functions the detector allows. * fix(gemini-live): accept a string output modality and drop values Live cannot produce A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once. * fix(gemini realtime): answer an error for session updates Gemini Live cannot apply An output modality the model cannot produce, or a session.update sent after the first content, now gets an invalid_request_error event instead of a silent swap or drop, unless drop_params is set. The synthetic session.created reports the model's default modality, setupComplete no longer adds an extra session.updated, and merging nested session updates is linear in their size * fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns A pre_call guardrail blocking the first typed message now goes out after the setup, so Gemini and Vertex no longer close the session with 1007. Push-to-talk sessions (turn_detection null or create_response false) wrap each audio turn in empty activityStart and activityEnd messages, which Gemini needs when automatic activity detection is off. A session with no content after 60 seconds sends its setup, since Vertex refuses a setup 300 seconds after connect. Held session.update frames past 10 MB answer session_update_too_large, and deferred mode now errors on a second session.update unless drop_params is set. * fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection An update that leaves the applied setup unchanged now gets session.updated instead of session_update_not_applied, so clients that resend their session with the first response keep working. Deferred setup mode merges held updates like the default mode, turn_detection null turns Gemini's activity detection off, drop_params drops a voice outside the documented Live voices, and the lazy setup deadline moves to 240 seconds * fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content * fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name. * fix(realtime): hold first content so a following session.update joins the Gemini setup Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too * fix(realtime): check AI Studio Live voices like Vertex * fix(realtime): forward any voice to AI Studio Live models that accept it * fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup. * fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock The deadline and grace tests waited on real 50 ms timers, and the merge cost check compared wall-clock runs. The session now reads an injected clock, time.monotonic by default, the two tests step it, and the merge cost check moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes the backend through wrap_backend, which main added after the test was written |
||
|
Filip Sinitsin
|
e81dd348a4
|
fix(proxy)!: log and bill Responses-style bodies sent to /cursor/chat/completions (#30277)
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126) The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests. * test(proxy): pin provider metadata and key billing on /cursor/chat/completions A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land * fix(proxy): pick the Cursor route's metadata field from the body shape A Responses-style body on /cursor/chat/completions keeps proxy metadata in litellm_metadata, the same as /v1/responses, while a Chat Completions body keeps metadata, the same as /v1/chat/completions. Adding the whole route to the internal-metadata list let a client's chat-body tags replace the key's tags in spend logs and skip tag budgets. * fix(proxy): keep get_metadata_variable_name callable with only the request * fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field * fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies * fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string * fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Devon Krisman
|
953750ed41
|
fix(websearch-interception)!: stream native server-tool blocks to clients (#40967)
* fix(websearch-interception): stream native server-tool blocks to clients Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code) never received the server_tool_use and web_search_tool_result blocks the interception layer builds for them, so citation panels stayed empty and result counts read zero. Two gaps in the same flow: FakeAnthropicMessagesStreamIterator._create_content_block_chunks only knew text, thinking, redacted_thinking and tool_use; any other block type emitted a bare content_block_stop with no start and no payload, silently dropping the block from the fake stream. server_tool_use now streams as content_block_start plus input_json_delta and any other server-generated block rides whole in content_block_start, matching Anthropic's native streaming of these types. try_short_circuit_search only recognized a native client by finding a web_search_* tool in the tools list, but the pre-request and deployment hooks convert that tool to the LiteLLM standard shape before the short-circuit runs, recording the conversion in WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so the synthetic response carries the native blocks on the converted path instead of only when hook ordering happens to leave the tool untouched. Fixes #35333 * fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500 * fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply * fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply * test(websearch-interception): build the cost test's logging object with a datetime start time * fix(websearch-interception): send search snippets once in the direct search answer * fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta. * fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite A legacy hook that masked a tool call's arguments on a buffered stream made the whole rewrite undeliverable, text included, so the original stream went out. With web search blocks in the stream that now hit any hook masking the search query. The adapter hands same-count tool-call rewrites to the translation's write-back, as a unified guardrail's already are, and still discards a rewrite that adds or drops a tool call. * fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled ModelResponse, which keeps web_search_tool_result blocks only in provider-specific fields, so a hook masking result titles on a stream never saw them. The hook's view now carries those blocks, its rewrite of them goes back through the ended-stream write-back, and the adapter no longer needs to pass texts outside the hook's view through untouched. * test(guardrails): type the legacy hook stream test helpers * fix(guardrails): scan a streamed web search query as text like the non-streaming reply * test(guardrails): type the masking guardrail test helpers * fix(guardrails): keep the block id when scanning a streamed web search query --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4a5c5665fa
|
fix(azure_ai)!: keep OpenAI prompt cache breakpoints on the Foundry chat completions path (#45299)
* fix(azure_ai): keep OpenAI prompt cache breakpoints on the Foundry chat completions path * fix(azure_ai): keep only OpenAI content part types when preserving prompt cache breakpoints * fix(azure_ai): share the breakpoint check with the cache-mark step and keep only text parts A deployment's model_info opt-in is registered under its provider-prefixed cost-map key without a litellm_provider, so that prefix now names the provider the flag speaks for. * refactor(azure_ai): read text parts through a walrus and mark new test values Final * fix(azure_ai): read a router-registered breakpoint opt-in only for azure_ai and keep a message mark off dropped file parts * fix(azure_ai): read the breakpoint opt-in from each deployment's own model_info and base_model A router deployment without its own supports_prompt_cache_breakpoint flag now reads the cost map as loaded, so a sibling's opt-in on the same model string no longer turns its requests into 400s. The shared hook no longer reads router registered keys for azure_ai. * fix(azure_ai): send breakpoints in Foundry's shape under drop_params and fall back to the model name * test(integration): expect the injected breakpoint on the Foundry GPT-6 chat wire --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
b91fc92b6d
|
fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054)
* fix(router): cap the default OTPM reservation for requests without max_tokens * fix(router): reconcile ITPM and OTPM reservations after every successful call * fix(router): read the default OTPM reservation from the served model and make its fraction configurable The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check. * fix(router): refund a failed attempt's io token reservation in make_call The proxy passes one logging object to every retry, and it runs failure callbacks only for the first attempt, so the last attempt's ITPM and OTPM reservation stayed held until the minute ended. Each attempt that raises now gives its reservation back before the retry or fallback * fix(router): count Anthropic cache writes once in ITPM reconcile Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens. * fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction * fix(router): reserve with the deployment cap only through the param the request's API sends * fix(router): never let a deployment's output cap raise the reservation above the default * test(router): move the reservation fraction env var test to the integration suite The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation. * test(router): retry the reservation fraction scenario when the minute rolls over The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute. * test(router): give the rollover retry its own deployment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
733cfeba93
|
fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced (#48953)
* fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced * test(integration): a reported cost is attributed to output in the gateway parity cases * fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects * fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge * fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost * fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill * fix(cost): read the stored breakdown total through the logging object's typed field * refactor(cost): name the providers billed from their own usage fields for what they are * fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
636e707089
|
fix(proxy)!: stop answering with the literal string "None" as error type and param on the proxy-server, auth, spend, and key health routes (#39540)
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes The error objects these routes return defaulted `type` and `param` to the four-character string "None", so a client's error handler matched no known OpenAI type and fell into its generic branch. Route them through the shared openai_error_payload helpers instead, so `type` comes from the carried type or the HTTP status and `param` serializes as JSON null. * fix(proxy): stop answering with the literal string "None" on the spend and management routes The error objects these routes return defaulted `type` and `param` to the four-character string "None", so a client's error handler matched no known OpenAI type and fell into its generic branch. Route them through the shared openai_error_payload helpers instead, so `type` comes from the carried type or the HTTP status and `param` serializes as JSON null. * test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param * fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None". * fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400 * fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap * fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param * fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error. * test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor |
||
|
yuneng-jiang
|
d76db470f9
|
test: update integration tests left stale by recent product changes (#49143)
* test: update integration tests left stale by recent product changes * test: expect proxy Messages calls to stay on Python with Rust enabled |
||
|
zoveki
|
bbc68e21f9
|
feat(dashscope): add image edits and text-to-speech (#48998)
* feat(dashscope): add image edits and text-to-speech * fix(dashscope): bill reported speech characters and input images, follow the chat region for edits, keep the call timeout on the audio download * fix(dashscope): bill deployment image prices as main does and route edits to the host speech uses A deployment that sets its own input_cost_per_image, output_cost_per_image or input_cost_per_pixel bills through the default image calculator again, so an override is never read as a per-input-image price and pixel pricing on qwen-image-3.0-pro no longer raises. Edits now resolve the caller's api_base or DASHSCOPE_API_BASE through the same API-root helper speech uses, on any host, with DASHSCOPE_API_BASE_IMAGE still winning. An image response whose usage is null or not an object keeps its images instead of raising. * fix(images): bill a deployment's own prices on image edits through the proxy The proxy builds the logging object before the router picks a deployment, so image_edit never handed the deployment's price fields to its cost calculation and billed the cost-map price instead. Pass the custom pricing params with the pre-call update, as the Responses API does. * fix(images): bill per-pixel image prices at a DashScope W*H size DashScope's mapped size reads 1328*1328, which the cost size reader rejected, so a deployment's input_cost_per_pixel billed every DashScope generation or edit at the 1024x1024 default. * fix(dashscope): bill image prices by their meaning, report the rendered size, prefer the deployment edit base, fetch audio over https * fix(dashscope): read the rendered image size from output_width and output_height * test(images): bill a fal edit at its deployment's own image price * fix(dashscope): keep the qwen-image-3.0 input-image fee out of the shared cost map row Released proxies read the cost map from main and bill input_cost_per_image as the flat price of each DashScope image, so the row's $0.003 input fee billed every qwen-image-3.0 image there at a tenth of its $0.03 price. A deployment can still set input_cost_per_image to bill DashScope's input images --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
08b539e6a8
|
fix(proxy)!: bill streamed /v1/completions without stream_options like chat (#45366)
* fix(proxy): bill streamed /v1/completions without stream_options like chat * test(proxy): mark the text completion stream usage test variable Final * fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage * test(spend): drop the module docstring from the text completion stream usage test * fix(proxy): keep the internal stream usage strip flag out of the logged request body --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yujonglee
|
e93e01ca4b
|
fix(ci): build base native extension for OpenAPI lint (#49136)
* fix(openapi): build native extension for base counts * test(openapi): exercise native import in base worktree --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
d884167bf2
|
fix(responses): only merge repeated Codex tools when drop_params is on (#49112)
Without drop_params, every copy of a tool from additional_tools input items is forwarded as sent, so a provider that rejects duplicate tools returns its own 400. Codex keeps the merge because the proxy turns drop_params on for it. |
||
|
yuneng-jiang
|
62b8ee78de
|
ci: run the integration groups on GitHub Actions (#48985)
* test: add a manually dispatched integration trial on larger runners Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one * test: run the integration trial on push to this branch * test: give integration trial workers a local redis-server * test: run the providers trial on a 64-core runner with 32 workers The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4 * test: run every integration group in the trial on sized runners * test: add a scan that refuses to upload integration results holding the license * test: drop the member-name pass the archive byte scan already covers * test: use a secret whose base64 and base64url encodings differ * ci: run integration groups on run-ci pull requests and scheduled main runs * ci: scope the integration license to a protected environment |
||
|
yucheng-berri
|
ffcc4f7a70
|
fix(otel)!: print OTel v2 spans to stdout only when the operator asks (#48723)
* fix(otel): stop OTel v2 vendor presets from printing every span to stdout With LITELLM_OTEL_V2 on and a vendor callback such as langfuse_otel or arize holding operator credentials, the proxy printed every span as JSON to stdout, prompts included, on the request path. Nobody set OTEL_EXPORTER, but the console placeholder that OpenTelemetryV2Config folds in for an empty exporter list stayed next to the vendor's own exporter. The provider now skips that placeholder when the config has any other exporter. A console exporter the operator names with OTEL_EXPORTER=console is kept, and so is the placeholder when it is the only exporter. * test(otel): type the console-placeholder tests and drop inherited OTEL_* env from the stdout check * fix(otel): print the console default only for the otel callback The proxy's background V2 logger (published when no callback built one, which is what key-only and team-only Langfuse destinations run under) also kept the unconfigured console exporter, so it printed every span with prompts to stdout. The console default now prints only for a listed otel callback or an exporter the operator named. * fix(otel): keep the otel callback's console output on service-name routes A key or team otel_service_name moves the call onto a provider the tenant tracer cache builds, which never got the console choice, so the plain otel callback stopped printing those calls. The no-operator-keys integration rig also drops inherited Langfuse keys. |
||
|
yucheng-berri
|
a54204b583
|
feat(guardrails): scan sampled stream windows in the background (#48952)
* feat(guardrails): scan sampled stream windows in the background (LIT-9338) streaming_scan_in_background runs each sampled post_call scan as a task so the stream keeps flowing; a block ends the stream before the next chunk. * fix(guardrails): hold the stream end for the background verdict and cancel the scan on upstream errors (LIT-9338) * fix(guardrails): wait for the cancelled background scan on upstream errors (LIT-9338) * test(guardrails): cover /v1/responses, a failing scan peer and a mixed burst for background scans (LIT-9338) * test(guardrails): build the burst calls without a nested comprehension (LIT-9338) |
||
|
yucheng-berri
|
1e7da87e1e
|
fix(otel)!: give key/team Langfuse destinations user, tags and session on OTel v2 without global Langfuse keys (#48718)
* fix(otel)!: keep a credential-less langfuse_otel on OTel v2 so key destinations get the Langfuse mapper With LITELLM_OTEL_V2 on, langfuse_otel in callbacks and no operator LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY, the preset raised at startup and the callback fell back to the legacy logger. Key and team destinations were then served by a generic v2 logger without the Langfuse mapper, so their traces lost tags, user and cost, and the legacy logger added an orphaned raw_gen_ai_request span. The preset now always degrades to the credential-gated exporter when the operator has no keys, the same way arize_preset does, so the startup logger stays on v2, exports nowhere for traffic without a destination, and carries the Langfuse mapper to every key and team destination. * test(otel): type the langfuse tenant-only test helpers * fix(otel): give key/team Langfuse destinations the Langfuse fields when langfuse_otel is not a callback With langfuse_otel only in a key's or team's logging, the startup logger is the generic one and its spans reached the key's Langfuse project without user, session or tags. The emitter now adds the Langfuse mapper for a request with a Langfuse destination, and every non-Langfuse exporter strips the Langfuse request metadata, since any logger can now write it. * fix(otel): stop a key's Langfuse logger from re-sending spans to the operator's collector With langfuse_otel only in key or team logging, the proxy publishes the generic V2 logger at startup and it already exports to the operator's OTLP collector. The Langfuse logger a key's request builds kept the same base exporter, so the collector got every model span twice. A logger built after the publish now keeps only its preset's own exporters, as it already did beside a configured V2 logger, and a later call that would rebuild one exporting nowhere reuses it. * test(otel): assert the collector gets one copy instead of reading the key logger's exporters * test(otel): count the collector's spans with one comprehension clause * fix(otel): keep the legacy langfuse_otel logger and stand it down for requests a Langfuse destination carries Reverts the preset degrade and the published-provider exporter trim, so every request without a key or team Langfuse destination keeps main's legacy logger. For a request with one, the V2 fan-out already delivers the whole trace with the Langfuse fields, so the legacy logger emits nothing for it. * test(otel): pin the legacy langfuse_otel paths a tenant's project still gets without a destination * fix(otel): shed a destination's Langfuse request metadata before its other Langfuse fields At its attribute limit a span sheds the Langfuse fields only a destination asked for first, in the mapper's order, so the generation marker went before the request metadata blob. Put the request metadata first, as a Langfuse-configured logger already does. * fix(otel): keep the legacy langfuse_otel span for a destination's call rejected before any provider The OTel v2 fan-out records no call for a request the proxy rejects before a provider (rate limit, pre-call hook), so standing the legacy logger down there lost the user, session and tags main sent the tenant. Stand it down only for a call the fan-out records, using the same check the v2 logger uses. The tenant and collector integration tests now hold their exact counts for a few seconds, so a late duplicate copy fails them. * fix(otel): give a key's Langfuse destination the langfuse_environment the legacy logger wrote The destination's resource carries langfuse.environment from the key's or team's langfuse_environment, else LANGFUSE_TRACING_ENVIRONMENT, so a key's project files its traces under the environment it chose instead of default. Only that destination's export carries it. * fix(otel): keep the Langfuse fields written for a key's Langfuse destination off every other exporter * fix(otel): give a key's Langfuse destination the prompt and response the legacy logger sent, only where it ran * fix(otel): keep a key's Langfuse destination on the global capture setting when it sets none, with or without global Langfuse keys |
||
|
yucheng-berri
|
83d6779158
|
fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask (#48944)
* fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask on_flagged block now returns 400 when Lakera flags only PII, instead of masking it and forwarding the request. on_flagged mask keeps the old behavior: mask PII-only detections and block anything else. * test(guardrails): cover lakera_v2 on_flagged on every endpoint, hook and a Lakera outage * test(guardrails): type the lakera_v2 on_flagged unit test helpers |
||
|
mrinal-berri
|
4a4c9cf7e3
|
feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block (#46219)
* feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): integration cells for PANW include_masked_response_on_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): reject non-boolean include_masked_response_on_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(guardrails): update PANW tool-call comment for include_masked_response_on_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): build PANW kwargs without mutation and clean up DB-created test guardrails Restore main's single-expression kwargs in the PANW initializer. The two DB-backed integration tests now create their guardrail inside the scenario and delete it on exit. * fix(guardrails): type raw PANW boolean setting before validation --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c213234783
|
feat(vertex_ai)!: use HTTP/2 httpx client for search_api vector store (#40631)
* feat(vertex_ai): use HTTP/2 httpx client for search_api vector store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(httpx): exercise configured request path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(httpx): verify HTTP/2 request negotiation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(httpx): preserve HTTP/2 for sync proxy mounts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): add h2 to proxy extra so vertex search HTTP/2 is on by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http_handler): isolate ssl context for http2 clients Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(http): add an HTTP/2 opt-out and harden the Vertex AI Search HTTP/2 client litellm.http2 is now unset by default, so http2: false or LITELLM_HTTP2=False turns the provider's HTTP/2 default off. A caller's own SSLContext keeps the search on HTTP/1.1, the HTTP/2 async client honors DISABLE_AIOHTTP_TRUST_ENV, forced IPv4 transports get the CA bundle and client cert, and vector store search retries once when the server drops the connection with a GOAWAY. * fix(http): retry vector store search only on HTTP/2 GOAWAY and keep aiohttp pool and proxy behavior Scope the one retry to a connection Google closes with an HTTP/2 GOAWAY and log it. When HTTP/2 replaces the aiohttp transport, keep aiohttp's idle keepalive and connection cap, and only then skip env proxies under DISABLE_AIOHTTP_TRUST_ENV, so the global LITELLM_HTTP2 toggle keeps HTTPS_PROXY * test(http): cover the vector store search GOAWAY retry in unit tests and drop duplicate SSL cache comments * fix(http): open another HTTP/2 connection once 100 streams are in use httpcore keeps every request to an origin on one HTTP/2 connection and caps it at 100 concurrent streams, so request 101 and later waited with no timeout. The handlers now spread requests over extra connections, opening one only when every open connection has 100 streams in flight. The force_ipv4 wire tests now cover ssl_verify=False and the client certificate. * fix(vertex_ai): reuse one Vertex search OAuth token across searches Every vector store search built a new search config, so the token cache it carries was thrown away and each search fetched a new Google OAuth token with a blocking call on the event loop. Under a burst of searches that blocking fetch held back every HTTP/2 response. The provider config is now shared, so searches with the same credentials reuse one token until it expires. * fix(http): keep the HTTP/2 connection spread within the connection cap * fix(http): keep sync vertex search on HTTP/1.1, since httpcore's sync HTTP/2 is not thread-safe httpcore 1.0.9 hands two threads the same HTTP/2 stream id, so threaded sync searches failed with protocol errors. Only the async client, which the proxy uses, now defaults to HTTP/2, and the sync connection spread is removed. * fix(http): split the connection cap across HTTP/2 lanes Each lane had its own pool with the full connection cap, so an origin answering HTTP/1.1, or several origins on one client, could exceed max_connections once more than 100 requests were in flight. Lanes now open only while lanes times 100 streams stay under the cap, and each lane pool gets at most 100 of it. * test(http2): scope wire-test peer logs to each fixture instance * fix(http2): keep TE, force_ipv4 trust, evicted-client and opt-out behavior on the HTTP/2 path - drop TE values other than trailers on https requests through the HTTP/2 spread, since h2 refuses them - force_ipv4 transports keep httpx's default trust store (SSL_CERT_DIR) unless SSL is configured - the evicted client closer asks the HTTP/2 spread whether a request is in flight - LITELLM_HTTP2 set to 0, no or off, and a quoted "false" http2 setting, opt out of HTTP/2 - the wire test's held-search barrier waits on an event instead of polling * fix(http2): count proxy-mounted transports when checking an evicted client for in-flight requests * test(http2): move the HTTP/2 eviction cases that open a loopback server into the integration wire tests * fix(http2): read an unquoted http2: 0 as off and free the stream slot of a response dropped without closing * fix(http2): free the stream slot of a partly read response dropped without closing * test(http2): check that the evicted client and its stream close after a partly read response is dropped --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
jesco
|
3057c47d6d
|
fix(caching): keep reasoning, citations, and audio on streamed cache hits (#32096)
* fix(proxy): preserve reasoning fields on streaming cache hits * fix(proxy): validate cached streaming delta fields Signed-off-by: jesco-absolut <team@srswti.com> * fix(caching): replay cached audio, images, reasoning and citations in stream order A cached answer replayed as a stream now ends with its own empty finish chunk, so a chunk that carries only audio or images keeps its payload and role. Reasoning fields lead on a chunk with no text, and annotations arrive once on the last text slice, matching a live stream's order. * fix(caching): replay a cached tool call after the text, like a live stream Only the role, reasoning fields and provider fields lead a replayed answer. Tool calls now arrive on their own chunk after the last text slice, and logprobs ride the first text chunk, so a cached answer with reasoning and a tool call keeps a live stream's reasoning, text, tool call order. * fix(caching): replay the reasoning before a cached tool call that has no text A cached answer that thinks and then only calls a tool now replays its reasoning on its own chunk before the tool call, the order a live stream sends, instead of one chunk carrying both. * fix(caching): keep blank-only text when replaying a cached answer with reasoning A cached answer that thinks, sends only whitespace as text, and then calls a tool now replays that whitespace between the reasoning and the tool call, instead of dropping it. --------- Signed-off-by: jesco-absolut <team@srswti.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
feda51453a
|
fix!: retain unattributed daily activity across reports and exports (#48925)
* fix: retain unattributed daily activity breakdowns * fix: scope daily activity bucket drilldowns consistently * fix: preserve all additive daily activity metrics * test: cover unattributed reporting end to end * fix: preserve daily activity bucket identities across exports and key details * test: drop SQL-text assertions changed by daily activity bucket fix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
4bc157e281
|
feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961)
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations * feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger * test(proxy): cover the dry run CLI and PrismaManager wrapper without a database * fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly * ci(codecov): stop carrying forward the roi-postgres and redis-compat flags Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c45b264024
|
fix(proxy): persist tag spend and drain success callbacks within the shutdown budget (#48988)
* fix: drain shutdown callbacks and persist pending tag spend * fix: catch shutdown timeouts on Python 3.10 * fix: share shutdown callback budget with HTTP draining * fix: type shutdown cancellation tasks and document its clock * test: retain callbacks after bounded worker cancellation * test(proxy): inspect the lifespan body after the SIGTERM wrapper split Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): keep the success callback enqueue on its own task The synchronous enqueue spawned the logging worker loop inside the awaited call and ran callbacks that per-call loops used to drop, which hung the SDK callback tests on the autouse worker flush. The shutdown worker flush covers the queued callback on its own, so the one-tick task is restored Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback The aggressive clear ran the rejected callback and marked it done against the queue it was never put in, so flush() could return while a real callback was still running and the shutdown drain cancelled it early. The rejected callback is now re-enqueued once the clear made room, and only runs uncounted when the queue is still full Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging_worker): build the aggressive-clear run set immutably Final tuple instead of an appended list, with the re-enqueue in a _requeue helper, and Final annotations on the overflow regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): track overflow callbacks through shutdown * chore(logging): explain shared callback ownership * test(logging): isolate deferred callback completion --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
paveln83
|
06cdee91da
|
fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails (#49035)
* feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines Gives OpenAITextCompletionHandler a process_output_streaming_response that assembles the buffered TextCompletionResponse chunks per choice index at end of stream, runs the guardrail over the assembled text, and writes text rewrites back into the first text-carrying chunk of each choice. Tests and live proof still pending. * fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs. * test(guardrails): cover streamed /v1/completions under post_call guardrails * fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through * fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter * fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent The content filter's streaming hook held back the end of a choice until its finish_reason, which litellm sends only for the first choice of an n>1 stream, so choice 1 lost its last characters on text and chat streams. A closing chunk now releases every unfinished choice when the stream ends. A blocked text stream's closing chunk reported 0/0 usage when the upstream sent none; it now carries usage only when the upstream reported it. * fix(guardrails): send no usage on the content filter's end-of-stream chunk * fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions * fix(guardrails): stream Responses API events through the content filter as they arrive * fix(guardrails): treat a null text completion choice index as choice zero when assembling streams --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
961fabbb76
|
feat(secret_managers): resolve oidc/google api_key references per request (#47703)
* test(secret_managers): cover oidc/google api_key references resolved per request Unit tests send a metadata server ID token for an admin oidc/google/<audience> api_key, reuse it within its lifetime, refresh it before it expires, fetch it off the event loop, fail closed with a 401 that names only the kind, and honor GCE_METADATA_HOST. The integration file scripts a metadata server and checks the bearer the upstream receives for config, stored credential and DB credential sources, plus a metadata failure * feat(secret_managers): resolve oidc/google api_key references per request An admin-set oidc/google/<audience> api_key now resolves on each call to the GCE metadata server's ID token for that audience, cached until a minute before it expires. Async callers fetch on a cache miss in a worker thread, so the event loop never waits on the metadata server. A failed fetch answers 401 naming only the kind, and GCE_METADATA_HOST points the fetch at another metadata server. * fix(oidc): bound the Google metadata fetch and answer 503 when it fails A Google OIDC api_key fetch now waits at most GCE_METADATA_TIMEOUT seconds (default 3), runs on its own thread pool instead of the loop's default executor, and sends the token read from the cache at call start even if it expires mid-call. A metadata timeout, network error, or 5xx answers a retryable 503, while a 4xx or empty token stays a 401. The expiry and empty-token warnings log the secret name on one line. * test(oidc): expect 503 when the metadata server answers 5xx in the integration test * fix(oidc): answer 503 on token-count routes during a metadata outage and fall back on a bad GCE_METADATA_TIMEOUT * fix(secret_managers): keep get_secret's metadata wait, share concurrent google token fetches, refuse oidc/google on pass-through A model's api_key fetch keeps the 3 second GCE_METADATA_TIMEOUT wait while get_secret callers keep the handler default. Any metadata transport error answers a retryable 503. Async cache misses for one reference share one fetch, and a call the reference may not go to is refused before any fetch. Pass-through routes answer 401 for an oidc/google credential instead of sending the reference text. The metadata client is closed after each fetch, and the unused sync resolver is removed. * fix(utils): refuse an oidc/google api_key on pass-through calls before any fetch A router-model or credential-name pass-through call with an oidc/google reference now fails before the metadata fetch and sends nothing upstream, since the pass-through config never sends the resolved key. The single-flight test also waits on the fetch-start signal with a bounded window instead of a fixed sleep. |
||
|
Mateo Wang
|
6cdb9959df
|
feat(router): opt in to non-chat deployment filtering (#46257)
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior. * fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions * fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog * fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags * fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released * fix(proxy): walk fallback layers without recursion and gate the compact Responses route The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes * fix(proxy): keep chat on rows whose cost map entry serves chat A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s * fix(proxy): check chat sent to a deployment id or backend model name * fix(proxy): check a model the read-through loads from the database * fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400 * fix(proxy): ignore a client model_group that is not a string when building rate-limit headers A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells. * fix(proxy): keep a non-string client model_group out of spend rows * fix(proxy): validate standard fallbacks before the typed fallback-group lookup * feat(router): opt in to non-chat deployment filtering * fix(router): skip unresolved local rate-limit fallbacks * fix(router): ignore caller-provided endpoint selector values * fix(router): normalize declared chat endpoint prefixes * test(router): cover preflight error-value behavior * test(router): derive replay metadata defaults from model info * fix(router): preserve fallback aliases and check synchronous text calls * test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission * test(routing): keep the endpoint gate's owned proxies on the worker's database * fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None so an omitted flag keeps the stored value The same-group retry of a stream dropped before content called the provider without the endpoint selection context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside endpoint_selection like every other router attempt * fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow Main's router fence lets code outside the router use only listed Router members and router_utils modules. The proxy's fallback probe now calls the listed get_routable_deployments with the request's call type, and the gate's 400 moves to litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's UnsupportedEndpointError, which the router retries on the group's other deployments, while the gate refuses the whole group * test(responses): cover the WebSocket error frame for a model that refuses the endpoint * fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup * fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame * fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path * test(router): type the load-warning test's caplog and router * test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket --------- Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain> |
||
|
Mateo Wang
|
be6d082bdc
|
fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166)
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts * refactor(anthropic): hold the chat completions document change for a later commit * fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai * fix(snowflake): turn file parts into document blocks for Cortex Claude * fix(prompt_security): sanitize chat file parts that carry file_data * fix: answer 400 or drop PDFs for models that cannot read them Models whose model_info or cost map entry sets supports_pdf_input to false, and Ollama, now get the PDF dropped under drop_params and a 400 from litellm without it. GigaChat uploads file parts, SAP keeps sending them as image_url data URLs, azure_ai passes dict content through, and annotations other than url_citation move to provider_specific_fields instead of failing the response. * fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider annotations to provider_specific_fields, and split them the same way when a stream is rebuilt. * fix(vertex_ai): send PDF file parts to Vertex partner models as image_url Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL. * fix(cost-map): leave native DeepSeek PDF support unset /v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there and would only change chat completions, which this PR does not otherwise touch. * fix(main): drop a user turn that held only a PDF under drop_params Dropping the PDF from a PDF-only user turn left an empty content list, which Foundry and OpenAI-shaped providers reject with a 400, so drop_params still failed for that request. The emptied user turn is now removed. * fix: read OCI PDFs, gate deployment base models, and stop empty drops OCI generic chat sends file parts as image data URLs again, so OCI Gemini keeps reading PDFs from /v1/messages and chat file parts. The PDF gate now reads model_info.base_model, drops PDFs under drop_params for azure_ai deployments that never declared PDF support (the old flattening behavior), and answers 400 when dropping the PDF leaves no user message. The cost map marks the Vertex partner text models that a live probe showed answering without reading the PDF. * test(integration): cover the PDF input gate, messages PDF file parts and provider annotations Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed * test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells * fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload * fix(main): PDF check reads the deployment's top-level base_model * fix(main): keep roles alternating after drop_params removes a PDF-only turn When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other * test: type the PDF and annotation tests the PR added * test: type the GigaChat, SAP and Snowflake PDF tests |
||
|
mika-lt
|
683b94b462
|
fix(fallback): read stored router_settings from the database on fallback writes (#48999)
* fix(fallback): read stored router_settings from the database on fallback writes
POST /fallback and DELETE /fallback/{model} read the current rules through the
config cache. When Redis refuses DEL, the eviction before that read fails and
Redis keeps serving the old copy, so each new rule dropped the earlier ones
The write path now reads the router_settings row straight from the writer
database and stores only that row plus the changed key, so config-file router
settings no longer get copied into the database row
* fix(fallback): edit the config file's fallback list when it owns the key
A delete of a rule only the config file holds answers the config-owned 400 again instead of 404, and restating a rule the file already has answers 200 without copying the file's list into the database row. Adds a writer-read regression test and an integration test against a Redis whose ACL refuses DEL
* fix(fallback): re-read the config file's fallbacks ownership on every fallback write
* test(fallback): run the real config reload in the fallback write tests
* test(fallback): key the refused-DEL Redis double by cache key
* fix(fallback): keep the config-owned 400 for an unset env reference and replace a non-object stored row
* fix(fallback): replace a stored router_settings row that holds a non-JSON string
* fix(fallback): compare a restated config-file rule against its resolved env value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
tavren27
|
b25ea48c4b
|
fix(anthropic)!: drop beta-gated body fields under drop_params (#48793)
* fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta When a provider leg drops an anthropic-beta value (the chat handler's allowlist filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields), also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates, mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the request degrades on that leg instead of failing with a provider 400. Copy-on-write so a fallback leg still sends the full request, and each strip is logged at WARNING. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove a leg that drops a beta sends neither the beta nor clear_at Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): strip beta-gated fields against the final body anthropic_beta list Converse counts the body anthropic_beta list as active, Invoke strips after its auto-injected betas are known and injects thinking-binding-controls for thinking.block_binding, test helpers are fully typed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn Models without supports_mid_conversation_system send no mid-conversation beta, so tool_addition and tool_removal blocks are dropped (and logged) before the turn is converted instead of riding along as user content. The Invoke output_config regression test reads the real cost map instead of patching _bedrock_model_supports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): annotate every new test local with Final and type the fixtures and returns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): strip beta-gated body fields only under drop_params Without drop_params the request now goes out as sent and the provider answers its own 400, like any other unsupported param. inline-tools-2026-09-15 also unlocks tool_addition and tool_removal blocks, so a leg that carries it keeps them. * fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field. * fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes. * fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown Vertex messages and Claude Platform chat now add the thinking betas when thinking carries block_binding or display updates. Converse sends only the caller's header betas when it has them. A provider with no row in the beta config forwards beta-gated fields instead of dropping them, an emptied trailing system turn that would leave an assistant prefill is sent as is, and the prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta that unlocks each field * fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks * fix(anthropic): validate messages only when a beta-gated drop runs A non-object messages entry on a Claude model without mid-conversation system support raised a raw pydantic ValidationError before the beta-config and drop_params checks. The strip now passes the messages through untouched unless it drops something, so the provider's 400 reaches SDK callers as BadRequestError again * fix(anthropic): keep system blocks with a non-string type instead of raising --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
ebf84f8649
|
fix(caching): read the cache key's model group from the router's metadata for the call (#48917)
* test(caching): audit #44542 response cache model group key * fix(caching): key the response cache by the router's metadata bucket only * fix(caching): read the router's model group before the caller's metadata * fix(caching): read caching groups only beside the router's model group A caller's own metadata.caching_groups on /v1/responses and /v1/messages could still pick the cache key, so a string group let one model group fill another's entry. The key now reads caching_groups only from the metadata that holds the router's model_group. * test(caching): type the planted caching groups parameter * fix(caching): read the model group from the metadata that carries the call's deployment A router call can carry a litellm_metadata it did not write, from the auto router's classifier embedding or an SDK caller, while the router writes its own group into metadata. The cache now picks the metadata source whose model_info matches the call's deployment, then falls back to the previous order. * fix(caching): keep main's model group lookup for calls outside the router --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
41602e318d
|
fix(proxy): stream a guardrail block on /v1/messages as Anthropic events (#49010)
* fix(proxy): stream a guardrail block on /v1/messages as Anthropic events * fix(proxy): keep post-call guardrails off the streamed messages block The streamed block is sent as-is, like the Responses endpoint's block, so post-call guardrails and streaming hooks never scan the proxy's own block message. Tests cover a post-call rail on the stream and the call id on the JSON block * test(proxy): expect Anthropic events from a streamed messages guardrail block The legacy guardrail suite pinned the old single bare data: line, which is the malformed block this branch replaces. The test now checks the six Anthropic stream events and the block text, so it fails if the bare line comes back * test(proxy): block messages through a real pre-call guardrail in endpoint tests The block tests replaced ProxyBaseLLMRequestProcessing.base_process_llm_request on the class to raise the block. They now send a real request through the endpoint with a pre-call guardrail that raises the passthrough block, and check that the x-litellm-call-id header carries the request's call id --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tavren27
|
e7d3b24d7c
|
fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs (#48888)
* fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs and logging payloads * fix(router): keep the mirror flag when litellm_metadata is set and strip a client-sent is_silent_experiment Spend rows read router metadata from litellm_metadata when a call carries it, so the router now sets the flag there too. The proxy drops a client-sent is_silent_experiment with the other router-owned keys, so real traffic cannot be recorded as mirrored. * fix(logging): mark mirrored calls with a router-only key so clients cannot set is_silent_experiment Spend rows and the standard logging payload now derive is_silent_experiment from _litellm_silent_experiment, which only the router writes and the proxy and pass-through routes strip from client metadata. A caller's own is_silent_experiment key stays untouched, so it keeps its current effect on mirroring and on provider metadata, and unmirrored calls log false in both places. * test(langfuse): expect is_silent_experiment false on every logged generation * fix(logging): mark mirrored calls with a router-only enum value so no client route can set is_silent_experiment The router stamps SilentExperimentMarker.MIRRORED, which client JSON can never produce, so batches, Responses WebSocket frames and pass-through need no strip. Langfuse layers the flag after its requester-metadata dedupe. * fix(logging): set is_silent_experiment only on mirrored calls so real calls keep their payloads Real calls no longer gain an is_silent_experiment key in spend rows or the standard logging payload, so OTEL, S3, GCS and every other logger receive what they did before. The Langfuse regression test now runs the real prompt helper. * fix(logging): keep the mirror marker out of callback metadata and read the spend-row flag from the logging payload Logging now strips the router's mirror marker from the request metadata buckets before any callback or vendor sees it, without touching the router's own dicts, so retries keep it. Spend rows read is_silent_experiment from the standard logging payload, which survives the collector's JSON round trip, so collector-mode rows keep the flag too. * fix(logging): mark mirrored calls through a context variable so router usage counts them once * fix(router): leave callback-started calls unmarked and stop client flags from turning mirroring off Logging handlers now run outside the mirrored-call context, so an LLM call a success or failure callback starts while logging a mirrored call is not flagged as mirrored. The guard against mirroring a mirror reads the same context variable instead of request metadata, so a client-sent is_silent_experiment no longer skips the mirrors. The unused module-level get_standard_logging_metadata stops copying resolved keys from raw metadata. * fix(router): keep main's client flag mirroring skip and the other logging metadata keys A client's is_silent_experiment: true skips mirroring again as on main, the module-level get_standard_logging_metadata drops only the flag, and the silent experiment tests wait on a logged-call signal instead of polling a deadline * fix(logging): take the mirror flag in Prometheus labels and Datadog cost tags only from the router * test: type the new parametrized tests and keep the recorder from changing its argument * fix(datadog): keep client request tags from setting the silent experiment cost tag A request tag is_silent_experiment:true set the Datadog cost tag on a direct call. Only the router flag sets it now. Also covers the sync failure handler's mirrored-call reset with a failure function callback test. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
simonp9
|
e8f0735d8a
|
feat(proxy): let a Claude Code gateway user choose the session team (#49001)
* feat(proxy): let a Claude Code gateway user choose the session team The gateway's browser verification page now lists the developer's teams, and the device code grant mints the session for the picked team. A team the developer is not on is refused with a 400, and a form without a team keeps the first-team default. * fix(proxy): refuse a stale gateway team pick and require an explicit choice * fix(gateway): list the browser-picked team first so an older replica mid-deploy issues it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c489aa7117
|
feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708)
* fix(proxy): keep every startup line JSON when json_logs is on Boot-time messages the proxy printed straight to stdout (banner, feedback box, model list, cache and callback notices, policy table, migration notices, CLI launch notes) and the first records of uvicorn- or granian-spawned workers came out as plain text while json_logs was on, breaking log pipelines that parse every line as JSON. - print_startup_message() prints unchanged in plain mode and, with json_logs on, emits one INFO record on the proxy logger (ANSI stripped) that skips the logger and handler level checks like the print it replaces; show_banner() routes to it in JSON mode only - the CLI exports JSON_LOGS=true when the config sets json_logs so spawned workers log JSON from their first record - the --enforce_prisma_migration_check deprecation notice prints after the config is read so a config-only json_logs covers it Resolves LIT-1624 * fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize() Since #46803 the `lite` shell calls show_banner() against an uninitialized litellm package, and reading litellm.json_logs inside it initialized the whole SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR removed on small VMs). show_banner() is back to its main version; the proxy's initialize() now emits the JSON banner record itself and calls show_banner() only in plain mode. The banner unit tests move to the initialize() call site. * fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger uvicorn applies --log_config after proxy_cli has switched JSON logging on and before initialize() switches it on again through load_config(). A logging config that names "LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where initialize() emits the banner and the feedback box, so at one worker both went nowhere; two-worker children import litellm after uvicorn's logging setup and were not affected. print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON handler uses, when the proxy logger has no handler of its own. Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the banner and the feedback box as plaintext, |
||
|
devin-ai-integration[bot]
|
bee9f679dc
|
fix(github_copilot): treat a malformed api-key.json like a missing one (#44723)
* fix(github_copilot): treat a malformed api-key.json like a missing one * fix(github_copilot): read the key and the endpoints of api-key.json as separate strict views * fix(github_copilot): ignore a non-UTF-8 api-key.json like any other malformed file Read the cached key file as bytes so a UTF-16 file, which Windows PowerShell 5.1 writes with >, fails JSON validation instead of raising the decoder's error, and the request gets the refresh or the login hint * fix(github_copilot): decode UTF-16 key files and refresh over a malformed endpoints field * fix(github_copilot): decode a UTF-16 access-token file and refuse an undecodable one like a missing one * fix(github_copilot): use the refreshed key when api-key.json cannot be written * fix(github_copilot): keep a refreshed key that cannot be saved in memory * fix(github_copilot): serve a kept key and its endpoint together after the key file is repaired A key held in memory is now checked before api-key.json, so the token and the endpoint come from the same source, and the map holds one entry per key file with the access token beside it * test(github_copilot): type the authenticator test fixture and the tests this branch adds --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
184ec7b894
|
fix(vertex_ai): build the cachedContents collection URL under a custom api_base (#44783)
* fix(gemini): use collection endpoint for custom context cache * fix(vertex_ai): build the cachedContents collection URL under a custom api_base * fix(vertex_ai): keep a gateway prefix that contains /publishers/ * test(integration): expect the cachedContents collection URL on a custom Gemini api_base * fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix * fix(vertex_ai): cache under a publisher model api_base in that model's location A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so no cache is created that the chat call cannot use. The cache body takes its project and location from the api_base model, matching the collection URL. * fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field. * fix(vertex_ai): check the cache token minimum against the api_base's model The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create. * fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached --------- Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e85cb4052e
|
test: fix flaky spend-log cleanup budget, login lockout, and wedged Redis tests (#47108)
* test: stabilize cleanup, lockout, and post-kill timing tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bound the wedged Redis burst p95 by the socket timeout the test sets The burst p95 under a fixed 0.75s cap tracked how fast the box ran the 30 concurrent writes, so a loaded runner failed it at unchanged code. A handler that waits on the wedged publish takes at least the coordination Redis socket timeout, so the test now sets that timeout and asserts the p95 stays under it. * test: pin the lockout clock and bound the wedged Redis burst by the proxy's 2s wait cap The lockout tests now set the login throttle's clock instead of reading the real one, so Retry-After is pinned at 77 and a round-down fails on every run The wedged Redis burst p95 is now bounded by the 2s cap the proxy puts on its own Redis waits, so a burst wait capped under the 5s socket timeout still fails the test The fresh post-kill clients keep the harness's 2s keep-alive expiry, here and in the two tests #47080 changed * test: tell ceil from round in the lockout tests and bound the burst by the imported eviction cap The refused attempt now runs with 76.4s left on the 77s block, where ceil gives 77 but round and int give 76, so both lockout tests fail if Retry-After stops rounding up. The burst bound imports _BULK_EVICTION_WAIT_SECONDS instead of copying its 2s value, so lowering the cap moves the bound with it. * test: bound the wedged Redis burst by the shorter of the eviction cap and the socket timeout Any wait the proxy puts on the stopped Redis lasts at least the shorter of the two, so the bound now follows whichever one a later change lowers. * test: drive the tool index cleanup test with the manual clock and keep the post-kill block as on main The post-kill reset is already fixed on main by making kill() wait for the worker's last thread, so the fresh-client and keep-alive edits go back to main's version. The tool index cleanup test now advances the injected clock instead of timing the real one, and the lockout clock helper gets a type hint. * test: advance the spend log cleanup tests' manual clock through its own method * test: freeze the throttle clock in the login_utils lockout tests and drive the run budget tests with the manual clock --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b93d26bc85
|
fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path (#45330)
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path * refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge * fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper * fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints Also drop a marker set on the message item itself, sanitize before the cache_control conversion so a client typo no longer erases that breakpoint, and skip items that carry no marker so unmarked history costs no extra walk * fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a litellm_proxy alias) kept its valid markers before this PR and must keep them under drop_params. base_model is read from litellm_params first, then model_info, as chat completions does. * fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged * fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers * fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied * fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one Under drop_params the marker step now hands back the caller's own objects when every marker is already wire-safe, so a request with valid markers is no longer copied and walked a second time. A deployment that turns cache_control into markers keeps the caller's own markers too, whatever its base_model says, so one request never gets two answers about breakpoint support. On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion, so the block gets the cache hint cache_control alone would give, as on /v1/responses. * test(responses): keep a null marker and the original key order on a model that reads breakpoints --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
rj-cairn
|
42d2909210
|
test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965)
* test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module Session scope booted one 2-worker proxy per xdist worker and kept it until the worker's session ended, so a CircleCI node holding two of these files piled up proxies until the VM stopped responding. Module scope lets the harness group each file's rig tests onto one worker and tears the proxies down after them. * docs(tests): keep proxy fixtures module scoped in integration tests * docs(tests): say how the harness groups proxy fixtures in the integration AGENTS.md rule --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
032342d2b7
|
fix(responses): resume background response streams with stream and starting_after (#48819)
* fix(responses): resume background response streams with stream and starting_after
Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}
Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally
* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams
A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.
* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve
A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.
* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params
* fix(responses): keep deployment stream off retrieves and reject oversized cursors
* fix(responses): keep router default stream off plain retrieves
* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing
* test(responses): replay a real stream in the billing test and type the fake wrapper chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
simonp9
|
dff474e450
|
fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912)
* fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn * fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards * fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set * fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit * fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum * fix(responses): leave a turn without a phase when its content message cannot be read * test(responses): add black box tests for chat turn history replay * test(responses): expect the replay record on bridged chat replies * test(responses): expect the replay record in bridged chat translation cases * fix(streaming): keep the served service tier on the finish chunk the stream adds * test(responses): read the expected chat message without JSON validation so the replay matcher stays --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
9a68573e9b
|
fix(responses): keep each Codex tool once when its tool list repeats across turns (#49006)
* test(responses): cover Codex tool lists repeated across turns on the Bedrock chat bridge and Mantle * fix(responses): keep each Codex tool once when its tool list repeats across turns * test(responses): type the new tool list dedupe unit tests |
||
|
tavren27
|
cbf4e40d38
|
fix(proxy)!: stop labelling management and health rejections as auth failures (#48739)
* test(proxy): cover management and health routes labelling rejections as auth failures * fix(proxy): name management and health route errors by their status instead of auth_error * fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls /key/list and /key/aliases now share the catch-all helper too, so they stop adding the Authentication Error prefix and typing a 400 as internal_server_error * fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error A real outage surfaces from prisma's query engine as a builtin AttributeError, which the prisma-module check skipped, so the route catch-alls still answered 500 or 400 * fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status * fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503 * fix(proxy): answer 503 for database failovers and label provider auth errors like chat A database that refuses connections while starting up, shutting down, in recovery, or read-only after a failover now answers 503 no_db_connection instead of 400. A provider's 401 or 403 keeps the type /chat/completions gives it, while a 401 or 403 the proxy raises keeps auth_error. An exception with no message is named by its type. * fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause * test(proxy): expect the OpenAI error shape from management route refusals Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tessellark
|
dbaae521f9
|
fix(responses): drop the null created_by OpenAI rejects when output items are replayed (#48783)
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed * fix(responses): drop a null created_by from replayed input items too A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null. * fix(responses): drop a null created_by from counted input items too * test(responses): cover saved reasoning nulls and re-compaction of a saved conversation * fix(responses): drop a set created_by from input under drop_params * fix(types): type the Responses output serializer's parameters * fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
e4932a4539
|
test(router): set string fallback targets after Router init in the sdk cooldown cases (#48954)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tavren27
|
07d314beb5
|
test(responses): audit #47445 (#48902)
* test(responses): audit #47445 Integration cells for billing every round of a streamed MCP auto-execute on one spend row, on a native Responses upstream and the chat bridge, across tool rounds, a rejected or broken follow-up, a post-call guardrail block, a client hang-up, the response cache, and the next bridged turn * test(responses): check a broken follow-up bills its own nonzero usage --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
zoveki
|
77cad6e903
|
test(proxy): audit #46997 (#48918)
* test(proxy): audit #46997 Add request-path cells for the global proxy spend counter: a key with no user spends toward litellm max_budget, and two proxies sharing Redis add every request to one counter that matches the proxy budget row * test(proxy): wait for the recorded budget spend and shared counter before checking them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
49ccce7813
|
test(extraction): audit #47473 (#48898)
* test(extraction): audit #47473 * test(extraction): check over-budget extraction never reaches Cloudflare --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moe-berri
|
9e878cfae2
|
test(gemini): cover every web source for a grounded passage (#48861)
* fix(gemini): preserve every grounding web source * test(gemini): cover all grounding sources per passage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Zabir Raihan <zabirraihan@icloud.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
joshua-berri
|
d30c702c5c
|
feat(accounting): add durable event ledger writer (#48883)
* feat(accounting): add durable event ledger writer * fix(accounting): validate delegated transaction managers locally * docs(accounting): clarify ledger integration contracts * fix(accounting): return ledger conflicts as values and keep cleanup on the kind index Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(accounting): import assert_never from typing_extensions for Python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(accounting): annotate ledger integration test locals as Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(accounting): finish ledger test conventions --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |