mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-04 07:27:51 +00:00
31 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mika-lt
|
c4b01b2ac8
|
fix(decisions)!: honor a deployment's own supports_audio_input and supports_video_input flags (#49207)
* fix(decisions): honor a deployment's own supports_audio_input and supports_video_input flags A Decisions deployment's model_info flags for audio and video now win over the cost map, per media kind, the same way supports_box_questions already does. An unlisted model flagged true gets its clips sent upstream, and a false flag refuses them (or drops them under drop_params). OpenAI-format providers still refuse clips since their request shape has no field for them, and the SystemOne request builder no longer re-reads the cost map. * fix(decisions): ignore media flags on a deployment whose own mode is not decisions A chat-mode deployment's supports_audio_input describes chat input, so it no longer turns on audio or video clips for Decisions. The cost map decides for it, as before. Adds a proxy integration test for flagged, partly flagged, unflagged, and drop_params deployments on both /v1/decisions and /v1/systemone, and for caller-sent model_info. * fix(decisions): follow the cost map's mode for a flagged deployment without its own mode A chat deployment flagged for audio or video usually leaves mode unset, so its flags described chat input yet still let clips through on /v1/decisions. When the deployment sets no mode, the cost map's mode for the model now decides whether its media flags apply, and an unlisted model still takes them. The audio and video lookups also run only when the request carries a clip. * fix(decisions): read the mode a flagged deployment falls back to from the shipped cost map The router copies each deployment's model_info into the model's shared cost map row, so another deployment's mode there could switch off a flagged deployment's media flags. The fallback now reads the cost map as loaded, untouched by router registrations * fix(decisions): route clips in a load-balanced group to a deployment that takes them A Decisions or SystemOne request carrying audio or video now skips group deployments whose own flags or the cost map refuse that media, and keeps the whole group when none takes it, so the 400 still names the media. * fix(decisions): keep deployments that drop or cannot resolve clips in the media filter * fix(router): skip clip-refusing Decisions deployments after cooldown and tag filtering Running the media filter before cooldown turned the 400 a group answered when its only clip-taking deployment was cooling down into a 429. It now runs next to the image filter, just before order filtering, in both the async and sync selection paths. * fix(router): honor router default drop_params when skipping clip-refusing Decisions deployments * test(router): list the router default params helper in the router coverage check * test(bedrock): stop reloading the cost map module in the GovCloud fixture --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
ec4f66bb94
|
fix(router): budget classifier input by provider limits (#49306) | ||
|
mika-lt
|
71babd931b
|
fix(llms): move later developer messages for models declared system-first (#48897)
* fix(transformation): merge consecutive system and developer messages for non-OpenAI providers and Responses API
* fix(base_utils): keep later cache_control and billing metadata boundaries when merging system messages
* perf(base_utils): fold consecutive system messages in a single linear pass
The tuple accumulator passed to reduce copied the whole prefix on every step,
making map_developer_role_to_system_role O(n^2) in the number of messages.
Message count is not bounded by default, so a caller could tie up a proxy
worker with thousands of small messages before any provider call. Use a list
and merge into its last element in place instead.
* fix(base_utils): hoist later developer messages into the leading system message for non-OpenAI providers
A developer message that arrives after the first user turn (Codex CLI's
per-turn permissions item, a re-injected instruction after compaction, a
mid-conversation policy update) was translated to a system message in
place, and system-first chat templates such as Fireworks' qwen3.8 reject
any system message that is not at the beginning with 400. The fold that
merges consecutive system messages never saw it because a user turn sat
in between.
Move every developer message that follows the leading system block to
the end of that block before the fold, so it merges into the single
leading system message; client-authored mid-conversation system messages
stay where the client put them.
* fix(openai): hoist later developer messages into one leading system message on OpenAI-compatible backends
Developer role translation moves onto the provider config. OpenAI's own endpoint,
Azure OpenAI, and the o-series keep their current behavior. Every other config
inheriting OpenAIGPTConfig (Fireworks, DeepSeek, Together, vLLM, Groq, openai/
with a custom api_base) hoists developer messages that arrive after the leading
system block into that block and folds the block into a single system message,
which is what system-first chat templates require. Providers with their own
message format are untouched.
Messages carrying cache_control or Anthropic billing metadata fold as their own
text blocks, so breakpoints and provider-side stripping survive the merge. The
Responses bridge goes back to prepending instructions as a separate system
message and lets the hoist fold it.
* refactor(base_utils): fold each run of system messages in one pass
The pairwise fold recopied the accumulated system text or block list on every
merge, so a request carrying many consecutive system or developer messages cost
quadratic time in preprocessing. Group consecutive system messages once and
build the merged message from the whole run: plain strings join with a blank
line, and any cached, billing, or block content keeps one text block per
message. The name of the last named message in the run survives, as before.
* fix(base_utils): keep a closing developer message in place and hoist for OpenAIConfig backends
A developer message that closes the conversation stays where the client put it, translated
to system in place, so the request never ends on the assistant's turn. OpenAIConfig and its
subclasses (Azure AI Foundry, GitHub Copilot, ChatGPT, RAGFlow) now share the OpenAI-compatible
hoist, litellm_proxy passes developer messages through to the downstream proxy, and the Anthropic
billing header literals use the shared constant
* fix(base_utils): keep a closing developer message in place only after an assistant turn
A developer message that closes the conversation after a user or tool message hoists like
any other, so the request shape the ticket reports keeps working on system-first templates;
only a closing run that follows an assistant turn stays in place
* refactor(openai): reuse is_openai_backed_api_base for the hosted endpoint check
* test(base_utils): cover an empty system message inside a block-content run
Codecov flagged the empty-string branch of _text_blocks as the one line the
PR left uncovered. The existing empty-string test only hits the plain-text
join; this one forces the block path by merging an empty system message with
a developer message that carries list content.
* fix(openai): hoist developer messages for litellm_proxy and harden the system fold
- drop the LiteLLMProxyChatConfig passthrough so litellm_proxy/ hoists later developer messages like every other OpenAI-compatible endpoint, whichever backend or LiteLLM version sits downstream
- a system message with null or missing content inside a folded run is dropped instead of raising KeyError('content')
- a message-level cache_control on a list-content member lands on its last block so the breakpoint survives the fold
* fix(litellm_proxy): keep the AllMessageValues import the transform signatures still name
* fix(fireworks_ai): hoist developer items after the first turn on the native Responses path
The native Fireworks Responses path landed on staging mapping developer input items to
system in place, so a developer item after the first input item still hit the same
"System message must be at the beginning" 400 on system-first chat templates. The
ordering rules now live in one shared helper over item roles, and the native path
reorders through it before mapping, keeping a closing developer item after an assistant
turn in place exactly like the chat path does
* fix(transformation): carry the developer-message hoist onto main
|
||
|
rj-cairn
|
d06b889d41
|
fix(streaming): keep merged think tags balanced and split SSE lines only on CR/LF (#48755)
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream The Gemini and Vertex AI streaming path read the SSE body with httpx iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085, U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds one of them was cut in two. The tail is not a data: line and the head never parses, and the accumulate-and-retry fallback then carries the unparsed prefix forward, so every later event is appended to it and the stream ends with no more content and no usage. SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines split decoded text on exactly those, treating a CR at the end of one chunk and an LF at the start of the next as one break. The first request and every long-decode hop now read through them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks. * fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed Switch the remaining httpx stream readers (Anthropic chat, Google GenAI, Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a raw U+2028 or U+2029 inside a JSON string no longer splits an event. Keep the chunk that closes a think block in the logged and cached response when a stream ends without a finish_reason, close the block before a legacy function_call, and make sync Codestral FIM streaming send stream=true and use the text-completion-codestral stream parser. Test fakes that only mocked iter_lines now stream real httpx bytes and assert the content they feed in. * fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use split_sse_lines from sse_lines.py, which also gains a str.split fast path. Adds tests/unit/llms/codestral to the unit workflow shard. * fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines * test: type the new stream tests, drop a comment, and benchmark the SSE line splitter * test: type the iterator class parameters in the line-separator stream tests * fix(responses): stream tool call arguments opened in a text chunk * fix(responses): hold a tool call that shares a chunk with text until the text ends * test: type the a2a SSE handler helper and the sandbox fake responses --------- Co-authored-by: Tim Vink <vinktim@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> |
||
|
michelligabriele
|
11559ed895
|
fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator (#47832)
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator * test(streaming): cover reasoning text and error frames that mention [DONE] --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
8e2e99670d
|
fix(router): continue Responses mid-stream fallbacks on models without assistant prefill (#47672)
* fix(router): avoid assistant prefill in Responses fallbacks * fix(router): keep prefill for Claude fallbacks that take it and skip constrained continuations A Responses fallback to a Claude model that takes assistant prefill now gets the partial response as prefill again, so it continues mid-word without the prompt-shape drift. Every other fallback keeps the request to continue. A stream cut after text no longer continues when the request constrains its output (a json_schema or json_object text format, or a forced tool_choice), since a continued JSON document or tool call would not parse. The client gets the primary's error, as on the chat path. The fallback's re-sent leading space at the seam is dropped, and the prefill shaping runs per deployment after the encrypted reasoning strip, so stripped reasoning is never replayed. * fix(router): guard SDK text_format, narrow forced tool_choice, read Claude base_model, and cap Responses continuation output Cut Responses streams that asked for structured output through the SDK's text_format now surface the primary's error instead of stitching two models' JSON. Only a required or named tool_choice counts as forced, so allowed_tools in auto mode and hosted tool choices still continue. A Claude deployment named only through base_model keeps its prefill continuation, and the continuation's max_output_tokens is lowered by what the cut stream already emitted. * fix(router): quote the partial's ending in the continue request and surface the primary's error after a cut tool call * fix(router): stop Responses fallbacks from continuing content-filter cuts and client-run tool calls A content filter that cuts a /v1/responses stream mid-answer now returns its error, as on chat, instead of letting a content_policy_fallbacks backup finish the filtered text. Shell, apply_patch, computer, local_shell, and MCP approval items count as a started tool call. Chat continuation goes back to treating any tool_choice object as constraining, as on main * fix(responses): count a stream of tool arguments as a started tool call * fix(router): let Responses fallbacks continue under allowed_tools auto and after server-run tool calls A Responses tool_choice of allowed_tools in auto mode no longer blocks the continuation, since the model may still answer in text. Hosted MCP calls and the bridge's server-run tool search no longer count as a started client tool call. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Niraj Kumar
|
6e09664118
|
fix(decisions): catch ValidationError from validate_json in transform_decisions_response (#48660) (#48662) | ||
|
ishaan-j-berri
|
ebe0871a28
|
fix(decisions): preserve omitted probabilities (#48449) (#48524) | ||
|
Mateo Wang
|
81d8c9af38
|
fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508)
* test: bill each completed request once and price unpriced returned decision models at the requested model Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(logging,decisions): bill each request once and price returned decision models only when they carry rates Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(decisions): leave the double-billing fix to #47510 and keep the decisions pricing and missing-answers fixes * fix(decisions): name malformed answer entries and pick the Arize cache-hit span by attribute * test(decisions): cover a returned-model row with metadata but no rates * fix(decisions): keep upstream reply text out of the parse error so it stays a 500 * fix(decisions): answer every malformed upstream reply with a 500, whatever text it carries A reply that fails the decisions response schema now raises DecisionsResponseValidationError, which the decisions call maps straight to InternalServerError instead of passing it through exception_type. Its text matching read an answer key like rate_limit, or a value like "Request timed out", as a 429 or a 408, so the router cooled down a deployment for a reply that fails the same way on every retry. The message also drops the empty location for a reply that is not an object, the Arize family test loses the cache_hit entries main already added, and the Bedrock Mantle count-tokens wire test calls _metadata_deployment, since main renamed the _deployment helper under it. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Charles
|
5a1fa1c879
|
fix(responses): keep strong refs to stream and websocket logging tasks (GC race) (#31074)
* fix(responses): keep strong refs to stream and websocket logging tasks * test(responses): patch _spawn_logging_task instead of asyncio.create_task The success-logging dispatch in _log_completed_response now goes through _spawn_logging_task, so tests that patched asyncio.create_task no longer observed it (and the sync ones hit get_running_loop with no loop). * test(responses): assert stream and websocket logging tasks survive garbage collection Adds regression tests that force a collection while the logging task waits on an otherwise unreferenced future, and types the new test parameters * test(responses): drive drain and GC tests through a real logging object with gated callbacks * test(responses): use asyncio.wait_for so the logging tests run on Python 3.10 * test(guardrails): type the spawn_logging_task tracking wrapper --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
6d0d0bb68b
|
feat: add Nace Drex decisions provider (#46523)
* feat: add Nace Drex decisions provider * fix: expose Nace in dashboard provider metadata * ci: include Nace tests in the provider shard * fix: complete Nace provider catalog and preserve OpenRouter tests * test: mark Nace test directories as packages * fix: share Drex score parsing across decision providers --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
7ddc9f1c96
|
fix(bedrock)!: keep reasoning, thinking blocks and the json_mode finish_reason on fake streams (#46863)
* fix(bedrock): keep reasoning, thinking blocks and the json_mode finish_reason on fake streams A Bedrock Converse model without native structured output fakes the stream when response_format carries a json_schema. The fake stream transformed the response without json_mode and rebuilt a chunk from text, tool_use and token counts only, so it dropped reasoning_content, the signed thinking_blocks and the reasoning token count, and ended on finish_reason tool_calls. The three fake-stream sites now pass json_mode to _transform_response and stream the resulting ModelResponse through the shared convert_model_response_to_streaming converter, which now numbers each streamed thinking block with its index the way real streams do * fix(bedrock): parse text tool calls on Converse fake streams like the non-stream answer * fix(responses): stream the answer text when one chat chunk carries reasoning and the answer * fix(streaming): keep every field on the OpenAI and openai_like fake streams and split multi-block thinking Azure o-series and openai_like fake streams now use the shared MockResponseIterator, so extra tool calls, reasoning and cached and reasoning tokens survive; the Bedrock-local iterator is deleted. The /v1/responses bridge also splits a chunk whose thinking block is signed or redacted with no reasoning text, and the sync iterator closes the reasoning item before the answer opens, like the async one. The /v1/messages bridge gives each thinking block in a chunk its own content block, starts a new block when the thinking block index changes, and streams redacted thinking as redacted_thinking. Snowflake replays drop the stream index from thinking blocks. * fix(anthropic): stream each thinking block's text once on /v1/messages An unsigned thinking chunk now opens its content block with empty text, since its thinking_delta carries the text. A multi-block chunk keeps each piece's thinking block, so an unsigned block followed by a signed one keeps its own text and index. * docs(anthropic): describe which thinking block start carries its text * fix(streaming): close merged reasoning before the answer when one chunk carries both * test(streaming): cover fake-stream reasoning across the proxy bridges and type the Snowflake replay test |
||
|
Mateo Wang
|
3fec5075fa
|
feat(typesafe): support Jev 1.14 images, box questions, and long-context headers (#46596)
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers * fix(cost): drop the typesafe/jev-1.14.0 entry main already prices * fix(typesafe): send images only with the images header and box questions only to models that answer them TypeSafe takes assets and input_image parts only when the request carries TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like any other provider that cannot take images. Box support now comes from a supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so jev-1.13.0 and jev-latest refuse or drop a box question before the call. A malformed provider_specific_header answers 400 instead of 500. * fix(typesafe): allow null box confidence and validate asset references and base64 * fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
6bd5580425
|
fix(streaming): add CustomStreamWrapper.close() and release the connection when a sync stream ends (#46354)
* fix(streaming): close sync streams so the connection goes back to the pool CustomStreamWrapper gets a sync close() (plus context manager support) that closes the wrapped provider stream, and both the sync and async wrappers close the provider stream when the stream reaches its end. The Anthropic stream iterator keeps the httpx response it reads so close()/aclose() can release the socket after message_stop instead of leaving it checked out of the pool Co-authored-by: Blair Jordan <blair.j.jordan@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): keep the basedpyright ceiling flat in CustomStreamWrapper.close() Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): close the response on the shared sync stream path too BaseModelResponseIterator gets a sync close() next to its aclose(), and the sync branch of BaseLLMHTTPHandler hands it the httpx response the way the async branch already does, so providers built on it release the socket on close() instead of waiting for garbage collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(streaming): unit tests for CustomStreamWrapper.close() and the shared iterator close() Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): close the provider stream before re-raising a mid-stream error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Blair Jordan <blair.j.jordan@gmail.com> |
||
|
devin-ai-integration[bot]
|
2f3ccbe7c7
|
feat(decisions)!: send images to Cloudflare Clef through its images field (#45734)
* feat(decisions): send images to Cloudflare Clef OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise. * fix(decisions): follow drop_params for images and use the call timeout for image downloads * fix(decisions): drop a malformed images entry on its own under drop_params * fix(decisions): refuse images next to input whatever drop_params says * fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs Refuse more than 4 images for Cloudflare clef before any download, keep the caller's timeout as one deadline across image fetch retries, accept images next to input under drop_params, validate image strings and objects, and pass the caller's image URLs (not inlined bytes) to logging callbacks. * fix(decisions): check input_image URLs and clef image sizes * fix(decisions): accept data URL media type parameters and check clef image types * fix(decisions): count clef images after dropping unsupported types and flag clef vision --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
a86ad9ebf8
|
feat(decisions): support Clef-omni audio and video (#45811)
* feat(decisions): support Clef-omni audio and video * fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes The clip patterns now mirror Cloudflare's input schema, so a data URL or content_type carrying parameters like audio/webm;codecs=opus is accepted. The four proxy routes get media tests of their own, the backup cost map is synced with the new Clef-omni rows, and the docs page moves to the docs repo as a follow-up. * fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields * fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation * fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models * fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers * test(decisions): cover the Fireworks System One router in the media refusal cells --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a7a3a6b375
|
test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e * test: fix disk cache collection, ungate saturation check, drop RSS memory tests * test: assert one delivered log per completion in Datadog and LangSmith checks * test: drop SDK-direct e2e copies and align cost-map path filter cases --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
e0398455d0
|
test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759)
* test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
805bb6888b
|
test: make 38 legacy live base-class translation tests offline (#45351)
* test: make 38 legacy live base-class translation tests offline * test: tighten offline base-class translation test assertions * test: cover remaining nova invoke and xai base copies offline * test: assert outbound bodies, restore dropped providers and fix shared tool dict mutation in base chat translation tests * test: assert outbound thinking bodies and streamed thinking blocks across anthropic and bedrock converse * test: restore volcengine/voyage max_retries and openai timestamp_granularities coverage, assert rerank id/meta/cost and scope the interactions transport fixture * test: call the raw provider helper for thinking stream cases * test: flatten thinking blocks before collecting signatures --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
0e48048bd5
|
chore(decisions): remove the System One converters and OpenAI spec types left dead by #45214 (#45442)
#45214 routed both decision formats through the shared decisions IR in litellm/llms/base_llm/decisions/transformation.py and litellm/types/decisions.py. That left litellm/llms/base_llm/decisions/systemone.py (to_system_one_request, question_keys, to_decisions_response, the SystemOne* models and their adapter) and litellm/types/openai_decisions.py with no importer outside their own unit tests, so both modules and both test files go. Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
33d908e0ae
|
feat(decisions): add OpenAI as a Decisions provider behind a shared decisions format (#45214)
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions The System One request format moves to /v1/systemone and /systemone. /v1/decisions and /decisions now accept the OpenAI Decisions API format, translate it into a System One request, route it through the same pipeline, and translate the answers back. * fix(decisions): import assert_never from typing_extensions for Python 3.10 * fix(decisions): cap OpenAI-format questions at the System One limit /v1/decisions accepted up to 200 questions, but the System One request it translates to takes at most 128, so 129 to 200 questions failed with an internal validation error. Both now share MAX_DECISION_QUESTIONS. * fix(decisions): return 400 for bodies that are not JSON * test(decisions): send System One bodies to /v1/systemone in integration tests * feat(decisions): add OpenAI as a Decisions provider System One requests to openai/ models are translated to OpenAI's Decisions shape on the way out and OpenAI's answers are translated back, so both /v1/systemone and /v1/decisions can route to gpt-6-luna. * refactor(decisions): move the OpenAI wire translation into llms/openai * feat(decisions): route both request formats through a shared decisions IR System One and OpenAI-format bodies now convert to one internal representation, and each provider translates it to its own wire format. OpenAI deployments receive the caller's messages, images, names, typed choice values, level descriptions and safety_identifier unchanged. System One providers return a 400 for image input. Cached and cache-write tokens are priced on both response shapes, and provider response extras survive the round trip. * fix(decisions): keep provider fields inside System One answers * fix(decisions): load on Python 3.10 and 3.11 and bill cached tokens once under custom pricing Decisions IR dataclasses used a mappingproxy default, which is only hashable from Python 3.12, so import litellm failed on 3.10 and 3.11. Decisions usage now reaches cost_per_token as a Usage with prompt_tokens_details, so custom pricing no longer adds cached and cache-write tokens on top of input_tokens that already include them * fix(decisions): honor litellm OpenAI key and base settings for the openai provider OpenAI decisions now resolve the key from litellm.api_key, litellm.openai_key, then OPENAI_API_KEY, and the base from litellm.api_base, OPENAI_BASE_URL, then OPENAI_API_BASE, matching other OpenAI calls. Providers own their configured key and base lookup through the endpoint config. * test(decisions): clear every OpenAI base setting in the proxy OpenAI deployment test * fix(decisions): send OpenAI instructions for System One questions without them and reject single-option questions * fix(decisions): return guardrail_information on OpenAI-format decisions when requested * refactor(decisions): validate proxy request data before reading guardrail settings |
||
|
devin-ai-integration[bot]
|
79f62db620
|
feat(decisions): add the OpenAI Decisions spec types and the System One translation (#45129)
* feat(decisions): add the OpenAI Decisions spec types and the System One translation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): share one DecisionsModel config and require model and usage on responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): match the SDK on strict choice values and drop the invented min_length bounds Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): compose DecisionsRequest from model and body and read the translator top down Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(decisions): use the plural Decisions prefix only for the request and response envelopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): import assert_never from typing_extensions for Python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
a7cf9c6c86
|
fix(openai-compat): send provider attribution headers on the default SDK path (+ Perplexity) (#44291)
* fix(openai-compat): send provider attribution headers on the default SDK path Provider-specific headers set in validate_environment are never sent for OpenAI-compatible providers on the default OpenAI SDK path, which doesn't call it. Novita's X-Novita-Source has been silently missing as a result. Add BaseConfig.get_attribution_headers() and merge it into the outbound headers in _complete_custom_openai, which feeds both the SDK and the experimental http-handler paths. Caller headers win, case-insensitively. * feat(perplexity): send X-Pplx-Integration attribution header Ports the change from #38565 onto the attribution-header hook so it is sent on the default SDK path too. Co-authored-by: Saleh Alghusson <1331721+qirh@users.noreply.github.com> * test: capture attribution headers in-process instead of over a socket Address review: unit tests now drive litellm.completion into an httpx transport rather than a local HTTP server; type the header helper as dict[str, str]; bind the merged headers to a Final instead of rebinding headers in _complete_custom_openai. * style: drop explanatory comments flagged by review --------- Co-authored-by: Saleh Alghusson <1331721+qirh@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Mateo Wang
|
d4a791d650
|
refactor(types): replace Any with proven types in 157 files (#44798)
* refactor(types): replace Any with proven types in 299 files Clears 871 basedpyright Any errors (reportAny 6,703 to 6,240, reportExplicitAny 1,651 to 1,243) without adding a cast, an ignore or a suppression, and without touching any budget file Most edits are annotation-only: a parameter, return or local goes from Any to object, Mapping[str, object] or the concrete type the value always held. Fourteen files validate untyped JSON once where it enters, through a module-level pydantic TypeAdapter or model_validate, and then use real types No HTTP status, error type or response shape changes. Mistral speech and fal.ai Bria image generation now report a pydantic ValidationError instead of an AttributeError when the provider answers 2xx with a body that is not a JSON object * refactor(types): make the config locals fix effective and trim no-op edits The Mapping[str, object] annotation on `locals().copy()` removed no error, because the checker narrows the variable back to the dict[str, Any] the call returns. 21 provider config constructors now build the same copy with dict(locals()), which the checker infers as object values under that annotation, so each file loses one reportAny. The same annotation is reverted in 31 other config files where it stayed a no-op, together with the tests that were added only to cover those lines, and the one MCP server manager line that no CI coverage shard executes is reverted too. The pull request drops from 345 to 297 changed files. * refactor(types): accept only int in the proxy state setter get_proxy_state_variable is annotated to return int, but set_proxy_state_variable still took Any, so the checker could not hold callers to the type the getter promises. The setter now takes int, which is what its only caller already passes. * refactor(types): index the proxy state key so the getter returns int * refactor(types): keep public annotations and provider error text unchanged Restore every public return, public method parameter, public attribute and exported alias to its annotation on main so code that type-checks against the package keeps type-checking, and take the Mistral speech and fal.ai Bria changes back out so no provider error message differs from main * refactor(types): leave the Vertex RAG chunking read as it is on main Take the chunking format validation back out of the Vertex RAG ingestion path. It needs the vertexai SDK, a storage bucket and a RAG corpus to execute, so nothing here could run it end to end, and it cleared only two errors * test(integration): pin the validated provider boundaries on a live proxy * test(integration): give the held burst a client that outlasts the gate The fault cell holds a burst at the upstream for up to 60 seconds while it kills a worker, but sent the burst through the shared 15 second client, so a slow box could time the survivors out before the gate opened. The burst now goes through its own client whose timeout is twice the gate, and the gate length is one named constant. * refactor(types): keep the license reply handling and experimental MCP signatures as they were The license check validated the whole reply as a mapping, which changed the error text logged for a reply that is not an object. It now validates only the verify value, so every reply is handled and logged exactly as before while the value is still typed. Three files under the experimental MCP server changed annotations on public functions and methods (three returns and three parameters). They go back to their previous content so no public signature in the diff is narrowed. * test(integration): answer the proxy's model-list call in the OpenAI stand-ins Every 300 seconds each proxy worker asks an OpenAI deployment for GET /v1/models. Four new cells own an OpenAI stand-in that accepted only the call under test, so a refresh landing inside a cell failed it. The stand-ins now answer that call through the suite's own helper and the cells count only the provider calls they drive. |
||
|
devin-ai-integration[bot]
|
0ed1c08f02
|
feat(anthropic): workload identity federation and pluggable identity sources (#44448)
* feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * fix(proxy): let a deployment write name a federated credential reject_federated_credential_reference runs from is_request_body_safe, which pre_db_read_auth_checks calls on every route, so it also fired on POST /model/new, /model/update, /model/{id}/update and /health/test_connection. A proxy admin could no longer attach a federated credential to a deployment over the API or the Admin UI, leaving a static config.yaml entry as the only way to configure the feature the rejection told the caller to go configure, and _reject_non_admin_wif_write never got to make the call it exists to make. is_request_body_safe now takes the route and skips only the credential-reference check on the routes that reach can_user_make_model_call. Federation fields typed inline into a body stay refused everywhere, and a call naming a federated credential still cannot pick the identity it mints as. * refactor(proxy): derive health display policy from the federation key sets The health check module hand-copied the five workload identity fields whose value is a credential, so a shared proxy surface named provider-specific parameters and a newly added secret-bearing field would have gone on being displayed until someone remembered both places WIF_SECRET_BEARING_KEYS now sits beside the key sets it splits out of, types/utils derives secret_bearing_wif_litellm_params from it, and the health layer splats that tuple the same way it already splats the admin-only one * fix(anthropic_wif): treat blank identity-source fields as unset * test(proxy): classify the federation params in the credential slot registry main's registry test (#43298) now fails the build for any credential-named deployment param without a classification. The five federation fields that carry a token, a token file path, or a signing or client secret reference are Unplanted, matching WIF_SECRET_BEARING_KEYS; the four remaining Keycloak settings name a URL, a client id, an auth method, or a scope and are NotSecret * fix(anthropic_wif): declare federation params as owned connection leaves and chart their metrics Register the 18 Anthropic and 3 OpenAI federation params as frozen ConnectionSettings leaves so the owned-kwarg registry, the kwargs funnel and the request-body ban list read one declaration. Pass the deployment api_base through to the count-tokens handler instead of a pre-suffixed URL, which doubled the /count_tokens path on main's prompt-cache predictor. Add the five litellm_anthropic_wif_* families to the all-metrics Grafana dashboard. * fix(credentials): gate PATCH on WIF fields resolved from model_id The credential PATCH handler checked server-owned workload identity federation fields only on the values the caller sent, while a body that named a deployment through model_id had its credential values resolved after that check. A non-admin could therefore copy a federated deployment's WIF fields onto an ordinary credential. Resolve the incoming values first and run the non-admin gate on them, matching the POST path * fix(anthropic): count tokens with ANTHROPIC_AUTH_TOKEN through the shared auth header Count-tokens walked its own credential ladder: a static key, else skip minting when ANTHROPIC_AUTH_TOKEN is set, else mint a federated token. With only the auth token set it forwarded nothing and the proxy silently fell back to its local tokenizer while chat on the same deployment authenticated with that token. The handler now takes the auth header that AnthropicModelInfo.aget_auth_header resolves, the same ladder chat, files, batches and skills use, and merges the oauth beta a minted or consumer token carries with the token-counting beta --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> Co-authored-by: mateo-berri <happymvw@gmail.com> |
||
|
ishaan-berri
|
0f6a06a6b9
|
feat: add litellm.agent() to run claude code, codex, opencode and deep agents through the ai gateway (#43885)
* feat(harness): add litellm/__init__.py * feat(harness): add litellm/constants.py * feat(harness): add litellm/harness/__init__.py * feat(harness): add litellm/harness/adapters/__init__.py * feat(harness): add litellm/harness/adapters/base.py * feat(harness): add litellm/harness/adapters/claude_code.py * feat(harness): add litellm/harness/adapters/codex.py * feat(harness): add litellm/harness/adapters/opencode.py * feat(harness): add litellm/harness/endpoint.py * feat(harness): add litellm/harness/errors.py * feat(harness): add litellm/harness/options.py * feat(harness): add litellm/harness/runtime.py * feat(harness): add litellm/harness/sandbox/__init__.py * feat(harness): add litellm/harness/sandbox/base.py * feat(harness): add litellm/harness/sandbox/docker.py * feat(harness): add litellm/harness/sandbox/local.py * feat(harness): add litellm/harness/sandbox/snapshot.py * feat(harness): add litellm/harness/sync.py * feat(harness): add litellm/harness/types.py * feat(harness): add litellm/sandbox/__init__.py * feat(harness): add README.md * feat(harness): add tests/harness_e2e/__init__.py * feat(harness): add tests/harness_e2e/conftest.py * feat(harness): add tests/harness_e2e/test_harness_e2e.py * feat(harness): add tests/test_litellm/harness/__init__.py * feat(harness): add tests/test_litellm/harness/adapters/__init__.py * feat(harness): add tests/test_litellm/harness/adapters/fixtures/claude_code/api_error.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/claude_code/max_turns.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/claude_code/resume_turn.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/claude_code/structured_output.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/claude_code/success_tools.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/codex/reasoning.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/codex/structured_output.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/codex/turn_failed.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/codex/turn1_bash.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/codex/turn2_resume_apply_patch.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/opencode/api_error.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/opencode/endpoint_requests.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/opencode/readonly_denied_bash.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/opencode/turn1_write_read.jsonl * feat(harness): add tests/test_litellm/harness/adapters/fixtures/opencode/turn2_session_skill.jsonl * feat(harness): add tests/test_litellm/harness/adapters/test_claude_code.py * feat(harness): add tests/test_litellm/harness/adapters/test_codex.py * feat(harness): add tests/test_litellm/harness/adapters/test_opencode.py * feat(harness): add tests/test_litellm/harness/core_fakes.py * feat(harness): add tests/test_litellm/harness/sandbox/__init__.py * feat(harness): add tests/test_litellm/harness/sandbox/test_docker.py * feat(harness): add tests/test_litellm/harness/sandbox/test_local.py * feat(harness): add tests/test_litellm/harness/sandbox/test_snapshot.py * feat(harness): add tests/test_litellm/harness/test_endpoint.py * feat(harness): add tests/test_litellm/harness/test_init.py * feat(harness): add tests/test_litellm/harness/test_runtime.py * feat(harness): add tests/test_litellm/harness/test_sync.py * feat(harness): add tests/test_litellm/harness/test_types.py * test(harness): use word recall in stream e2e test * refactor(harness): update litellm/__init__.py * refactor(harness): update litellm/constants.py * refactor(harness): update litellm/harness/__init__.py * refactor(harness): remove litellm/harness/adapters/__init__.py * refactor(harness): update litellm/harness/context.py * refactor(harness): update litellm/harness/endpoint.py * refactor(harness): update litellm/harness/handlers/__init__.py * refactor(harness): update litellm/harness/handlers/base.py * refactor(harness): update litellm/harness/handlers/cli_handler.py * refactor(harness): update litellm/harness/handlers/deepagents_handler.py * refactor(harness): update litellm/harness/runtime.py * refactor(harness): update litellm/harness/sandbox/docker.py * refactor(harness): update litellm/harness/sandbox/local.py * refactor(harness): update litellm/harness/sync.py * refactor(harness): update litellm/harness/types.py * refactor(harness): update litellm/llms/base_llm/harness/__init__.py * refactor(harness): update litellm/llms/base_llm/harness/transformation.py * refactor(harness): update litellm/llms/base_llm/harness/utils.py * refactor(harness): update litellm/llms/claude_code/__init__.py * refactor(harness): update litellm/llms/claude_code/harness/__init__.py * refactor(harness): update litellm/llms/claude_code/harness/transformation.py * refactor(harness): update litellm/llms/codex/__init__.py * refactor(harness): update litellm/llms/codex/harness/__init__.py * refactor(harness): update litellm/llms/codex/harness/transformation.py * refactor(harness): update litellm/llms/deepagents/__init__.py * refactor(harness): update litellm/llms/deepagents/harness/__init__.py * refactor(harness): update litellm/llms/deepagents/harness/sandbox_backend.py * refactor(harness): update litellm/llms/deepagents/harness/transformation.py * refactor(harness): update litellm/llms/opencode/__init__.py * refactor(harness): update litellm/llms/opencode/harness/__init__.py * refactor(harness): update litellm/llms/opencode/harness/transformation.py * refactor(harness): update litellm/utils.py * refactor(harness): update README.md * refactor(harness): update tests/harness_e2e/conftest.py * refactor(harness): update tests/harness_e2e/test_harness_e2e.py * refactor(harness): remove tests/test_litellm/harness/__init__.py * refactor(harness): remove tests/test_litellm/harness/adapters/__init__.py * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/claude_code/api_error.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/claude_code/max_turns.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/claude_code/resume_turn.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/claude_code/structured_output.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/claude_code/success_tools.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/codex/reasoning.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/codex/structured_output.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/codex/turn_failed.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/codex/turn1_bash.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/codex/turn2_resume_apply_patch.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/opencode/api_error.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/opencode/endpoint_requests.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/opencode/readonly_denied_bash.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/opencode/turn1_write_read.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/fixtures/opencode/turn2_session_skill.jsonl * refactor(harness): remove tests/test_litellm/harness/adapters/test_claude_code.py * refactor(harness): remove tests/test_litellm/harness/adapters/test_codex.py * refactor(harness): remove tests/test_litellm/harness/adapters/test_opencode.py * refactor(harness): remove tests/test_litellm/harness/core_fakes.py * refactor(harness): remove tests/test_litellm/harness/sandbox/__init__.py * refactor(harness): remove tests/test_litellm/harness/sandbox/test_docker.py * refactor(harness): remove tests/test_litellm/harness/sandbox/test_local.py * refactor(harness): remove tests/test_litellm/harness/sandbox/test_snapshot.py * refactor(harness): remove tests/test_litellm/harness/test_endpoint.py * refactor(harness): remove tests/test_litellm/harness/test_init.py * refactor(harness): remove tests/test_litellm/harness/test_runtime.py * refactor(harness): remove tests/test_litellm/harness/test_sync.py * refactor(harness): remove tests/test_litellm/harness/test_types.py * refactor(harness): update tests/unit/harness/__init__.py * refactor(harness): update tests/unit/harness/core_fakes.py * refactor(harness): update tests/unit/harness/handlers/__init__.py * refactor(harness): update tests/unit/harness/handlers/test_deepagents_handler.py * refactor(harness): update tests/unit/harness/sandbox/__init__.py * refactor(harness): update tests/unit/harness/sandbox/test_docker.py * refactor(harness): update tests/unit/harness/sandbox/test_local.py * refactor(harness): update tests/unit/harness/sandbox/test_snapshot.py * refactor(harness): update tests/unit/harness/test_endpoint.py * refactor(harness): update tests/unit/harness/test_init.py * refactor(harness): update tests/unit/harness/test_runtime.py * refactor(harness): update tests/unit/harness/test_sync.py * refactor(harness): update tests/unit/harness/test_types.py * refactor(harness): update tests/unit/llms/claude_code/__init__.py * refactor(harness): update tests/unit/llms/claude_code/harness/__init__.py * refactor(harness): update tests/unit/llms/claude_code/harness/fixtures/api_error.jsonl * refactor(harness): update tests/unit/llms/claude_code/harness/fixtures/max_turns.jsonl * refactor(harness): update tests/unit/llms/claude_code/harness/fixtures/resume_turn.jsonl * refactor(harness): update tests/unit/llms/claude_code/harness/fixtures/structured_output.jsonl * refactor(harness): update tests/unit/llms/claude_code/harness/fixtures/success_tools.jsonl * refactor(harness): update tests/unit/llms/claude_code/harness/test_transformation.py * refactor(harness): update tests/unit/llms/codex/__init__.py * refactor(harness): update tests/unit/llms/codex/harness/__init__.py * refactor(harness): update tests/unit/llms/codex/harness/fixtures/reasoning.jsonl * refactor(harness): update tests/unit/llms/codex/harness/fixtures/structured_output.jsonl * refactor(harness): update tests/unit/llms/codex/harness/fixtures/turn1_bash.jsonl * refactor(harness): update tests/unit/llms/codex/harness/fixtures/turn2_resume_apply_patch.jsonl * refactor(harness): update tests/unit/llms/codex/harness/fixtures/turn_failed.jsonl * refactor(harness): update tests/unit/llms/codex/harness/test_transformation.py * refactor(harness): update tests/unit/llms/deepagents/__init__.py * refactor(harness): update tests/unit/llms/deepagents/harness/__init__.py * refactor(harness): update tests/unit/llms/deepagents/harness/test_transformation.py * refactor(harness): update tests/unit/llms/opencode/__init__.py * refactor(harness): update tests/unit/llms/opencode/harness/__init__.py * refactor(harness): update tests/unit/llms/opencode/harness/fixtures/api_error.jsonl * refactor(harness): update tests/unit/llms/opencode/harness/fixtures/endpoint_requests.jsonl * refactor(harness): update tests/unit/llms/opencode/harness/fixtures/readonly_denied_bash.jsonl * refactor(harness): update tests/unit/llms/opencode/harness/fixtures/turn1_write_read.jsonl * refactor(harness): update tests/unit/llms/opencode/harness/fixtures/turn2_session_skill.jsonl * refactor(harness): update tests/unit/llms/opencode/harness/test_transformation.py * ci: allowlist tests/harness_e2e, which needs live runtimes and a gateway * fix(harness): bound the turn event queue * fix(harness): bound the turn event queue with backpressure * style: sort imports in utils * ci: exclude agent-harness config folders from provider docs check * refactor(harness): update tests/harness_e2e/conftest.py * refactor(harness): update tests/harness_e2e/test_harness_e2e.py * refactor(harness): update tests/unit/harness/core_fakes.py * refactor(harness): update tests/unit/harness/handlers/test_deepagents_handler.py * refactor(harness): update tests/unit/harness/test_runtime.py * refactor(harness): update tests/unit/harness/test_sync.py * refactor(harness): update tests/unit/llms/deepagents/harness/test_transformation.py * refactor(harness): update tests/unit/llms/opencode/harness/test_transformation.py * refactor(harness): update litellm/harness/endpoint.py * refactor(harness): update litellm/harness/handlers/deepagents_handler.py * refactor(harness): update litellm/harness/options.py * refactor(harness): update litellm/harness/runtime.py * refactor(harness): update litellm/harness/sandbox/snapshot.py * refactor(harness): update litellm/llms/base_llm/harness/transformation.py * refactor(harness): update litellm/llms/base_llm/harness/utils.py * refactor(harness): update litellm/llms/claude_code/harness/transformation.py * refactor(harness): update litellm/llms/codex/harness/transformation.py * refactor(harness): update litellm/llms/deepagents/harness/sandbox_backend.py * refactor(harness): update litellm/llms/deepagents/harness/transformation.py * refactor(harness): update litellm/llms/opencode/harness/transformation.py * refactor(harness): update litellm/sandbox/__init__.py * refactor(harness): update tests/unit/llms/claude_code/harness/test_transformation.py * refactor(harness): update tests/unit/llms/deepagents/harness/test_sandbox_backend_symlinks.py * refactor(harness): update tests/unit/llms/opencode/harness/test_transformation.py * refactor(harness): update litellm/llms/base_llm/harness/utils.py * refactor(harness): update litellm/llms/codex/harness/transformation.py * refactor(harness): update tests/code_coverage_tests/recursive_detector.py * refactor(harness): update tests/unit/llms/base_llm/harness/__init__.py * refactor(harness): update tests/unit/llms/claude_code/harness/fixtures/__init__.py * refactor(harness): update tests/unit/llms/codex/harness/fixtures/__init__.py * refactor(harness): update tests/unit/llms/opencode/harness/fixtures/__init__.py * refactor(harness): update litellm/harness/__init__.py * refactor(harness): update litellm/harness/context.py * refactor(harness): update litellm/harness/endpoint.py * refactor(harness): update litellm/harness/handlers/__init__.py * refactor(harness): update litellm/harness/handlers/base.py * refactor(harness): update litellm/harness/handlers/cli_handler.py * refactor(harness): update litellm/harness/handlers/deepagents_handler.py * refactor(harness): update litellm/harness/runtime.py * refactor(harness): update litellm/harness/sandbox/__init__.py * refactor(harness): update litellm/harness/sandbox/base.py * refactor(harness): update litellm/harness/sandbox/docker.py * refactor(harness): update litellm/harness/sandbox/local.py * refactor(harness): update litellm/harness/sandbox/snapshot.py * refactor(harness): update litellm/harness/sync.py * refactor(harness): update litellm/harness/types.py * refactor(harness): update litellm/llms/base_llm/harness/transformation.py * refactor(harness): update litellm/llms/base_llm/harness/utils.py * refactor(harness): update litellm/llms/claude_code/harness/transformation.py * refactor(harness): update litellm/llms/codex/harness/transformation.py * refactor(harness): update litellm/llms/deepagents/harness/sandbox_backend.py * refactor(harness): update litellm/llms/deepagents/harness/transformation.py * refactor(harness): update litellm/llms/opencode/harness/transformation.py * refactor(harness): update litellm/types/llms/custom_http.py * refactor(harness): update tests/unit/harness/test_endpoint.py * refactor(harness): update litellm/harness/runtime.py * refactor(harness): update litellm/llms/deepagents/harness/sandbox_backend.py * refactor(harness): update tests/unit/harness/test_init.py * refactor(harness): update tests/unit/harness/test_runtime.py * refactor(harness): update tests/unit/llms/deepagents/harness/test_sandbox_backend_symlinks.py |
||
|
devin-ai-integration[bot]
|
3a6744cd02
|
feat(sail): add Sail as a provider with service_tier mapped to its completion window (#42840)
Register Sail (providers.json, LlmProviders.SAIL, OpenAI-compatible lists, ProviderConfigManager) for chat, Responses and /v1/messages, and add its 12 models to both cost maps with asap, balanced and flex price columns. Sail picks speed and price with metadata.completion_window and rejects service_tier, so the Sail chat and Responses configs translate the tier: default and priority to asap, flex to flex, balanced to balanced, auto to no window. Billing prices the window that was sent. A tier Sail has no window for, or a window or tier set where billing cannot see it (request metadata, extra_body), is a 400 unless drop_params is set. Add balanced to ServiceTier and its _balanced price columns to the model info types, the Rust catalog and the dashboard schema. A transform_extra_body hook on the chat and Responses base configs, which returns extra_body unchanged by default, lets Sail keep the window when a caller also sends extra_body.metadata. Sail is listed in the Add Model form and model picker. Co-authored-by: shrey kharbanda <shrey@berri.ai> |
||
|
yuneng-jiang
|
5e6dc89ba1
|
test: move tests/test_litellm/llms into tests/unit/llms (#43191)
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: keep the job's UNIT_FLAG out of the shard-script tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
e4a58ef91a |
test(unit): make every tests/unit directory a package so pytest collection is unique
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng
|
02ccdbae90 | test(llms): migrate bedrock, baseten and base_llm batch tests to tests/unit |