9 commits

Author	SHA1	Message	Date
Vineeth Sai Varikuntla	a7b66dd8b9	fix(openai): stop stripping cache_control from the caller's own messages and tools (#35602) ... * fix(openai): stop stripping cache_control from the caller's messages Copy before removing the field, so the caller's own message objects are left alone. * fix(fireworks_ai): strip cache_control on a copy so a fallback keeps the caller's hints * fix: keep the caller's request intact on more providers and trim extra Claude cache hints under drop_params * fix: drop only the oldest extra cache hints inside one message when trimming to the Claude limit * fix: keep replayed session turns yielding their hints as whole turns while a request's own message keeps its newest hints * fix(cache_control): count only hints that reach Claude and cap hints on OpenAI-compatible Claude routes under drop_params * fix(cache_control): keep the capped Claude messages a list in OpenAI-compatible request bodies * fix(caching): cap Claude hints for string drop_params, a Claude base_model, and Converse tool messages The 4-hint cap now reads drop_params through drops_unsupported_params on chat, Converse, and Anthropic routes, names Claude by base_model on OpenAI-compatible routes, and counts a Converse tool message's hints as the one cache point it sends. * fix(caching): trim Claude cache hints in one wire-order pass and reserve a top-level cache_control on OpenAI-compatible routes Dropping the oldest hint one at a time re-walked the whole request per hint, which took seconds on long tool histories. The cap now drops each message's oldest hints in the order the provider counts them, so a string assistant turn loses its text hint before its tool call hints. OpenAI-compatible Claude routes also keep room for a request-level or extra_body cache_control under drop_params. * test: drop narrating comments and stacked generator clauses from cache_control tests * fix(vertex_ai): put every system message in the Gemini context cache so generateContent never sends it beside cachedContent --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 00:29:10 -07:00
tin-berri	516c796e86	fix(rerank): preserve token usage in spend logs (#49308)	2026-10-30 13:46:27 -07:00
mika-lt	71babd931b	fix(llms): move later developer messages for models declared system-first (#48897) ... * fix(transformation): merge consecutive system and developer messages for non-OpenAI providers and Responses API * fix(base_utils): keep later cache_control and billing metadata boundaries when merging system messages * perf(base_utils): fold consecutive system messages in a single linear pass The tuple accumulator passed to reduce copied the whole prefix on every step, making map_developer_role_to_system_role O(n^2) in the number of messages. Message count is not bounded by default, so a caller could tie up a proxy worker with thousands of small messages before any provider call. Use a list and merge into its last element in place instead. * fix(base_utils): hoist later developer messages into the leading system message for non-OpenAI providers A developer message that arrives after the first user turn (Codex CLI's per-turn permissions item, a re-injected instruction after compaction, a mid-conversation policy update) was translated to a system message in place, and system-first chat templates such as Fireworks' qwen3.8 reject any system message that is not at the beginning with 400. The fold that merges consecutive system messages never saw it because a user turn sat in between. Move every developer message that follows the leading system block to the end of that block before the fold, so it merges into the single leading system message; client-authored mid-conversation system messages stay where the client put them. * fix(openai): hoist later developer messages into one leading system message on OpenAI-compatible backends Developer role translation moves onto the provider config. OpenAI's own endpoint, Azure OpenAI, and the o-series keep their current behavior. Every other config inheriting OpenAIGPTConfig (Fireworks, DeepSeek, Together, vLLM, Groq, openai/ with a custom api_base) hoists developer messages that arrive after the leading system block into that block and folds the block into a single system message, which is what system-first chat templates require. Providers with their own message format are untouched. Messages carrying cache_control or Anthropic billing metadata fold as their own text blocks, so breakpoints and provider-side stripping survive the merge. The Responses bridge goes back to prepending instructions as a separate system message and lets the hoist fold it. * refactor(base_utils): fold each run of system messages in one pass The pairwise fold recopied the accumulated system text or block list on every merge, so a request carrying many consecutive system or developer messages cost quadratic time in preprocessing. Group consecutive system messages once and build the merged message from the whole run: plain strings join with a blank line, and any cached, billing, or block content keeps one text block per message. The name of the last named message in the run survives, as before. * fix(base_utils): keep a closing developer message in place and hoist for OpenAIConfig backends A developer message that closes the conversation stays where the client put it, translated to system in place, so the request never ends on the assistant's turn. OpenAIConfig and its subclasses (Azure AI Foundry, GitHub Copilot, ChatGPT, RAGFlow) now share the OpenAI-compatible hoist, litellm_proxy passes developer messages through to the downstream proxy, and the Anthropic billing header literals use the shared constant * fix(base_utils): keep a closing developer message in place only after an assistant turn A developer message that closes the conversation after a user or tool message hoists like any other, so the request shape the ticket reports keeps working on system-first templates; only a closing run that follows an assistant turn stays in place * refactor(openai): reuse is_openai_backed_api_base for the hosted endpoint check * test(base_utils): cover an empty system message inside a block-content run Codecov flagged the empty-string branch of _text_blocks as the one line the PR left uncovered. The existing empty-string test only hits the plain-text join; this one forces the block path by merging an empty system message with a developer message that carries list content. * fix(openai): hoist developer messages for litellm_proxy and harden the system fold - drop the LiteLLMProxyChatConfig passthrough so litellm_proxy/ hoists later developer messages like every other OpenAI-compatible endpoint, whichever backend or LiteLLM version sits downstream - a system message with null or missing content inside a folded run is dropped instead of raising KeyError('content') - a message-level cache_control on a list-content member lands on its last block so the breakpoint survives the fold * fix(litellm_proxy): keep the AllMessageValues import the transform signatures still name * fix(fireworks_ai): hoist developer items after the first turn on the native Responses path The native Fireworks Responses path landed on staging mapping developer input items to system in place, so a developer item after the first input item still hit the same "System message must be at the beginning" 400 on system-first chat templates. The ordering rules now live in one shared helper over item roles, and the native path reorders through it before mapping, keeping a closing developer item after an assistant turn in place exactly like the chat path does * fix(transformation): carry the developer-message hoist onto main 963333d654 Resolves the main merge for this branch: the tests move to tests/unit, every translate_developer_role_to_system_role override takes and returns a Sequence, Azure OpenAI keeps the in-place mapping, and the system content reads tolerate a missing key * fix(transformation): join a lone plain text block as text when hoisting developer messages The Responses bridge turns every developer item into a one-block list, so the hoisted run merged the instructions string and that block as a block list, and Databricks could no longer collapse it into the single system string it sends today. A lone text block with no extra keys now joins the run as text, while multi-block, cached, and billing-header content keep the block path * fix(transformation): hoist later developer messages only for models that reject mid-conversation system messages Runs of consecutive system messages still fold into one, which keeps the cached prefix stable. A developer message after the first turn moves into the leading system message only when the model declares supports_mid_conversation_system false, or when it is a Fireworks model with no declaration, matching the native Fireworks Responses route. Every other backend keeps it in place as before, so their prompt cache prefix is unchanged. Malformed system content passes through unmerged so the provider returns its own 400. * fix(responses): fold later developer items into instructions on hosted_vllm for system-first models Moves the Responses developer fold shared by Fireworks into one module and applies it to hosted_vllm native Responses when the model declares supports_mid_conversation_system false. Renames the run-merging helper so it no longer shares a name with the Databricks one, and returns tuples from the block helpers. * fix(llms): place developer messages only for models declared system-first Models without a supports_mid_conversation_system declaration now keep the in-place developer-to-system translation they had before, so their cached prompt prefix does not change. Hoisting and folding run only when the deployment's model_info or the cost map declares the flag false, with the deployment value read first. A developer message that closes the conversation after an assistant turn is sent as a user message on those models instead of being hoisted, merged list content keeps a blank-line separator before each later part, and the cost map declares the flag false for fireworks_ai qwen3p8-max * fix(proxy): keep a closing developer message's cache mark when it is sent as a user turn * test: type the developer-hoist tests and fake HTTP with respx * fix: count non-string roles as turns, keep the Fireworks fold for twins, and copy closing developer content A message whose role is a list or dict now counts as a conversation turn instead of raising a 500. Fireworks native Responses skips the developer-item fold only when the deployment's own model_info declares supports_mid_conversation_system true, since the router registers every deployment's flag under its shared backend key. A hoisted developer message keeps only role, content, name, and cache_control, a closing developer message's content is deep-copied so the caller's list is left alone, and a hosted_vllm developer item with no content is forwarded unchanged. The Fireworks in-place test now declares true on the deployment itself, because a cost-map true can no longer be told apart from another deployment's registration. --------- Co-authored-by: sowmy narayanan <sowmysuresh86@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 04:12:13 -07:00
devin-ai-integration[bot]	e0398455d0	test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) ... * test: migrate selected llm translation contracts * test: migrate anthropic completion translation tests * test: migrate bedrock and cohere translation tests * test: tighten bedrock translation migration * test: migrate azure and openai model translation tests * test: migrate openai and small-provider translation tests * test: migrate gemini and vertex translation tests * test: migrate optional params and rerank translation tests * test: migrate huggingface, json and openrouter translation tests * test: fix openrouter integration expectations * test: migrate databricks, xai, cometapi and fireworks translation tests * test: migrate ovhcloud, prompt and small provider translation tests * test: split reasoning effort grid into unit and e2e * test: migrate router and realtime guardrail translation tests * test: migrate google interactions translation tests * test: migrate a2a, evals, skills and hosted vllm translation tests * test: remove remaining llm translation suite * test: tighten migrated llm translation unit tests and land missing rows Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover * test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths * test: close lost-intent gaps found in review of the llm translation migration Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised * test: assert provider acceptance in the reasoning effort e2e The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid * test: cover async and stream-end paths the legacy translation suite exercised Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker * test: type the new llm translation e2e calls and mark moved fixture dirs as packages * test: name the reasoning effort e2e models as module constants * test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name * test: block the prompt the shared guardrail denies and drop the reasoning presence check * test: drop the duplicate litellm import left by the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 07:58:53 +00:00
devin-ai-integration[bot]	a2e334febe	test: migrate local_testing SDK and provider tests out of build_and_test (#45760) ... * test: migrate local_testing SDK and provider tests out of build_and_test * test: cover sagemaker region, hf template and nova paths end to end Region rows now go through litellm.completion on the sagemaker/ handler that the legacy tests exercised, the prompt template test mocks the Hugging Face tokenizer config instead of switching to custom_prompt_dict, streaming tests assert role, finish reason and body, and Nova single turn and streaming run through litellm.completion with a recorded request body * test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot Split the catch-all providers migration file by behavior. The fake endpoint route tests now hit the owned upstream directly, as the legacy harness tests did. LM Studio asserts the outbound key, path and stripped model. The embedding timeout test asserts the proxy maps a deployment timeout to a 408 and forwards the read timeout to the provider client, instead of timing out the test client The Azure streaming tool-call test also asserts the second identical stream is served from the Redis cache with the same reassembled tool call The Docker network test always skipped in CI because the security job has no litellm/litellm image and no egress to pull one, and it tested a published image rather than the checkout. It now boots the checkout proxy with every outbound HTTP connection refused and asserts readiness, the bundled cost map fallback and that the cost map host is the only egress attempted * test: tighten the migrated e2e rows for local_testing SDK and provider tests Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main. Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers so the header actually reaches the provider, and mark the creation-token case 1h then 5m like the legacy test did Reuse the Bedrock file's own registration for the pydantic response test, swap the hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in markers the same way as the other opt-in markers * test: restore lost assertions in migrated test_main tests Restore the role alternation check in test_completion_anthropic_hanging, the embedding and streaming response-header checks, the logit_bias drop check and the gemini system and safety settings checks. Add an azure 429 retry-after test for test_exception_with_headers, trigger the router fallback with mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s * test: restore lost legacy assertions in migrated vertex ai unit tests * test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard * test: strengthen migrated anthropic, azure, bedrock and cohere provider tests Assert full request bodies and usage instead of presence checks, restore the dummy-tool and custom_llm_provider assertions the legacy tests made, add the async and per-model variants the legacy parametrization covered, split streamed tool arguments across deltas, and move the Azure chat streaming tests to the azure.py mirror file * test: tighten migrated llms provider tests and cover legacy intents lost in the move Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt * test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients * test: restore single request assertions in the deep_infra and azure_ai embedding tests * test: drop a no-op streaming cost flag from the openai streaming cost test * test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke * test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard * test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e * test: send the vertex maps e2e request through the openai sdk * test: drop the local_testing jobs that select no tests after the migrations * test: wait for the streamed cache entry and keep custom Mistral rates test-local * test: keep untouched Mistral test lines as they were --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-10 23:14:25 +00:00
devin-ai-integration[bot]	e2971e0af4	refactor(llms): expose public names for private provider helpers (#45037) ... * refactor(litellm): migrate private usage in llms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve Bedrock batch signature marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): migrate llms private usage symbols Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve UUID Watsonx project IDs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): retarget llms mocks to public names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): limit llms changes to renames and forwarders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): keep GCS mock client patching private Vertex auth methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): assert forwarder arguments and type forwarder signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 08:49:25 -07:00
devin-ai-integration[bot]	657bb777fa	fix(hosted_vllm): keep reasoning_content on replayed assistant messages (#43599) ... * fix(hosted_vllm): keep reasoning_content on assistant messages in _transform_messages vLLM accepts reasoning_content (200 on the wire) and qwen/deepseek/glm chat templates consume it, so popping it made reasoning models lose earlier reasoning across tool loops. thinking_blocks is still removed for vLLM compatibility. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(hosted_vllm): forward replayed reasoning_content only when it is a string * test(integration): cover hosted_vllm reasoning_content replay across endpoints * test(integration): require the surviving worker to serve its held requests in the sigkill chaos cell --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai>	2026-09-30 13:12:28 -07:00
yuneng	e4a58ef91a	test(unit): make every tests/unit directory a package so pytest collection is unique ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-20 11:50:59 +00:00
yuneng	4defed7f2e	test: migrate wave 1 phase 8 legacy llm tests to tests/unit ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-20 08:03:54 +00:00