55486 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Priyansh Nandwana	67ae3fb527	fix(websearch)!: keep the provider prefix on the agentic follow-up model (#38885) ... * fix(websearch): keep the provider prefix on the agentic follow-up model The agentic follow-up rebuilt its model string with "a slash means it is already qualified". Handlers are handed the provider-stripped model, and for a provider that routes through a sub-path the remainder still holds a slash, so bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5 and stayed that way. The follow-up acompletion could not resolve a provider from it and raised "LLM Provider NOT provided". Prefix unless the model is already qualified with that same provider, in one shared helper used by both the handler and the websearch patch builder, which carried copies of the same heuristic. This also fixes openrouter, whose stripped model keeps a slash for the same reason, and the old startswith test that read openai as a prefix of openai_like. Fixes #38829 * test(websearch): cover the agentic follow-up call site The helper had unit tests but the call site did not, so the two changed lines in _execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with mock_response instead of patching litellm, which keeps the test on the behaviour: against the old code the sub-path case raises "LLM Provider NOT provided", and the ordinary-model case passes either way as a guard. * refactor(websearch): trim the comments on the provider-qualify helper * refactor(websearch): cut the helper and test docstrings to one line * fix(websearch): leave a hook-qualified follow-up model on its own provider A hook that patches in its own model owns that choice, so re-qualifying it sent a cross-provider follow-up to the original request's provider instead. Only a bare patched model and the request's provider-stripped model take the prefix now, and the sibling agentic-loop path resolves the model the same way. * fix(websearch): fall back to the request model when a patch carries no model An empty patched model dispatched a provider-only name instead of falling back, and a provider-stripped sub-path copied into the patch lost its prefix. A patched model now keeps its own provider only when its leading segment names one. * fix(websearch): keep the provider prefix on the responses follow-up model too The responses patch builder still used the old "a slash means already qualified" test, so a provider-stripped sub-path model reached the follow-up unqualified on that surface. It now resolves the model the same way the chat completion builder does. * fix(websearch): read custom_llm_provider without a member call on untyped kwargs * fix(websearch): type the chat patch builder kwargs instead of working around them * test(websearch): run the real search path and flatten the qualify helper tests * fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model * fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider * fix(websearch): send the agentic follow-up to the model the caller requested completion() and responses() record the caller's model before routing strips it, and the chat and Responses follow-up builders prefer that model. This keeps bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/* deployments on their own route for the follow-up. * test(websearch): cover the recorded caller model in Headroom's retrieval follow-up * test(websearch): cover the recorded caller model in Compresr's retrieval follow-up Also merges the repeated provider-prefix check in core_helpers into one helper and drops a test docstring * test(websearch): drive the real search path through a mocked Perplexity HTTP route * refactor(websearch): read the provider from kwargs inside _caller_model * test(http-handler): type the agentic follow-up provider tests * refactor(websearch): record agentic loop params through a Logging method Moving the recording out of completion() keeps it under basedpyright's code-flow complexity limit after the merge with main. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 04:47:18 -07:00
Mateo Wang	0cf2bb7296	fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params (#47761) ... * fix(openai): warn when the GPT-5 reasoning effort gate drops a level under drop_params A GPT-5 request whose reasoning_effort the model map does not enable for that model lost the effort with no log line under drop_params. An Azure deployment name missing from the cost map lost reasoning_effort none this way and ran at the model's default effort. The gate now logs a warning naming the model, the level, the map entry it checked, and the model_info.base_model and model_info.reasoning_effort_levels fixes. * fix(openai): warn when the Responses route or a deployment's declared levels drop a reasoning effort The /v1/responses gate dropped a row-disabled GPT-5 effort, and reasoning for an OpenAI model without supports_reasoning, with no log line, and the router logged a declared-level drop only at INFO. Both Responses drops now log a WARNING and the router logs its drop at WARNING. The warning now leads with model_info.reasoning_effort_levels, since base_model only helps a deployment named differently from its underlying model. * fix: warn only when a reasoning level is actually dropped, and on generic reasoning_effort drops * fix: warn when thinking or a Bedrock Converse reasoning level is dropped, and name the router drop a reasoning level * fix: skip the drop warning when a non-reasoning model is asked for no reasoning, and honor declared levels on Bedrock Responses * fix: stay silent when reasoning is turned off on /v1/responses and warn on the Bedrock Responses fallback drop * fix(reasoning): warn when any reasoning-off request is dropped and give each route only the advice it honors A request to turn reasoning off on a model name the catalog lacks now warns on every route, since it may be a reasoning model. Chat suggests model_info.reasoning_effort_levels and, for a name with no catalog entry, model_info.base_model. OpenAI Responses suggests declared levels only where the config honors them, and Bedrock suggests neither. The Bedrock and Mantle Responses forwarding of declared levels is reverted until it can be proven live. * fix(reasoning): suggest reasoning_effort_levels and base_model only where the route reads them The chat gate suggested model_info.reasoning_effort_levels on routes that never pass declared levels (github_copilot, chatgpt and other OpenAIConfig delegates) and model_info.base_model on OpenAI and Azure AI chat, whose gate looks up the deployment name rather than base_model. /v1/responses/compact suggested levels it never reads either. Declared levels are now None when a caller passes none, so the advice appears only when a caller fed them, and the base_model advice is limited to the Azure chat gate, the one that looks up base_model. The openai row of the chat warning test now expects no base_model advice, the Responses warning test goes through get_optional_params_responses_api the way /v1/responses and compact call it, and the Azure direct-config test passes declared levels the way get_optional_params does. Long test lines are wrapped. * fix: read only the built-in catalog when choosing the reasoning drop warning	2026-10-29 04:46:29 -07:00
mika-lt	71babd931b	fix(llms): move later developer messages for models declared system-first (#48897) ... * fix(transformation): merge consecutive system and developer messages for non-OpenAI providers and Responses API * fix(base_utils): keep later cache_control and billing metadata boundaries when merging system messages * perf(base_utils): fold consecutive system messages in a single linear pass The tuple accumulator passed to reduce copied the whole prefix on every step, making map_developer_role_to_system_role O(n^2) in the number of messages. Message count is not bounded by default, so a caller could tie up a proxy worker with thousands of small messages before any provider call. Use a list and merge into its last element in place instead. * fix(base_utils): hoist later developer messages into the leading system message for non-OpenAI providers A developer message that arrives after the first user turn (Codex CLI's per-turn permissions item, a re-injected instruction after compaction, a mid-conversation policy update) was translated to a system message in place, and system-first chat templates such as Fireworks' qwen3.8 reject any system message that is not at the beginning with 400. The fold that merges consecutive system messages never saw it because a user turn sat in between. Move every developer message that follows the leading system block to the end of that block before the fold, so it merges into the single leading system message; client-authored mid-conversation system messages stay where the client put them. * fix(openai): hoist later developer messages into one leading system message on OpenAI-compatible backends Developer role translation moves onto the provider config. OpenAI's own endpoint, Azure OpenAI, and the o-series keep their current behavior. Every other config inheriting OpenAIGPTConfig (Fireworks, DeepSeek, Together, vLLM, Groq, openai/ with a custom api_base) hoists developer messages that arrive after the leading system block into that block and folds the block into a single system message, which is what system-first chat templates require. Providers with their own message format are untouched. Messages carrying cache_control or Anthropic billing metadata fold as their own text blocks, so breakpoints and provider-side stripping survive the merge. The Responses bridge goes back to prepending instructions as a separate system message and lets the hoist fold it. * refactor(base_utils): fold each run of system messages in one pass The pairwise fold recopied the accumulated system text or block list on every merge, so a request carrying many consecutive system or developer messages cost quadratic time in preprocessing. Group consecutive system messages once and build the merged message from the whole run: plain strings join with a blank line, and any cached, billing, or block content keeps one text block per message. The name of the last named message in the run survives, as before. * fix(base_utils): keep a closing developer message in place and hoist for OpenAIConfig backends A developer message that closes the conversation stays where the client put it, translated to system in place, so the request never ends on the assistant's turn. OpenAIConfig and its subclasses (Azure AI Foundry, GitHub Copilot, ChatGPT, RAGFlow) now share the OpenAI-compatible hoist, litellm_proxy passes developer messages through to the downstream proxy, and the Anthropic billing header literals use the shared constant * fix(base_utils): keep a closing developer message in place only after an assistant turn A developer message that closes the conversation after a user or tool message hoists like any other, so the request shape the ticket reports keeps working on system-first templates; only a closing run that follows an assistant turn stays in place * refactor(openai): reuse is_openai_backed_api_base for the hosted endpoint check * test(base_utils): cover an empty system message inside a block-content run Codecov flagged the empty-string branch of _text_blocks as the one line the PR left uncovered. The existing empty-string test only hits the plain-text join; this one forces the block path by merging an empty system message with a developer message that carries list content. * fix(openai): hoist developer messages for litellm_proxy and harden the system fold - drop the LiteLLMProxyChatConfig passthrough so litellm_proxy/ hoists later developer messages like every other OpenAI-compatible endpoint, whichever backend or LiteLLM version sits downstream - a system message with null or missing content inside a folded run is dropped instead of raising KeyError('content') - a message-level cache_control on a list-content member lands on its last block so the breakpoint survives the fold * fix(litellm_proxy): keep the AllMessageValues import the transform signatures still name * fix(fireworks_ai): hoist developer items after the first turn on the native Responses path The native Fireworks Responses path landed on staging mapping developer input items to system in place, so a developer item after the first input item still hit the same "System message must be at the beginning" 400 on system-first chat templates. The ordering rules now live in one shared helper over item roles, and the native path reorders through it before mapping, keeping a closing developer item after an assistant turn in place exactly like the chat path does * fix(transformation): carry the developer-message hoist onto main 963333d654 Resolves the main merge for this branch: the tests move to tests/unit, every translate_developer_role_to_system_role override takes and returns a Sequence, Azure OpenAI keeps the in-place mapping, and the system content reads tolerate a missing key * fix(transformation): join a lone plain text block as text when hoisting developer messages The Responses bridge turns every developer item into a one-block list, so the hoisted run merged the instructions string and that block as a block list, and Databricks could no longer collapse it into the single system string it sends today. A lone text block with no extra keys now joins the run as text, while multi-block, cached, and billing-header content keep the block path * fix(transformation): hoist later developer messages only for models that reject mid-conversation system messages Runs of consecutive system messages still fold into one, which keeps the cached prefix stable. A developer message after the first turn moves into the leading system message only when the model declares supports_mid_conversation_system false, or when it is a Fireworks model with no declaration, matching the native Fireworks Responses route. Every other backend keeps it in place as before, so their prompt cache prefix is unchanged. Malformed system content passes through unmerged so the provider returns its own 400. * fix(responses): fold later developer items into instructions on hosted_vllm for system-first models Moves the Responses developer fold shared by Fireworks into one module and applies it to hosted_vllm native Responses when the model declares supports_mid_conversation_system false. Renames the run-merging helper so it no longer shares a name with the Databricks one, and returns tuples from the block helpers. * fix(llms): place developer messages only for models declared system-first Models without a supports_mid_conversation_system declaration now keep the in-place developer-to-system translation they had before, so their cached prompt prefix does not change. Hoisting and folding run only when the deployment's model_info or the cost map declares the flag false, with the deployment value read first. A developer message that closes the conversation after an assistant turn is sent as a user message on those models instead of being hoisted, merged list content keeps a blank-line separator before each later part, and the cost map declares the flag false for fireworks_ai qwen3p8-max * fix(proxy): keep a closing developer message's cache mark when it is sent as a user turn * test: type the developer-hoist tests and fake HTTP with respx * fix: count non-string roles as turns, keep the Fireworks fold for twins, and copy closing developer content A message whose role is a list or dict now counts as a conversation turn instead of raising a 500. Fireworks native Responses skips the developer-item fold only when the deployment's own model_info declares supports_mid_conversation_system true, since the router registers every deployment's flag under its shared backend key. A hoisted developer message keeps only role, content, name, and cache_control, a closing developer message's content is deep-copied so the caller's list is left alone, and a hosted_vllm developer item with no content is forwarded unchanged. The Fireworks in-place test now declares true on the deployment itself, because a cost-map true can no longer be told apart from another deployment's registration. --------- Co-authored-by: sowmy narayanan <sowmysuresh86@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 04:12:13 -07:00
quietjules	d037d86a94	fix(token_counter)!: keep a loaded Hugging Face tokenizer when a slower download fails (#48916) ... * fix(utils): keep a loaded Hugging Face tokenizer over a racing failure and retry once per window A download that fails after another count loaded the same tokenizer no longer hides it, the first count after the retry window claims the retry while the others keep using tiktoken, and HUGGINGFACE_TOKENIZER_LOAD_RETRY_SECONDS defaults to 60s to match the proxy's tokenizer reuse window * fix(utils): keep Hugging Face tokenizer outcomes until their retry time instead of a TTL The outcome store no longer expires entries after twice the retry window, so after a long idle stretch the first count still claims the one retry and the other counts keep using tiktoken instead of each downloading again. The store is a small lock-guarded holder of an immutable mapping keyed by tokenizer kind and backend * fix(token_counter): keep the loaded tokenizer in its outcome and run one retry at a time The loaded outcome now holds the tokenizer, so an LRU eviction can no longer send every count back to an unguarded download. A retry blocks further retries until it ends, even when it runs past the window, and a download that fails after another count loaded the tokenizer counts with that tokenizer * fix(token_counter): retry a failed Hugging Face tokenizer download in the background After the retry window, the count that finds the stored failure starts one background download and answers with tiktoken, so no live count waits on a retry. A failure from a download that started before the first failure no longer resets a running retry, and a failed retry never replaces a tokenizer another download loaded. The outcome store takes its clock and retry starter as dependencies, so the tests inject them instead of patching time.monotonic. * fix(token_counter): keep retrying a failed Hugging Face tokenizer download when its retry thread cannot start If starting the background retry thread raises, the count answers with tiktoken and the failure is stored with a fresh retry time, so the next window retries again instead of the tokenizer staying on tiktoken until restart. * fix(token_counter): let a forked process or a crashed retry thread retry a failed Hugging Face tokenizer download again A failed download's in-flight retry mark is now the pid of the process running it, so a process forked during a retry starts its own instead of waiting on a thread it does not have. The retry thread clears its mark in a finally, so a native panic or fork guard error escaping it leaves the next window free to retry. The fork guard error is no longer swallowed in the retry thread and reaches the thread's exception hook, as on the first load * fix(token_counter): let a process that exits during a background tokenizer retry exit cleanly instead of aborting --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:50:58 -07:00
zoveki	c87a25a357	feat(proxy)!: general_settings.claude_code_marketplace_auth_required puts the Claude Code marketplace behind a LiteLLM key (#48980) ... * test(proxy): require a key for the Claude Code marketplace when configured and keep exception text out of its 500s * feat(proxy): general_settings.claude_code_marketplace_auth_required requires a LiteLLM key for the Claude Code marketplace When true, GET /claude-code/marketplace.json answers 401 without a key in Authorization, x-litellm-api-key, the litellm_key_header_name header, or ?key=, and GET /public/skill_hub returns an empty list. Both routes' 500 bodies carry a fixed message instead of the exception text. * fix(proxy): refuse ?key= and send WWW-Authenticate when the Claude Code marketplace requires a key * fix(proxy): close the marketplace key gate's outage, quoted-setting, and dashboard gaps Refuse the database-outage fallback identity with a 401, read a quoted "true" as on and an unreadable value as on, keep the skill hub's 500 detail a fixed string, and have the dashboard setup tab show a settings snippet with a key header when the marketplace requires a key * fix(proxy): keep the no-database 500, validate the marketplace key setting at boot, and show the key snippet until the probe answers * fix(claude-code): read the marketplace key setting from the UI config and answer database outages with 503 * fix(claude-code): keep the no-database hint on skill_hub and warn that the remove step uninstalls plugins --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:48:35 -07:00
simonp9	e8f0735d8a	feat(proxy): let a Claude Code gateway user choose the session team (#49001) ... * feat(proxy): let a Claude Code gateway user choose the session team The gateway's browser verification page now lists the developer's teams, and the device code grant mints the session for the picked team. A team the developer is not on is refused with a 400, and a form without a team keeps the first-team default. * fix(proxy): refuse a stale gateway team pick and require an explicit choice * fix(gateway): list the browser-picked team first so an older replica mid-deploy issues it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:47:27 -07:00
Mateo Wang	c489aa7117	feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708) ... * fix(proxy): keep every startup line JSON when json_logs is on Boot-time messages the proxy printed straight to stdout (banner, feedback box, model list, cache and callback notices, policy table, migration notices, CLI launch notes) and the first records of uvicorn- or granian-spawned workers came out as plain text while json_logs was on, breaking log pipelines that parse every line as JSON. - print_startup_message() prints unchanged in plain mode and, with json_logs on, emits one INFO record on the proxy logger (ANSI stripped) that skips the logger and handler level checks like the print it replaces; show_banner() routes to it in JSON mode only - the CLI exports JSON_LOGS=true when the config sets json_logs so spawned workers log JSON from their first record - the --enforce_prisma_migration_check deprecation notice prints after the config is read so a config-only json_logs covers it Resolves LIT-1624 * fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize() Since #46803 the `lite` shell calls show_banner() against an uninitialized litellm package, and reading litellm.json_logs inside it initialized the whole SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR removed on small VMs). show_banner() is back to its main version; the proxy's initialize() now emits the JSON banner record itself and calls show_banner() only in plain mode. The banner unit tests move to the initialize() call site. * fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger uvicorn applies --log_config after proxy_cli has switched JSON logging on and before initialize() switches it on again through load_config(). A logging config that names "LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where initialize() emits the banner and the feedback box, so at one worker both went nowhere; two-worker children import litellm after uvicorn's logging setup and were not affected. print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON handler uses, when the proxy logger has no handler of its own. Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the banner and the feedback box as plaintext, e2303a3925 prints neither, this head prints both as one JSON record each. New unit test fails on e2303a3925; new integration cell test_json_logs_with_operator_log_config_keeps_every_line_json passes at one and two workers. * fix(proxy): keep a long startup listing whole in its JSON record print_startup_message ran its record through the proxy logger's filters, so in JSON mode the required-migration SQL list and the policy list were middle-truncated at MAX_STRING_LENGTH_STDOUT_LOG (4096 by default) with the litellm_truncated marker, where the prints they replace showed them whole. That cap bounds echoed request payloads; a startup message is operator text, so the record now gets the uncapped processing of a DEBUG line: secret redaction and base64 collapse still apply, the text cap does not. Red on 41f92b745a, green here: - tests/unit/test_logging.py: a listing longer than the cap comes out whole; a credential in a startup message is still redacted - tests/integration/configuration/test_json_logs_startup.py: a 60-policy config boots with json_logs on and the policy listing record (9762 chars) carries every policy with no truncation marker (4096 chars, marker, on the unfixed tree) * fix(proxy): keep the --enforce_prisma_migration_check notice on --health and --test This PR moved the deprecation notice after the config is read, so that a json_logs set only in the config applies to it; --health and --test return before that point and stopped printing it, a visible change main did not have. The notice is now also emitted before those two returns, through the same helper, so a deployment running `litellm --health --enforce_prisma_migration_check` still learns the flag is dead, as on main. A normal server start emits it once, after the config, as in the previous commit. Red on 2d7a2d6844, green here: tests/unit/proxy/test_proxy_cli.py asserts the notice on --health and on --test. * fix(proxy): keep the JSON startup records on stdout under an operator handler on the proxy logger An operator --log_config that gives "LiteLLM Proxy" a handler of its own, a NullHandler that silences it or a plain WARNING-level handler on stderr, is in force in the window where the banner, the feedback box and the cache notice are emitted: uvicorn applies it after the CLI process switched JSON on and before initialize() re-installs the JSON handler through load_config(). 41f92b745a sent the startup records through the proxy logger's handlers and fell back to a JSON stdout handler only when there were none, so under such a config the records landed in the operator's handler, as plaintext on stderr, or vanished, where the prints they replace reached stdout whatever the logging config said. print_startup_message() now writes every startup record through the proxy's own JSON handler on stdout, filters included, and never through the proxy logger's handlers; the operator's handler gets nothing from a startup message, as it got nothing from the print. The normal JSON handler is the same class, formatter and filters, so the records a pipeline reads are unchanged when no operator config is in play. Tests: tests/unit/test_logging.py gets the two handler shapes (red on f6f5957d16: the record went to stderr as plaintext or nowhere); tests/integration/configuration/ test_json_logs_startup.py gets a silencing operator --log_config cell at 1 and 2 workers (red on f6f5957d16: banner, feedback box and cache notice missing; green here). * fix(proxy): print the --enforce_prisma_migration_check notice when the config fails to load f6f5957d16 printed the deprecation notice after the config is read, so a json_logs set only in the config makes it a JSON record. A boot whose config file is missing or does not parse then exited before that point and never printed it, where main printed it before the error (10-20 gauntlet finding). The notice is printed where main printed it, unless a config file can still switch json_logs on: then it is held until the config is read and printed right after the json_logs switch, and a click close callback prints it on the way out if the boot fails first. functools.cache keeps it to one print either way. Tests: tests/unit/proxy/test_proxy_cli.py covers a failed config read (red on 75043aed83), a plain config printing it once and a config-only json_logs making it one JSON record; tests/integration/configuration/test_json_logs_startup.py adds the JSON cell and a missing and an unparseable config cell (red on 75043aed83, green here), through refused_boot_log, which now takes extra CLI arguments. * feat(proxy): add suppress_startup_output to turn off startup prints * fix(proxy): print startup messages once and without color codes off a terminal * test(policy-engine): check the policy table keeps colors on a terminal and drops them on a pipe * fix(proxy): keep startup warnings when suppress_startup_output is on and apply it from CONFIG_FILE_PATH and before other settings Warnings that name something the operator must fix now print through print_startup_warning, which skips the suppress gate. The config key is read with the CLI's json_logs handling for --config and CONFIG_FILE_PATH, applied before the rest of litellm_settings in load_config, and can only turn the setting on. The Prometheus config panel obeys the setting, and startup prints use the plain log format's color rule. * fix(proxy): read CONFIG_FILE_PATH in the CLI only when workers would load it Workers load CONFIG_FILE_PATH only when it names an existing YAML file, so the CLI's early read of json_logs and suppress_startup_output skips any other value instead of failing a boot that used to start. * fix(proxy): read only suppress_startup_output from CONFIG_FILE_PATH in the CLI, without secrets The CLI now reads CONFIG_FILE_PATH as a plain file with its includes, skips it when a config bucket is set, and applies only suppress_startup_output, also when --config is passed. json_logs there stays with the workers as before. Fatal boot errors drop their color codes off a terminal like the startup prints. * fix(logging): emit startup warnings as WARNING records in JSON mode * fix(proxy): keep one JSON record per search tool in the startup listing * fix(proxy): keep booting with an ASCII stdout or an empty litellm_settings Startup messages now go through click.echo, which writes UTF-8 when stdout is ASCII, as the banner did before. An empty litellm_settings section reads as no settings in the CLI, as the workers already read it * test(proxy): boot the CLI tests on the real ProxyConfig config readers * test(proxy): cover the startup warnings that print with startup output suppressed * test(proxy): boot the deprecated resolver flag test on the real ProxyConfig * test(proxy): drop the startup output contract's narrating docstrings * test(proxy): cover the pgbouncer start failure message off a terminal --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-29 03:47:04 -07:00
devin-ai-integration[bot]	bee9f679dc	fix(github_copilot): treat a malformed api-key.json like a missing one (#44723) ... * fix(github_copilot): treat a malformed api-key.json like a missing one * fix(github_copilot): read the key and the endpoints of api-key.json as separate strict views * fix(github_copilot): ignore a non-UTF-8 api-key.json like any other malformed file Read the cached key file as bytes so a UTF-16 file, which Windows PowerShell 5.1 writes with >, fails JSON validation instead of raising the decoder's error, and the request gets the refresh or the login hint * fix(github_copilot): decode UTF-16 key files and refresh over a malformed endpoints field * fix(github_copilot): decode a UTF-16 access-token file and refuse an undecodable one like a missing one * fix(github_copilot): use the refreshed key when api-key.json cannot be written * fix(github_copilot): keep a refreshed key that cannot be saved in memory * fix(github_copilot): serve a kept key and its endpoint together after the key file is repaired A key held in memory is now checked before api-key.json, so the token and the endpoint come from the same source, and the map holds one entry per key file with the access token beside it * test(github_copilot): type the authenticator test fixture and the tests this branch adds --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:46:44 -07:00
devin-ai-integration[bot]	184ec7b894	fix(vertex_ai): build the cachedContents collection URL under a custom api_base (#44783) ... * fix(gemini): use collection endpoint for custom context cache * fix(vertex_ai): build the cachedContents collection URL under a custom api_base * fix(vertex_ai): keep a gateway prefix that contains /publishers/ * test(integration): expect the cachedContents collection URL on a custom Gemini api_base * fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix * fix(vertex_ai): cache under a publisher model api_base in that model's location A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so no cache is created that the chat call cannot use. The cache body takes its project and location from the api_base model, matching the collection URL. * fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field. * fix(vertex_ai): check the cache token minimum against the api_base's model The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create. * fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached --------- Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:46:35 -07:00
devin-ai-integration[bot]	e85cb4052e	test: fix flaky spend-log cleanup budget, login lockout, and wedged Redis tests (#47108) ... * test: stabilize cleanup, lockout, and post-kill timing tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bound the wedged Redis burst p95 by the socket timeout the test sets The burst p95 under a fixed 0.75s cap tracked how fast the box ran the 30 concurrent writes, so a loaded runner failed it at unchanged code. A handler that waits on the wedged publish takes at least the coordination Redis socket timeout, so the test now sets that timeout and asserts the p95 stays under it. * test: pin the lockout clock and bound the wedged Redis burst by the proxy's 2s wait cap The lockout tests now set the login throttle's clock instead of reading the real one, so Retry-After is pinned at 77 and a round-down fails on every run The wedged Redis burst p95 is now bounded by the 2s cap the proxy puts on its own Redis waits, so a burst wait capped under the 5s socket timeout still fails the test The fresh post-kill clients keep the harness's 2s keep-alive expiry, here and in the two tests #47080 changed * test: tell ceil from round in the lockout tests and bound the burst by the imported eviction cap The refused attempt now runs with 76.4s left on the 77s block, where ceil gives 77 but round and int give 76, so both lockout tests fail if Retry-After stops rounding up. The burst bound imports _BULK_EVICTION_WAIT_SECONDS instead of copying its 2s value, so lowering the cap moves the bound with it. * test: bound the wedged Redis burst by the shorter of the eviction cap and the socket timeout Any wait the proxy puts on the stopped Redis lasts at least the shorter of the two, so the bound now follows whichever one a later change lowers. * test: drive the tool index cleanup test with the manual clock and keep the post-kill block as on main The post-kill reset is already fixed on main by making kill() wait for the worker's last thread, so the fresh-client and keep-alive edits go back to main's version. The tool index cleanup test now advances the injected clock instead of timing the real one, and the lockout clock helper gets a type hint. * test: advance the spend log cleanup tests' manual clock through its own method * test: freeze the throttle clock in the login_utils lockout tests and drive the run budget tests with the manual clock --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:46:32 -07:00
devin-ai-integration[bot]	b93d26bc85	fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path (#45330) ... * fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path * refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge * fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper * fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints Also drop a marker set on the message item itself, sanitize before the cache_control conversion so a client typo no longer erases that breakpoint, and skip items that carry no marker so unmarked history costs no extra walk * fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a litellm_proxy alias) kept its valid markers before this PR and must keep them under drop_params. base_model is read from litellm_params first, then model_info, as chat completions does. * fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged * fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers * fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied * fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one Under drop_params the marker step now hands back the caller's own objects when every marker is already wire-safe, so a request with valid markers is no longer copied and walked a second time. A deployment that turns cache_control into markers keeps the caller's own markers too, whatever its base_model says, so one request never gets two answers about breakpoint support. On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion, so the block gets the cache hint cache_control alone would give, as on /v1/responses. * test(responses): keep a null marker and the original key order on a model that reads breakpoints --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:37:54 -07:00
zoveki	310c96162b	fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) ... * fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema * fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema. * fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them. * fix(bedrock): drop tool custom object on the Mantle count_tokens fallback Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way. * fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages * test(vertex): annotate count_tokens tool test locals with Final * fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names * fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:36:20 -07:00
rj-cairn	42d2909210	test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module (#48965) ... * test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module Session scope booted one 2-worker proxy per xdist worker and kept it until the worker's session ended, so a CircleCI node holding two of these files piled up proxies until the VM stopped responding. Module scope lets the harness group each file's rig tests onto one worker and tears the proxies down after them. * docs(tests): keep proxy fixtures module scoped in integration tests * docs(tests): say how the harness groups proxy fixtures in the integration AGENTS.md rule --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:28:39 -07:00
paveln83	032342d2b7	fix(responses): resume background response streams with stream and starting_after (#48819) ... * fix(responses): resume background response streams with stream and starting_after Retrieving a response with stream=true and starting_after=N now returns the replayed event stream from OpenAI and Azure instead of one JSON object, on the SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id} Providers that cannot replay a stored stream drop the params under drop_params and raise UnsupportedParamsError without it. Malformed values answer 400 with the param named. A replay that reaches its terminal event settles the background response's bill once and passes the provider's usage through without estimating one. An empty replay (cursor at the end) ends normally * fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams A replayed background stream now gets the same response id encryption as a created one, so the key that made the response can reuse the id. Azure AI Foundry deployments resume streams like Azure ones, and the chat-bridged Responses iterator closes its wrapped stream instead of raising. * fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query. * fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params * fix(responses): keep deployment stream off retrieves and reject oversized cursors * fix(responses): keep router default stream off plain retrieves * test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing * test(responses): replay a real stream in the billing test and type the fake wrapper chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 02:31:22 -07:00
simonp9	dff474e450	fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912) ... * fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn * fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards * fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set * fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit * fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum * fix(responses): leave a turn without a phase when its content message cannot be read * test(responses): add black box tests for chat turn history replay * test(responses): expect the replay record on bridged chat replies * test(responses): expect the replay record in bridged chat translation cases * fix(streaming): keep the served service tier on the finish chunk the stream adds * test(responses): read the expected chat message without JSON validation so the replay matcher stays --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:56:02 -07:00
devin-ai-integration[bot]	3f01341f0a	fix(health)!: openai realtime health check reports healthy with a missing or invalid api key (#41233) ... * fix(health): report openai realtime deployments with a missing or invalid api key as unhealthy * fix(health): time out a silent openai realtime handshake and keep the first-event parsing in the openai handler * fix(health): treat invalid_model as healthy, read the first event only on OpenAI hosts, map rate limits to 429 * fix(health): respect the global api_base, report a missing key as 401, keep invalid_model healthy only for known models Drop the separate first-event timeout so model_info.health_check_timeout governs the wait, and keep the shared stream error table unchanged by raising AuthenticationError for invalid_api_key in the realtime check * fix(health): keep invalid_model healthy unless the cost map lists the model outside realtime, and load realtime types lazily * fix(health): read the cost map's listed endpoints, not its mode, since registering a realtime deployment overwrites the mode * fix(health): read a realtime gateway's first event and keep Live deployments listing /v1/live healthy Behind a custom api_base the OpenAI realtime health check now waits up to 3 seconds for the first event, so a gateway that passes on OpenAI's invalid_api_key error reports unhealthy while a silent gateway stays healthy. /v1/live counts as a realtime endpoint for the invalid_model rule. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:40:02 -07:00
AaronHowell	904c092ffe	feat(providers): add BigModel support and model pricing (#48563) ... * feat: add BigModel provider with native Chat, Responses, and Messages APIs Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * test(providers): make BigModel unit tests a package Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * docs(providers): clarify BigModel parameter interface Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> * fix(bigmodel): keep the Coding Plan base, surface Responses auth errors, and scope routes Chat Completions keeps a GLM Coding Plan api_base. Responses raises the status from BigModel's success=false body instead of returning an empty 200, and a JSON body answering a stream request is replayed as a one-shot stream. Messages streams keep ping and [DONE] lines. bigmodel leaves openai_compatible_providers so speech, image, and transcription routes send it no request * fix(responses): replay a JSON body on a stream request only when it has a Content-Length A gateway that labels a chunked SSE stream as application/json kept streaming before the replay, so only a sized body is read in full * fix(bigmodel): scope the JSON stream replay to BigModel, map its business error codes, and echo requested tools * fix(bigmodel): forward Codex's include and parallel_tool_calls, keep the OpenAI org header off BigModel, and pass unchanged Messages stream events through * fix(bigmodel): map disabled thinking to low effort, keep Messages temperature, raise context window and content policy errors, and default parallel_tool_calls to true * fix(bigmodel): keep the caller's reasoning_effort when Messages rewrites disabled thinking * fix(bigmodel): keep context-window and content-policy errors on streamed Responses replays --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:39:11 -07:00
ryan-crabbe-berri	9a68573e9b	fix(responses): keep each Codex tool once when its tool list repeats across turns (#49006) ... * test(responses): cover Codex tool lists repeated across turns on the Bedrock chat bridge and Mantle * fix(responses): keep each Codex tool once when its tool list repeats across turns * test(responses): type the new tool list dedupe unit tests	2026-10-29 00:33:40 -07:00
devin-ai-integration[bot]	3c1eb73a06	fix(proxy)!: list every priced Vertex family under a vertex_ai/* grant (#41959) ... * fix(proxy): list every priced Vertex family under a vertex_ai/* grant A vertex_ai/* key listed none of the 68 cost map rows priced under the Mistral, Llama, Qwen, AI21, OpenAI, image, video and embedding labels because their sets were never part of the vertex_ai union, and Qwen rows had no ingestion branch at all. Wildcard expansion also dropped a meta/ or openai/ publisher segment as if it were a provider prefix, and the wildcard health check would have picked $0-per-token rows first once the union grew, so it now sorts paid rows ahead of unpriced ones. The VERTEX_FAMILIES_A_VERTEX_WILDCARD_GRANT_DOES_NOT_LIST allowlist is gone so the cost map guard enforces every label from now on. * fix(health): route every wildcard health-check pick through the listed provider * fix(health-check): keep per-character-billed rows ahead of rows priced $0 everywhere in the wildcard pick * fix(health-check): break $0-per-token ties toward per-character-billed rows in the wildcard pick * fix(health): only health-check ids the router sends to the provider The wildcard health check picker now skips ids that get_llm_provider routes to a different provider or to none, so the longer vertex_ai union cannot pull partner rows into the check. Drops the forced custom_llm_provider and the per-character tie-break from earlier revisions, and pins the replicate and wandb wildcard listing to each row's full owner id. * fix(health): match wildcard health picks against the provider's own route The previous check kept a bare id only when get_llm_provider routed it to the exact provider name, which dropped every ai21 row because bare ai21 ids route to the ai21_chat alias, so ai21/* health checks sent no request. Compare the bare id's route with the route of the provider-prefixed id instead, so aliases match. Resolve routes with inferred_provider, which returns the declared provider for github_copilot and chatgpt without starting their device-code login. * fix(health): keep wildcard health picks that only route through a saved custom_llm_provider Rank picks routed to the checked provider first and ids routed to no provider second, each by cost, and skip only ids routed to a different provider, so a wildcard saved with custom_llm_provider still sends its health request * refactor(vertex_ai): mark the new Vertex registry sets Final * chore: re-run CI against current main --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:31:04 -07:00
Mateo Wang	65d912fc2e	feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) ... * feat(responses): support Codex client tool_search on Anthropic and other bridged providers * test(router): cover get_search_tool_support provider defaults and model_info overrides * fix(responses): keep tools loaded by earlier Codex tool searches across previous_response_id * perf(responses): read stored turns once on client tool search continuations * fix(router): avoid OAuth refreshes when listing search capabilities * test(cli): expect disabled search on fallback Codex models	2026-10-29 00:29:40 -07:00
devin-ai-integration[bot]	ec7162aea2	fix(cost_map): mark fireworks_ai/minimax-m3 as vision and pin capabilities verified against live calls (#43390) ... * fix(cost_map): mark fireworks_ai/minimax-m3 as vision and pin capabilities verified against live calls Fireworks' serverless minimax-m3 answers image inputs, but its model listings say supports_image_input false, so the last two registry syncs flipped supports_vision back to false and the Fireworks pre-flight turned every image request into a 400 without ever calling the provider. Flip both minimax-m3 rows back to supports_vision true and record the live-call evidence in ci_cd/cost_map_pins.json. The cost map guard now fails any PR that lowers a pinned capability, quoting the evidence, and the sync bot may not touch the pins file. * fix(cost_map): fail the guard when the pins file is missing * fix(cost_map_guard): skip the pin checks on a branch cut before the pins file existed * fix(ci): check cost map pins against the merge result and keep price sync PRs off the pins file * fix(ci): run the cost map guard against the live base branch tip and re-run it when main's pins change * fix(ci): pass the live base tip to the cost map guard through env and re-run every guard run on a pin change main now runs the guard on pull_request with the base branch's copy of the script, so the workflow passes the live tip in COST_MAP_GUARD_TARGET, which the older script ignores. The recheck reads past GraphQL's 100-file cap, re-runs every guard run for a shared head, and fails when a re-run cannot start. The guard tells a merge conflict apart from a failed merge-tree call. * fix(ci): run the cost map pins recheck on litellm_ base branches too * fix(ci): bar bot-opened PRs from the pins, recheck pin PRs on cost map changes, and skip the merge for untouched PRs * fix(ci): read the pushed files from the compare API in the cost map pins recheck The push event GitHub hands to Actions leaves each commit's added, modified and removed lists null, so the recheck never saw a pins change and re-ran only the PRs that touch the pins. Compare the push's before and after commits instead, and re-check every cost map PR when the branch is new, the push was forced, the compare fails, or it lists 300 files, the most it returns. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:23:11 -07:00
rj-cairn	5eb1042bbf	fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base (#48728) ... * fix(token_counter): count OpenAI models newer than tiktoken's table with o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): keep tiktoken's encoding for suffixed variants of models it knows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): move the o200k provenance note into the assert message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): strip openai/ and azure/ prefixes before picking the encoding Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust_bridge): azure/gpt-4o now resolves to o200k_base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(token_counter): check the provider prefix through _fix_model_name to keep the type gate flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(token_counter): check o200k on the gpt-6 models tiktoken 0.14 still does not know * fix(token_counter): count Azure deployment names and OpenAI alias groups with their OpenAI tokenizer * fix(token_counter): count mixed, alias, and base-model deployments with every served model's tokenizer A group that serves OpenAI and non-OpenAI models now reserves at least each served model's count, model_group_alias names count as their target group, custom Azure deployment names count with their base model, and azure_ai/, chatgpt/ and similar prefixes resolve to the listed OpenAI model. * fix(token_counter): keep mixed groups on OpenAI tokenizers, cache served models, and count context checks, failed rows and stream fallback with the served model * fix(token-counter): keep mixed groups at their old reservation and count new OpenAI release names with o200k A group that serves an OpenAI model next to another vendor now counts its own name the way tiktoken reads it as written, plus its OpenAI models, so it reserves what main did instead of the lower o200k count base_model picks the tokenizer for estimated usage only when it names an OpenAI model, now also in the Cursor stream counter and the output-limit response Unlisted gpt-5+ and o-series names count with o200k_base, and wildcard routes run the context-window check with the matched OpenAI model * fix(token-counter): count an alias to a group no OpenAI model serves by the alias as written Keeps the count main gives such an alias, so its budget reservation and rate limit match main and no other vendor's tokenizer loads for it. Also sorts the import the main merge left unsorted. * test: type the new token-counting tests' parameters * fix(token-counter): count a release-named group no OpenAI model serves as main does, and snapshot the OpenAI model set before iterating it * fix(token-counter): read served models through the router's allowlisted deployments_named, and coordinate the registration race test * perf(token-counter): classify each distinct served model once when walking a model group Budgeted requests walk every deployment in their model group, since the router's cached copy is outside the router surface the proxy may use. Classifying each distinct served model once instead of once per deployment cuts a 200-deployment group from 2.3-4.5 ms to about 1.4 ms with the same result. --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:08:12 -07:00
Zabir Raihan	c068d1350b	fix(responses): stream every citation batch before completion (#48747) ... * fix(responses): stream every citation batch before completion * test(responses): cover citations arriving before answer text * fix(responses): close reasoning before a citation opens the message item * test(responses): build citation batches without rebinding locals * fix(responses): hold citations until reasoning closes Citations that arrive while reasoning streams now wait and stream once the message item opens, so a citation no longer ends reasoning early. Annotation events also stop sending sequence_number, matching the text deltas. * fix(responses): keep held citations when reasoning ends without answer text A citation that arrives before the answer message opens now waits for the message, then streams once text arrives or at stream end when the final response has a message. A disconnect keeps the held citation in the billed response, and a tool call with no answer text no longer streams a message item that response.completed lacks. * fix(responses): keep a held citation's message ahead of the next tool call When the answer already has content, a citation held before the message opens now opens it before the next tool call takes its output index, so the stream and response.completed keep the message ahead of the function call like the non-streaming response. * test(responses): simplify citation comprehensions * test(responses): set the new citation state on the hand-built streaming iterator * fix(responses): keep held citations ahead of tool calls and in the final message A citation that arrived before a tool call with no answer text was dropped from response.completed and the non-streaming response, because the final transform built a message only when content was set. Keep an empty-text message for those citations, and flush a held citation before the next tool call opens, so the streamed message stays ahead of the function call. * test(responses): annotate the cited tool call test locals as Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 00:03:00 -07:00
devin-ai-integration[bot]	ff22f3de06	build(deps): bump langgraph and tornado and add dated scan ignores to clear osv-scan and image-scan (#48302) ... * build(deps): suppress filelock Windows advisories and drop stale certifi entry to clear osv-scan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): suppress dev-only msal-node advisory in vscode-extension until azure identity allows 7.1 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): bump ddtrace and msgpack to clear osv-scan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): bump pylint to 4.0.9 to clear osv-scan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): lock dnspython 2.9.0 on Python 3.11+ and defer 3.10 to clear osv-scan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): bump qs in vscode-extension to clear osv-scan Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): extend the pillow 12.3.0 override for Python 3.10 to 2026-11-27 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * build(deps): extend the diskcache ignore, drop the withdrawn braces entry, and name the filelock cap in its reasons * build(deps): bump langgraph and add dated ignores for the 2026-10-28 filelock advisories * build(deps): move tornado to 6.6 on Python 3.11+, name huggingface_hub's filelock use, and scope a grype rule to the filelock High * build(deps): say which packages import tornado in its ignore reasons --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 23:51:53 -07:00
paveln83	3f5e42d032	fix(proxy): enforce team and key Allowed Vector Stores when no vector store registry is loaded (#48915) ... * test(proxy): cover team and key vector store allowlists when no vector store registry is loaded * fix(proxy): enforce team and key Allowed Vector Stores when no vector store registry is loaded * fix(proxy): skip non-string vector_store_ids entries instead of failing auth * fix(proxy): answer 400 on a malformed vector_store_ids in legacy mode too Legacy vector store access checks now parse the request the same way vector_store_deny_by_default does, so a vector_store_ids that is not a list of strings is a 400 with or without a registry. * fix(proxy): honor drop_params and a tool's string vector_store_ids in the legacy vector store check Under drop_params the check reads the stores registry retrieval uses instead of answering 400, and retrieval reads only the string entries of a top-level list. Without it, a tool's single string id is checked as one store, the way the registry reads it. * fix(proxy): honor deployment and router drop settings for a malformed vector_store_ids * test(proxy): drop vector store registry mocks the access check no longer reads * fix(auth): answer 400 for a malformed vector_store_ids only under a key or team allowlist * fix(auth): match managed vector store ids by their provider store and read a string id as one store A LiteLLM-managed store id now matches an allowlist entry for the provider store it decodes to, in both directions, so an allowlist naming store A no longer refuses A's managed id. A top-level vector_store_ids string, which is how ?vector_store_ids= arrives, is checked as one store id instead of answering 400. Registry retrieval reads an SDK tuple of ids again --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 23:25:14 -07:00
Mateo Wang	c666df3dac	docs(qa): allow focused proof for mechanical PRs (#49031)	2026-10-28 23:24:52 -07:00
ryan-crabbe-berri	bad98c4b02	fix(responses): call the renamed drops_params helper on the compact path (#49030) ... Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 06:08:24 +00:00
tessellark	b7da9a2755	fix(credentials): keep the stored secret when PATCH echoes its masked read-back (#48950) ... * test(credentials): pin that PATCH keeps the stored secret when it echoes the masked read-back * fix(credentials): keep the stored secret when PATCH echoes its masked read-back * fix(credentials): only treat a mask on a key GET masks as a placeholder PATCH now skips mask-shaped values only on the keys get_masked_values masks, using the masker's own keyword check, so a value like api_base ab****cd is saved as sent again. Also drops the new test docstrings and tightens the new test helper types --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 22:51:39 -07:00
devin-ai-integration[bot]	a66dc50efb	test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio (#43370) ... * test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio The drain regression guard compared the wall-clock time of draining 80,000 values against 40,000 and tripped its 3x bound whenever the CI runner was busy. A cursor drain allocates only per-value decode scratch while every re-slice of the remainder copies it, so the guard now bounds peak allocation during the drain, which does not move with runner load and still fails on every quadratic variant * test(json_fragment_accumulator): widen the drain's peak bound to a quarter of the payload * test(json_fragment_accumulator): guard copying drains by peak memory and rescanning drains by CPU-time scaling * test(json_fragment_accumulator): move the drain's timing guard from the unit suite into a CodSpeed benchmark * test(json_fragment_accumulator): measure only the pops after the buffer is joined, so a drain that copies once passes * test(json_fragment_accumulator): separate every other streamed value with a space, so a copy after whitespace fails the drain test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 22:04:47 -07:00
tavren27	cbf4e40d38	fix(proxy)!: stop labelling management and health rejections as auth failures (#48739) ... * test(proxy): cover management and health routes labelling rejections as auth failures * fix(proxy): name management and health route errors by their status instead of auth_error * fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls /key/list and /key/aliases now share the catch-all helper too, so they stop adding the Authentication Error prefix and typing a 400 as internal_server_error * fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error A real outage surfaces from prisma's query engine as a builtin AttributeError, which the prisma-module check skipped, so the route catch-alls still answered 500 or 400 * fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status * fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503 * fix(proxy): answer 503 for database failovers and label provider auth errors like chat A database that refuses connections while starting up, shutting down, in recovery, or read-only after a failover now answers 503 no_db_connection instead of 400. A provider's 401 or 403 keeps the type /chat/completions gives it, while a 401 or 403 the proxy raises keeps auth_error. An exception with no message is named by its type. * fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause * test(proxy): expect the OpenAI error shape from management route refusals Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:49:03 -07:00
zoveki	a11e76fe8f	fix(google_genai): read one-object and null spellings of repeated fields in countTokens and generateContent (#48771) ... * fix(count_tokens): normalize Google system parts and null tools * fix(count_tokens): preserve typed validator return * fix(count_tokens): validate single system part without typing.cast * fix(count_tokens): accept every Google spelling of a repeated field Google countTokens now reads a repeated field sent as one object, null, or a list with null items the same way as its list form, for system parts, tools, and function declarations, matching what Vertex accepts. * fix(count_tokens): read every Google spelling of contents and skip null system text contents and each content's parts now read one object, null, and null items like their list form, the same as system parts and tools. A one-object parts used to translate to no messages, so non-Google models counted it as empty. A system part whose text is null is skipped, as Vertex AI does. * fix(google_genai): read null-text media system parts as media and one-object content parts on generateContent * fix(google_proxy): count null-text system parts that carry only part metadata, like Vertex AI A system part with null or missing text and only metadata such as thought or thoughtSignature, or an empty part, now counts as empty instead of answering 400. Parts that carry an image or file with null text still answer 400. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:46:06 -07:00
Alon Sokolovski	764a7a6536	fix(context_management)!: count the system prompt toward the clear_tool_uses input_tokens trigger (#47857) ... * fix(context_management): count the system prompt toward the clear_tool_uses input_tokens trigger Anthropic's API compares an input_tokens trigger with the whole prompt, system included; the polyfill counted messages and tools only, so it cleared late by the system prompt's size. compact_20260112 already counts it: share its system_to_text. * fix(context_management): take read-only types in the system prompt count The type-discipline gate (LIT001) rejects new list and dict annotations. * chore: rerun CI (proxy-build-from-pip timed out) * fix(context_management): count the system prompt through compact's shared counter Finishes the main merge's conflict in clear_tool_uses.py. The input_tokens trigger now adds the system prompt through count_system_tokens, the same counter compact uses, while the count reused for cleared_input_tokens stays messages and tools only, so the reported savings are just the cleared tool results * perf(context_management): skip the token count when nothing can be cleared The keep check runs before the trigger count, the system prompt reader accepts any mapping block, and the system prompt test gets short parameter ids * test(context_management): keep the off-loop count test counting after the keep check moved first * test(context_management): cover a client compaction summary counting toward the trigger The trigger also skips the system prompt count when messages and tools alone already pass it * fix(context_management): move a client compaction summary into the system prompt when clearing runs before compact * fix(context_management): clear only the history after a client compaction block so compact still replaces its summary * fix(context_management): keep tool calls beside a client compaction block clearable --------- Co-authored-by: AlonSoko <25619863+AlonSoko@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:35:32 -07:00
mika-lt	de42b67252	fix(proxy)!: enforce an end user's per-model budget from their linked budget (#48815) ... * fix(proxy): enforce an end user's per-model budget from their linked budget The shared auth checks now refuse a request once the end user's spend on that model reaches the model_max_budget of their budget (a linked budget or the default applied to them), for virtual keys and the master key alike. The cap is read from the end-user row, since the master key's cached auth object keeps whichever end user's limits it was cached with. The per-model end-user check inside virtual-key validation is removed: it read a token field that is never set there, so it never ran. * fix(auth): enforce the end user's per-model budget on JWT auth and skip it on budget-exempt routes JWT tokens now carry the end user's per-model caps so spend is charged against them and over-cap requests are refused. Routes that skip budget checks, like /spend/calculate, skip this check too. The proxy default end-user budget is covered by a test. * fix(auth): write the end user's per-model cap onto every token and check it after the member fallback The shared auth checks now resolve the end user's per-model cap, falling back to the key's or the proxy's default end-user budget when the end user has no row yet, and write it onto the token so the spend hook charges the cap the check enforces on every auth path. The check runs after the team-member budget fallback, on the model it picked, and the JWT builder's duplicate check is removed * fix(auth): keep only a per-model end-user cap that custom auth itself set A key-auth token reached through custom auth's auto hand-back now takes the current end user's cap, so a cached master-key or virtual-key object no longer carries an earlier end user's cap into another end user's request. * test(auth): build the custom-auth token with model_copy instead of setting the flag in place * test(auth): hold an end user to their own or their key's budget cap when the proxy has no default budget * fix(proxy): give each request its own copy of a custom-auth token so end-user caps and ids stay per request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:24:11 -07:00
tessellark	5bd47442fb	fix(cli): give lite codex aliases of a Codex-known model that model's catalog entry (#48894) ... * fix(cli): give lite codex aliases of a Codex-known model that model's catalog entry lite codex built its Codex model catalog from /v1/models ids only, so a proxy alias of a model Codex knows (api-gpt-6-luna for openai/gpt-6-luna) got the fallback entry. That entry has no supports_search_tool or tool_mode, so Codex sent every MCP and app tool inline and each turn cost far more input tokens. lite now reads each listed model's upstream from the proxy's /v1/model/info (team_public_model_name when set, else model_name, first deployment wins) and passes it to the shared catalog builder, the same way the proxy's own Codex route does. If that request fails, the catalog is built as before. * fix(cli): give a lite codex alias an upstream only when all its deployments route to one OpenAI model * fix(cli): cap a lite codex alias's context window at its listed max_input_tokens and count only OpenAI-hosted deployments * fix(cli): keep Codex's tier, upgrade and reasoning defaults off lite codex aliases * fix(cli): keep Codex's own effort, verbosity and promo off lite codex aliases and skip credential-hosted deployments --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:23:55 -07:00
tessellark	dbaae521f9	fix(responses): drop the null created_by OpenAI rejects when output items are replayed (#48783) ... * fix(responses): drop the null created_by OpenAI rejects when output items are replayed * fix(responses): drop a null created_by from replayed input items too A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null. * fix(responses): drop a null created_by from counted input items too * test(responses): cover saved reasoning nulls and re-compaction of a saved conversation * fix(responses): drop a set created_by from input under drop_params * fix(types): type the Responses output serializer's parameters * fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:11:49 -07:00
devin-ai-integration[bot]	b1b5d16f0b	docs(adrs): add ADRs for the Python SDK and proxy, starting with provider usage extras (#36668) ... * docs(adr): add architecture decision records, starting with provider usage extras transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(adrs): correct where provider usage extras survive and scope the live proof bar to runtime changes * docs(adrs): say which chat transforms keep unknown usage keys and where a new field is passed * docs(adrs): correct ADR 001's account of the xAI billing PR, date its code facts, and move the billing recipe to CONTRIBUTING.md * docs(adrs): say the stream builder carries Anthropic's inference_geo by name and drops other extras * docs(adrs): cover /v1/messages, the cost_per_token arm, and the Gemini tier side channel in the usage-field ADR and contribution guide * docs(adrs): decide that usage extras reach callers and shared paths copy them, scope out provider-reported cost, and fix the proof steps --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:09:04 -07:00
Mateo Wang	be649f66a6	feat(azure_ai): reach Foundry agents through the project Responses API (#47717) ... * test(azure_ai): cover Foundry agents on the project Responses API * feat(azure_ai): reach Foundry agents through the project Responses API * fix(azure_ai): keep system prompts and project-endpoint auth on Foundry agent calls and follow-ups * fix(azure_ai): read AZURE_SCOPE before the Foundry audience default on project endpoints * fix(azure_ai): reject or drop instructions on previous_response_id turns and a caller's other agent for Foundry agents * fix(azure_ai): apply the other-agent rule to agent_reference sent at the top level of a Foundry agent request * fix(azure_ai): match the caller's agent by name and version, and hide the agent's own instructions * fix(azure_ai): keep instructions on plain Foundry models that return a null agent_reference * fix(azure_ai): keep a Foundry agent version requested at the top level when extra_body also names the agent * fix(azure_ai): answer 400 for /responses/compact on a Foundry agent instead of calling the project with the agent name	2026-10-28 20:49:58 -07:00
zoveki	b96c1f30a6	feat(ocr)!: support outputContentFormat=markdown for Azure Document Intelligence (#48926) ... * feat(ocr): support outputContentFormat=markdown for Azure Document Intelligence * fix(ocr): map only span endpoints when slicing Azure Markdown pages * fix(ocr): accept any casing of outputContentFormat and report code point offsets Azure reads outputContentFormat case-insensitively, so Markdown and TEXT now map to the same values instead of returning 400. When the proxy asks Azure for code point offsets to split Markdown pages, the response now carries stringIndexType so callers know the unit of table and key-value spans. Tests use rstest and the enum derives VariantArray per the Rust workspace rules. * fix(ocr): ask Azure for code point offsets only on prebuilt-layout and keep line-joined pages when spans are malformed * fix(ocr): trim Azure DI outputContentFormat, accept output_content_format, and drop invalid values under drop_params * fix(ocr): fall back to output_content_format when outputContentFormat is empty --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:38:23 -07:00
devin-ai-integration[bot]	a2ef18b257	test(unit): keep unix socket paths in unit tests under the macOS AF_UNIX limit (#45040) ... * test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit * test(unit): share the short socket fixtures across unit tests Move socket_dir and socket_path to tests/unit/conftest.py so the datadog redis test's absent socket fails for the right reason on a Mac, rename the pgbouncer helper's directory argument, and record the rule in tests/unit/AGENTS.md * test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path socket_dir now uses /tmp when TMPDIR plus the longest socket suffix would pass macOS's sun_path limit, and its teardown ignores errors so a failed test reports only its own failure * test(unit): keep socket fixtures inside tmp_path behind a short link socket_dir now links a short temp directory to tmp_path, so artifacts keep pytest's retention and a killed run leaks only an empty link. The fallback to /tmp counts the temp directory in bytes, and socket_tempdir points tempfile at it for the pooler tests whose socket directory production picks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:23:22 -07:00
Kerry Lu	56e60d3b7d	feat(router): daily_reset_time picks when deployment rpd and tpd daily limits start over (#48957) ... * feat(router): reset_time picks when deployment rpd and tpd daily limits start over reset_time is "HH:MM" in UTC next to rpd/tpd in litellm_params or model_info. The daily cache keys are dated by now minus that offset and the daily 429 retry-after counts to the next reset, so a free tier whose provider quota resets at 08:00 UTC comes back at 08:00 instead of midnight. Without it the window stays the UTC calendar day. * chore(ui): regenerate schema.d.ts with reset_time * refactor(router): rpd, tpd and reset_time are read from litellm_params only * feat(router): rename reset_time to daily_reset_time, drop the request-sent value, accept only HH:MM * chore(ui): regenerate schema.d.ts for daily_reset_time --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 20:20:01 -07:00
Mateo Wang	f46adf80e4	fix(proxy): let typesafe/* allowlists cover /typesafe Jev calls and list Jev in the model picker (#47697) ... * fix(proxy): list TypeSafe Jev models in the admin model picker * fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones. * fix(typesafe): match only published Jev ids and keep picker loads off the secret manager Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file. * test(typesafe): type the picker test fixture and tests * fix(typesafe): match only published Jev versions and cover non-version suffixes in tests * fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker * fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers The dashboard sends include_model_access_groups on every /models call, so gate the TypeSafe names on wildcard routes too and skip access-group-only listings. Also say the typesafe-jev-ids rule matches case-insensitively. * fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main A typesafe/* router deployment registers typesafe/* in the cost map, so the allowlist pickers offered a bare *, which allows every model. The pickers now list only priced names the rule sends to TypeSafe. The non-expand list also stops de-duplicating every entry and only skips TypeSafe names already listed.	2026-10-28 20:19:58 -07:00
kailian zhou	3a350cacc9	feat!: add StreamLake provider metadata (#41006) ... * feat: add StreamLake provider metadata * fix(streamlake): complete provider discovery and test request flows * test(streamlake): assert cost and tool-param invariants instead of pinning vendor prices * fix(streamlake): accept tools and reasoning on endpoint ids and match the current price page Endpoint ids have no cost-map row, so tools and reasoning_effort were rejected before reaching StreamLake. The provider now declares both capabilities, and named non-reasoning rows opt out explicitly. The rows now follow StreamLake's 2026-09-17 price page: drop the KAT-Coder and Qwen3 rows it no longer serves, add the served models that had no price, drop account-level rpm and tpm, and fill missing context windows. StreamLake is also registered for streamlake/* wildcards and its messages and responses support. * fix(streamlake): regenerate the endpoint matrix and type the new tests The backup endpoint matrix advertised chat only while the root map listed Messages and Responses, so /public/endpoints hid both. Regenerating with scripts/generate_provider_endpoints_support.py makes the two files match. Types the StreamLake tests and sets the Add Model test's API key with fireEvent.change, per the dashboard test guide. * fix(streamlake): drop the retired max_tokens field from the StreamLake cost-map rows * fix(streamlake): price the four served models StreamLake publishes rates for and add the Qwen3.8 cache prices * fix(streamlake): send token caps as max_completion_tokens and mark image models StreamLake ignores max_tokens on kimi-k2.6, kimi-k2.7-code, qwen3.6-27b and qwen3.6-35b-a3b, while every served model honors max_completion_tokens, so the JSON provider now maps max_tokens to max_completion_tokens instead of the reverse. Rows whose models read an image in a live probe set supports_vision, and the Qwen3.8 rows drop the explicit cache write price, which this route never uses since cache_control is dropped * fix(streamlake): keep cache_control marks, price explicit cache reads, and use API input limits StreamLake reads cache_control, so the provider keeps the marks through a preserve_cache_control flag. Qwen3.8 explicit cache hits now use their own read price, rows with a cache read price declare prompt caching, three models StreamLake serves without reasoning drop reasoning_effort, and max_input_tokens follows the limits the API reports. * fix(utils): warn when a custom_provider_map name matches a built-in provider --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:19:06 -07:00
simonp9	0f68b9919c	fix(guardrails): skip embeddings input in google_text_moderation (#48934) ... * fix(guardrails): skip embeddings input in google_text_moderation and openai_moderations Since v1.103.0 the proxy dispatches these legacy moderation callbacks on every route, and they read the embeddings input field as text, so benign embedding inputs were blocked. Return early for embedding call types, matching the Aim and Cato guardrails. * test(guardrails): type the embeddings moderation test parameters * fix(guardrails): keep openai_moderations moderating embeddings, skip only google_text_moderation Google's moderation scores topic categories like Finance and Health, so it blocks documents being indexed by subject. OpenAI's moderation scores harm categories only, so it keeps moderating embedding input as it has since v1.103.0 * fix(guardrails): skip Gigachat passthrough embeddings in google_text_moderation Treat an allm_passthrough_route call whose endpoint ends in embeddings as non-conversational, log the skip, return None from the early exit, and type the hook's data so the basedpyright gate does not grow * fix(guardrails): match only plain embeddings passthrough paths in google_text_moderation --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:05:03 -07:00
mika-lt	4e5f585c2a	fix(guardrails): scan Responses tool results in texts-mode guardrails (#48827) ... * fix(guardrails): scan Responses tool results in texts-mode guardrails * fix(guardrails): keep guardrail-added tools and non-text parts on Responses tool scans Tool result texts now come from the same conversion that builds the chat tool message, so items without a call_id, dict outputs, and outputs with no text parts stay aligned for Bedrock and PANW. A changed joined text replaces the first text part and keeps the other parts. scan_only_tool_results appends tools the guardrail returns with new names, like chat, and tool_result and computer_call_output items are scanned through the shared type list. * fix(guardrails): skip empty and screenshot tool outputs and keep text part types on Responses scans * fix(guardrails): keep Bedrock latest-turn masking aligned when tool outputs send no text * fix(guardrails): scan each Responses tool output text part on its own * fix(guardrails): split a Responses tool output's structured row when only textless outputs share its call_id * fix(guardrails): pair each Responses tool output with its own structured row --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:01:07 -07:00
simonp9	83ae1bd63e	fix(responses): drop params compact rejects and send Azure compaction to the v1 route (#48876) ... * fix(responses): drop metadata from compact requests to OpenAI and Azure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): drop user and drop_params-rejected fields on compact and route Azure compaction to v1 * fix(responses): keep stream on compact and keep non-dated Azure api-versions on the v1 compact route * fix(responses): drop compact params only for providers verified to reject them --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 19:46:36 -07:00
Kerry Lu	4fc2bba1f2	fix(stream): the last usage frame carries litellm's bill in usage.cost (#48955) ... * fix(proxy): copy the stream's hidden response_cost into the client's streamed usage.cost The stream is priced once into hidden response_cost (stream builder for chat and text completions, the Responses iterator for the terminal event) and the proxy egress copies that bill into the streamed usage.cost on a copy of the event. Inside the pipeline usage.cost stays what the upstream reported. Spend rows do not change; the billing fix for upstream-reported usage.cost follows in the next PR cut from #48287. * fix(proxy): type the bridge bill carry and the builder's reported-cost guard for the lint gates * style: format the chat-over-Responses bridge helper signature * refactor(bridge): build the billed Responses event with model_copy instead of a TypeAdapter chain * fix(stream): keep writing the computed cost into usage.cost as on main, the hidden copy is only for the proxy egress * fix(stream): write litellm's bill into the usage frame's usage.cost instead of a hidden field the proxy copies * test(stream): keep main's formatting in test_streaming_handler.py, only the new test changes * refactor(stream): drop the docstring on _stamp_usage_cost, the body says it --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 19:32:44 -07:00
ryan-crabbe-berri	019a9de603	fix(logging): stop dumping the request body at every layer under --detailed_debug (#48973) ... * fix(logging): stop dumping the request body at every layer under --detailed_debug The proxy dumped the full request body at every layer it passed through (proxy entry, pre-call, router, SDK call echo, param mapping, logging object), so one Codex "hi" wrote over 500KB of debug log. Keep the inbound request line and the outgoing provider request, summarize large values in the SDK call echo, and stop per-request lines that report nothing (callback disable checks with no disable header, cost map name probes, response id decode misses, vector store path misses). * fix(logging): summarize tools in the param mapping debug lines Non-Default params and Final returned optional params still printed full tool definitions, so a Codex-style request repeated them twice more. Use the same size summary as the SDK call echo and cover tools in the test. * perf(logging): build the param mapping debug summary only when debug is on	2026-10-29 02:30:33 +00:00
David Koh	57f05e56f6	fix(bedrock): omit system prompt from audio transcription and report its token usage (#41565) ... * fix(bedrock): omit system prompt from audio transcription * fix(bedrock): report Converse token usage on audio transcription Bedrock transcriptions returned usage null and were billed at $0. Map Converse inputTokens, outputTokens and totalTokens into the transcription usage so spend uses the model's per-token price. input_token_details is optional because Converse does not split audio and text tokens. Also switch the updated request-shape test to rstest. * fix(bedrock): default transcription temperature to 0 and price ap-southeast-2 Voxtral regionally Voxtral wrapped some transcripts in quotes at its default sampling temperature. In direct Converse calls in ap-southeast-2, 0 of 48 calls at temperature 0 were framed against 7 of 72 at the default, and instruction wording made no clear difference, so the request now sends temperature 0 when the caller sets none, matching OpenAI's transcription default. Calls in ap-southeast-2 billed at the us-east-1 price. The cost map gains bedrock/ap-southeast-2 Voxtral rows with all four service tiers, the Bedrock transcription response carries the deployment region in its hidden params, and the transcription token cost uses the region-resolved model key. The bridge now builds the response with model_copy instead of assigning usage after construction, and the native wheel test checks the usage the compiled route returns. * fix(bedrock): forward transcription language, reject unsupported formats, and bill env and Tokyo/London regions get_optional_params_transcription had no Bedrock branch, so language, response_format and timestamp_granularities were dropped before the Rust transform ever saw them. Bedrock now forwards language, prompt and temperature, accepts response_format json or text (the only shapes it can return), and answers 400 for srt, vtt, verbose_json or timestamp_granularities unless drop_params is set. The billing region now falls back to AWS_REGION_NAME and then AWS_REGION when aws_region_name is unset, the same order the Rust resolver uses to pick the endpoint, so an env-configured region gets its regional price. Adds ap-northeast-1 and eu-west-2 Voxtral rows (standard, batch, flex and priority) from the AWS Bedrock offer files, the other two regions AWS serves Voxtral audio in. * fix(bedrock): send transcription temperature as a number and name the rejected param The proxy keeps multipart form values as strings, so -F temperature=0.2 reached Converse as "0.2" and Bedrock answered 400. The Bedrock mapper now converts temperature to a float and answers 400 (or drops it under drop_params) when it is not a finite number. get_optional_params_transcription copied prompt and temperature back from the raw call after the Bedrock branch ran, which undid that conversion and any drop. The Bedrock branch now owns every OpenAI transcription param. An unsupported param such as timestamp_granularities now gets a 400 that names it instead of the generic "user/encoding format" message. * refactor(bedrock): return unsupported transcription params as a value and raise at the param step * fix(bedrock): drop the retired max_tokens field from the regional voxtral price rows --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 19:23:15 -07:00
ryan-crabbe-berri	d9c9d84298	fix(proxy): stop log lines from leaving terminal colors on (#48966) ... * fix(proxy): stop log lines from leaving terminal colors on * test(proxy): drop source scan for unclosed terminal colors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 18:45:24 -07:00
Mateo Wang	7aa7948a47	fix(health): ask OpenAI and Azure audio chat models for audio output in the health check probe (#47730) ... * fix(health): ask OpenAI and Azure audio chat models for audio output in the health check probe * fix(health): check the probed model when health_check_model is set, and mark Azure audio models as audio output A deployment with an audio base_model and a text health_check_model got the audio fields sent to its text probe model. The Azure gpt-audio and gpt-4o audio preview cost-map entries now set supports_audio_output, so a dated Azure name like azure/gpt-audio-1.5-2026-02-23 gets the audio probe too * fix(health): read litellm_params.base_model, and drop the stored base_model when Test Connect swaps the model * fix(health): judge the audio probe on a swapped-in health_check_params model, and keep Test Connect's stored base_model * fix(health): look up a provider-prefixed base_model under its own provider and keep text-only operator modalities	2026-10-28 18:37:29 -07:00