* test: migrate proxy security, schema migration and MCP legacy tests
* test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp
* test: warm the baseline store before timing the locked projection
* test(e2e): hold the stack exclusively while the semantic filter flag is flipped
* test(mcp): scope the in-process proxy e2e fixtures to their module
The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite
* test(e2e): assert the semantic filter keeps listed tools instead of a ranking
Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k
* test(mcp): keep the in-process proxy e2e tests on one worker
loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out
* test(mcp): name the upstream requests that carried the wrong token
The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* refactor(rust_bridge): group the Python modules Rust imports by name under rust_bridge/host
Rust reads Python-owned state (legacy callback registries, model capabilities) through
modules it imports by path. Put them in one folder with an AGENTS.md that states the
contract, and update the module path constants on the Rust side.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust_bridge): route Python state reads through host accessors
* docs(rust_bridge): point test_support at the moved host contract test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust_bridge): move secret manager, transport and timeout accessors under host
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(greptile): register rust_bridge/host AGENTS.md in files.json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GPT reasoning-series models (gpt-5.x, gpt-6) on Bedrock Mantle chat completions now send max_completion_tokens, carrying a deployment or client max_tokens over with its value. Every other Mantle chat model keeps sending max_tokens.
* test: migrate tests/proxy_behavior into integration and unit tiers (partial)
* test(integration): run settings-dependent management tests on an owned proxy
* test(integration): assert stored values in the rewritten management rows
* test(integration): pin sibling team allocations against the org guaranteed throughput cap
* test(integration): keep Prisma clients out of the shared integration conftest import path
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* ci: run every CircleCI check from the integration workflow
* test(ci): keep the startup test CircleCI-owned
* test(ui): find reset log rows by request id so concurrent workers cannot page them out
* test(ui): retry the key budget window spec until the draft refresh fix lands
* ci(ui): set Playwright workers and retries on the CircleCI command only
The shared playwright.config.ts also drives the Buildkite UI suite, where 3 workers
ran out of memory on one of two runs. Buildkite keeps main's 1 worker and 2 retries
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e
* test: fix disk cache collection, ungate saturation check, drop RSS memory tests
* test: assert one delivered log per completion in Datadog and LangSmith checks
* test: drop SDK-direct e2e copies and align cost-map path filter cases
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(home): serve the bundled whats_new.json by default so a release shows only its own launches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4dd96800e8)
* style: annotate whats_new_url with str | None
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit d1ae7ea085)
* test(home): pin whats_new_url to the env var and regenerate the dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4bfec53865)
* test(public_endpoints): prove the /public/whats_new route serves the bundled list when no URL is configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(public_endpoints): drop the unused os import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(home): pin whats_new_url to the env var so a baked-in default URL fails the suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(tests): sort the import blocks ruff flagged in test_public_endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(home): let the route test read the real whats_new_url default so a baked-in URL fails it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for customer spend rows and use owned OpenRouter cost
* ci: isolate integration workers and clean up resources after crashes
* docs: describe isolated extensions workers and shard census
* fix(ci): restore integration service and fixture guarantees
* fix(integration): preserve replay IDs and native route fixtures
* fix(tests): scope integration harness imports to each unit case
* fix(ci): preserve fixtures and wait for owned workers
* fix(ci): isolate nested runs and telemetry cases
* fix(ci): make worker readiness and fault tests deterministic
* chore(ci): drop MCP fixture repair already covered by main
* chore(ci): keep the local catalog fix from main
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: migrate selected llm translation contracts
* test: migrate anthropic completion translation tests
* test: migrate bedrock and cohere translation tests
* test: tighten bedrock translation migration
* test: migrate azure and openai model translation tests
* test: migrate openai and small-provider translation tests
* test: migrate gemini and vertex translation tests
* test: migrate optional params and rerank translation tests
* test: migrate huggingface, json and openrouter translation tests
* test: fix openrouter integration expectations
* test: migrate databricks, xai, cometapi and fireworks translation tests
* test: migrate ovhcloud, prompt and small provider translation tests
* test: split reasoning effort grid into unit and e2e
* test: migrate router and realtime guardrail translation tests
* test: migrate google interactions translation tests
* test: migrate a2a, evals, skills and hosted vllm translation tests
* test: remove remaining llm translation suite
* test: tighten migrated llm translation unit tests and land missing rows
Restores rows the first pass dropped or weakened (bedrock agentcore and openwebui thinking stream, anthropic prefix, citations and tool_choice, azure agents handler, huggingface bare api_base, optional params num_retries, alternating roles, rerank response type), replaces truthiness and call-count checks with whole-value assertions, and moves tests to the mirror file of the code they cover
* test: move cohere, databricks, fireworks and gemini rows to mirror files over mocked HTTP
Restores three TestCohereV2Transform methods the first pass nested out of collection, replaces direct transform calls with respx-backed completion calls, and drops the databricks and fireworks files that sat outside their mirror paths
* test: close lost-intent gaps found in review of the llm translation migration
Points the hosted-mock guard at the triton test's new integration path, restores exact rerank result assertions for infinity, drops the reasoning e2e fallback that passed for any multi-line answer, and adds the mixtral invoke system prompt variants, eval deletion, and async pdf url download that the legacy suite exercised
* test: assert provider acceptance in the reasoning effort e2e
The live half of the split grid proves each route accepts reasoning_effort. claude-fable-5-1 on the direct route answers without surfacing reasoning content, so a reasoning-presence check either fails on head or needs a fallback that passes for any answer. Wire shape stays covered by the unit grid
* test: cover async and stream-end paths the legacy translation suite exercised
Adds async variants for bedrock converse file and thinking content, bedrock embedding timeout and region resolution, interaction stream termination and success logging, xai reasoning token folding, and bytes chunks with a prefix before data:. Also scopes the vertex safety settings global in test_vertex_safety_settings with monkeypatch so it stops leaking into gemini request bodies on the same worker
* test: type the new llm translation e2e calls and mark moved fixture dirs as packages
* test: name the reasoning effort e2e models as module constants
* test: assert reasoning content, register the new gemini coverage ids and give the guardrail config e2e a unique name
* test: block the prompt the shared guardrail denies and drop the reasoning presence check
* test: drop the duplicate litellm import left by the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move guardrails, image, audio, batches, search legacy tests to unit
* test: restore the custom guardrail and remaining chunk 1 unit rows
* test: move the ocr matrix to e2e
* test: drop comments from moved tests and fix vertex ocr credentials
* test: move litellm utils and unified google legacy tests to their tiers
* test: move logging callback and pass-through unit legacy tests to their tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move store_model_in_db legacy tests and start the admin ui migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: finish the proxy job rows and drop their CircleCI jobs
Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI
* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names
* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist
* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in
* test: type the e2e beta header, prompt cache and secret manager values
* test: assert the missing model error message on the OpenAI routes
* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks
* test: make the langfuse fixture directory a package for the layout check
* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane
* test: keep the ocr e2e pdf fixture next to the test
* test: put meta below covers on the streaming tool search e2e
* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures
* test: count only OCR calls in the OCR matrix recorder
* test: pin OCR parse bodies and billed cost, group new e2e cases into classes
* ci: restore the python image anchor dropped in the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): call the ai_decide AI Function route for decisions and the auto-router
Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not
on a serving endpoint's /invocations route. The decisions provider and the
auto-router classifier now accept only the ai_decide model, post to that route
without a model field, unwrap the response envelope, and map a noul answer's
probability to the System One noul field.
* test(databricks): bound the megabyte model name test by scaling, not wall time
* test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound
* test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route
* fix(databricks): keep serving endpoint decisions beside the ai_decide route
* test(databricks): assert the whole classifier request body in the test routing case
* test(databricks): correct System One spend call type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): load the Home page What's new launches from /public/whats_new at runtime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): show the three newest What's new launches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build: ship whats_new.json in the wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the evaluation model mode to decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): canonicalize the health check mode without rebinding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): replace canonical_model_mode with is_decisions_model_mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): preserve evaluation mode compatibility
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop the rebuilt dashboard bundle from this PR
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(dashboard): format decision mode helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(headroom): drop null tool_calls before calling /v1/compress
headroom-ai 0.27.0 through 0.30.0 iterate msg.get("tool_calls", []) and fail
with 'NoneType' object is not iterable when an assistant row carries an explicit
tool_calls: null, which the fail-closed guardrail then surfaces as a 502. Strip
the null key from rows sent to the compression service; real tool_calls lists
are forwarded unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(headroom): pin that SDK-echoed null tool_calls never reach /v1/compress
Main now withholds every assistant row from /v1/compress (#39974), so the
null tool_calls strip added here never runs on a row headroom-ai 0.27-0.30
fails on. Drop the strip and pin the customer's history shape through the
public OpenAI chat translation handler instead.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(ui): add a Home page as the dashboard root with usage summary, recent launches, blog updates and a model leaderboard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back to spend-log request counts on Home when the gateway has recorded none
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover the Home page totals, launches, leaderboard and updates feed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): render Home Updates as rows and return up to 6 blog posts from /public/litellm_blog_posts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): filter Home Updates by the new-model and api-change blog tags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(blog): import NotRequired from typing_extensions for py3.10 and test the endpoint passes every fetched post through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(ui): drop the Updates filter chips, keep Updates as a plain row list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): restore generated schema.d.ts to main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): label the Home key button Create New Virtual Key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): drop the What's new sublabel on Home
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show an error with retry when Home usage fails to load and move the Home test to the integration tier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit b4f9e14a44)
* refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 819faa01c3)
* feat(model_hub): surface model_info.description in model group info and Model Hub UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit b224b15b9d)
* refactor(router): aggregate model group description without in-place mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 342b93fe0a)
* test(dd_span_tagger): assert the user_email tag on the active span instead of on the patched setter
The two ported tests only inspected the mock that replaced the tag setter, which the test quality gate counts as two new mock-echo tests over main. They now read the tags that land on the active span
* refactor(router): read the model group description through the typed model_info reader
The ported lookup read model_info off the untyped deployment dict, which added three basedpyright errors over main. It now goes through the existing validated model_info reader
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(guardrails): scan or refuse attachments in the bedrock guardrail
* fix(guardrails): cover native pre_call and computer screenshots in the bedrock attachment scan
* fix(guardrails): classify every attachment shape the bedrock guardrail can receive
* test(guardrails): type the attachment-scanning test guardrail
* fix(guardrails): keep base behavior for Bedrock subclasses and malformed messages
- skip the attachment scan when a subclass overrides apply_guardrail
- ignore non-string message role/type instead of raising
- treat a null Converse document/video/audio block as absent
- keep the base request-body debug log for text-only calls
* fix(guardrails): let text documents, unpadded base64 images and request-override subclasses through as before
* fix(guardrails): let Converse text documents through and refuse documents that carry images
* fix(guardrails): scan images inside content documents instead of refusing them
* fix(guardrails): scan text document content with the Bedrock guardrail
* fix(guardrails): scan document title, context and nested documents
* Expand nested content documents without recursion
* Keep attachment scan within the type-discipline budget
* Take the image format and pixel size from the image bytes
* chore(guardrails): satisfy type-discipline gate after merging main
* chore(guardrails): drop docstrings added by the attachment scan
* fix(guardrails): scan Converse text documents and guardContent images
A Converse document whose source is text (or a list of text parts) now
goes through the Bedrock text scan instead of being refused as
unscannable, so the pass-through request that worked before this PR
keeps working. A Converse guardContent image is now scanned like any
other image. Byte and S3 document sources stay unscannable.
Adds the integration test that drives every scanned endpoint through a
real proxy against a scripted Bedrock ApplyGuardrail stub, and retypes
the touched hook signatures for the type gates.
* test(guardrails): cover Bedrock checks mode in the attachment integration test
Adds a checks-mode proxy to the rig: images are refused without calling Bedrock on every route, and a Converse text document is sent to InvokeGuardrailChecks.
* fix(guardrails): scan a Converse document's name and context whatever its source
A Converse PDF or S3 document let through by skip_unscannable_attachments now still has its name and context scanned as text, the same as the Anthropic document path.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(langfuse): log token usage for /v1/responses calls
ResponseAPIUsage exposes input_tokens/output_tokens, so the Langfuse
generation logger read prompt_tokens/completion_tokens as 0 while cost
was still correct. Normalize the usage with
ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage
before extracting usage_details.
Resolves LIT-8238
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): adapt responses-api usage regression test to langfuse v4 spans
The v4 migration exports usage_details as an OTEL span attribute via an
in-memory exporter instead of a mocked generation() call. Assert on the
exported span's langfuse.observation.usage_details with the same token
counts.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): log token usage for streamed /v1/responses calls
An assembled /v1/responses stream reaches the logger with its usage stored
as a plain dict of chat fields, which the attribute reads saw as zero. Route
both ResponseAPIUsage and that dict through one normalizer.
* fix(langfuse): keep generations whose usage dict fails validation; cover streaming in the e2e
A usage dict whose token counts are not integers falls back to the raw
object, so the generation is still logged with zero usage instead of being
dropped. The live e2e now runs streamed and non-streamed /v1/responses.
* ci: re-run the e2e gate
* test(langfuse): run the real prompt helper and label the responses usage e2e cases
Drop the _add_prompt_to_generation_params pass-through patch from the
Responses usage unit tests so the real helper runs with the empty metadata
they already pass. Declare a Subject on both e2e parametrizations
(observability, /v1/responses, openai, stream and nonstream).
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(integration): move the openai cost tracking cases from cost_tracking_cases.json to CostTrackingTestCase literals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move the remaining cost tracking cases, the cost map and the batch/realtime cases from cost_tracking_cases.json to Python literals (#45885)
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(credentials): keep credential labels and edits consistent across UI, API, and replicas
The credential modal flattened stored credential values into the same form object as the
credential's own name, label, and provider, so a stored value keyed credential_name or
display_name replaced the real one and the edit went to the wrong credential or relabeled it.
The modal now submits the top-level fields and credential_values separately
PATCH /credentials now reads the row it merges onto from the writer, so an edit inside the read
replica's lag window no longer writes back an older label or key
POST /credentials answers 409 for a name defined in config instead of creating a DB copy that
shadows it
display_name rules live on the server only. It rejects control, formatting, and line separator
characters, and the UI stops enforcing its own length check and shows the server's error
The models table, model detail view, and model edit picker show the credential's technical name
under its label whenever the two differ
* fix(ui): show credential source in its own column like models
The config marker sat under the credential name. The credentials table now has a Source column
with a Config or DB status badge, matching the Source column on the models table
* test(credentials): unit-test label and prefill helpers and keep the full credential name on hover
Moves displayNameChange and initialFormValues out of the modal so their rules are unit tested,
keeps only the wiring cases in the modal's integration test, makes the label acceptance test
write through the real credentials repository, and adds the full credential name as a hover
title where the models pages truncate it
* test: migrate local_testing SDK and provider tests out of build_and_test
* test: cover sagemaker region, hf template and nova paths end to end
Region rows now go through litellm.completion on the sagemaker/ handler that
the legacy tests exercised, the prompt template test mocks the Hugging Face
tokenizer config instead of switching to custom_prompt_dict, streaming tests
assert role, finish reason and body, and Nova single turn and streaming run
through litellm.completion with a recorded request body
* test: tighten migrated integration tests for local OpenAI routes, timeouts and air-gapped boot
Split the catch-all providers migration file by behavior. The fake endpoint
route tests now hit the owned upstream directly, as the legacy harness tests
did. LM Studio asserts the outbound key, path and stripped model. The embedding
timeout test asserts the proxy maps a deployment timeout to a 408 and forwards
the read timeout to the provider client, instead of timing out the test client
The Azure streaming tool-call test also asserts the second identical stream is
served from the Redis cache with the same reassembled tool call
The Docker network test always skipped in CI because the security job has no
litellm/litellm image and no egress to pull one, and it tested a published image
rather than the checkout. It now boots the checkout proxy with every outbound
HTTP connection refused and asserts readiness, the bundled cost map fallback and
that the cost map host is the only egress attempted
* test: tighten the migrated e2e rows for local_testing SDK and provider tests
Drop the duplicated Anthropic covers marker and the Anthropic cell wrongly claimed by
the Bedrock and Vertex cache tests. Keep the file-wide Anthropic model as it is on main.
Send the 2023-06-01 version header to Anthropic through the deployment's extra_headers
so the header actually reaches the provider, and mark the creation-token case 1h then
5m like the legacy test did
Reuse the Bedrock file's own registration for the pydantic response test, swap the
hand-built PNGs for the shared cat.jpg fixture, assert the SageMaker Nova image answer
names the cat and that top_logprobs=3 is honored, and register the SageMaker Nova opt-in
markers the same way as the other opt-in markers
* test: restore lost assertions in migrated test_main tests
Restore the role alternation check in test_completion_anthropic_hanging, the
embedding and streaming response-header checks, the logit_bias drop check and
the gemini system and safety settings checks. Add an azure 429 retry-after
test for test_exception_with_headers, trigger the router fallback with
mock_timeout like the legacy test, and stop the mock timeout tests sleeping 3s
* test: restore lost legacy assertions in migrated vertex ai unit tests
* test: tighten migrated cost, streaming and embedding tests and scope the hosted mock guard
* test: strengthen migrated anthropic, azure, bedrock and cohere provider tests
Assert full request bodies and usage instead of presence checks, restore the
dummy-tool and custom_llm_provider assertions the legacy tests made, add the
async and per-model variants the legacy parametrization covered, split
streamed tool arguments across deltas, and move the Azure chat streaming
tests to the azure.py mirror file
* test: tighten migrated llms provider tests and cover legacy intents lost in the move
Strengthen the migrated fireworks, gemini, mistral, together, openai text completion, openrouter, perplexity and openai error tests so they assert what the legacy tests proved (cost, provider tag, usage-only-on-last-chunk, function_call rebuild, reasoning param, sync/async/stream error fields). Replace the huggingface and jina tests that doubled litellm code with respx end-to-end tests. Add the tests the pre-existing mapped tests never covered: cohere 400 mapping, token-array text completion, hosted_vllm/llamafile/lm_studio env api base embeddings, watsonx embedding auth, ollama ssl_verify client and ollama vision prompt
* test: cover residual legacy paths for anthropic tool streams, watsonx async stream, ollama http images and caller async embedding clients
* test: restore single request assertions in the deep_infra and azure_ai embedding tests
* test: drop a no-op streaming cost flag from the openai streaming cost test
* test: cover residual legacy-test lines in caching, main, utils, replicate, vertex and bedrock invoke
* test: cover streamed audio assembly, choice membership, untranslated responses events and the stream log guard
* test: pin cost rates in unit tests, use sdk for cohere e2e, drop vertex llama e2e
* test: send the vertex maps e2e request through the openai sdk
* test: drop the local_testing jobs that select no tests after the migrations
* test: wait for the streamed cache entry and keep custom Mistral rates test-local
* test: keep untouched Mistral test lines as they were
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(guardrails): add decision_model guardrail for decisions-API models
Adds a new guardrail provider that scores request and response text with a
decisions-API model (e.g. Jev) against named predicate checks. Each check
carries its own threshold and action (block or log); a flagged block check
rejects the request with a 400 naming the flagged checks, and call failures
honor unreachable_fallback (fail_open or fail_closed).
Ships four preset checks for the Admin UI (prompt injection, jailbreak,
system prompt extraction, data exfiltration) served from
/guardrails/ui/add_guardrail_settings, plus a Decision Model step in the
add-guardrail form filtered to evaluation-mode models.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): fail closed on verdict errors and drop comments in decision_model
* feat(guardrails): cap decision_model input with head and tail elision
* fix(guardrails): list only block checks in decision_model 400s and tighten inputs
* feat(guardrails): honor timeout, cap checks at MAX_DECISION_QUESTIONS, and reject duplicate check names
* fix(guardrails): decorate decision_model apply_guardrail with log_guardrail_information
* fix(guardrails): suppress untyped-decorator pyright error on log_guardrail_information import
* feat(guardrails): screen each text in overlapping chunks with a shared decisions-call semaphore
Replace head/tail elision with per-text screening: each non-empty text is
sent to the decision model on its own, split into overlapping chunks when
it exceeds max_input_chars, and all calls run under a per-event-loop
semaphore shared across requests (max_concurrent_decision_calls,
default 8). Check verdicts aggregate the max probability across calls; a
missing predicate answer on a Block check blocks under fail_closed and
only warns under fail_open, while log-only checks are never flagged for
it. The 400 detail now lists flagged check names only. The provider
combobox also matches Decision Model on jev, typesafe, prompt injection,
jailbreak, and decision
* fix(ui): align decision model custom check rows with preset rows
* fix(ui): move decision model custom check remove button to the row end
* style(ui): run prettier on decision model check files
* fix(ui): inline decision model evaluation-mode filter after lens module removal
* fix(guardrails): keep decision model detections when a chunk call fails
* fix(ui): require an enabled decision model check before create
* fix(guardrails): screen decision model texts sequentially like presidio
* feat(guardrails): replace decision model presets with provider picker and per-question test
* fix(ui): render decision model question test as a full-width row
* feat(ui): add run-history test section to decision model form
* fix(ui): prevent threshold overlap and show test shortcut
* feat(guardrails): default decision model threshold to 0.7
* test(ui): expect decision model default threshold 0.7 in characterization payload
* feat(guardrails): screen tool call arguments with the decision model
* refactor(guardrails): freeze the decision model config model
* feat(guardrails): list only evaluation-mode deployments in the decision model picker
* feat(guardrails): hide the decision model questions table until a question exists
* style(ui): format DecisionModelFields
* style(ui): format decisionModelQuestion tests
* refactor(guardrails): type the tool call function extraction
* refactor(guardrails): build tool call texts without a mutable accumulator
* refactor(ui): share isDecisionMode for the decision model picker
* fix(guardrails): screen prior Responses function call arguments in the decision model guardrail
The Responses input handler leaves function_call arguments out of texts and tool_calls, so a prompt injection in a prior call's arguments passed unscreened. Assistant tool calls from structured messages are now screened too, deduplicated against the request's tool calls
* test(integration): send instructions with the decision model guardrail check
Checks require instructions, so creating the guardrail returned 422 and the test never sent a chat request
* fix(ui): score the decision model test preview the way the guardrail does
An unanswered block question now shows Block, disabled questions no longer count, and a run is hidden once the model or a question's text changes since it was scored
* fix(ui): clear the decision model error on pick and list questions under Add a question
Picking a model after a failed Create left "Select a decision model" on screen, because the form only validates on submit. The field now revalidates when it already shows an error. The Questions table and its heading appear only once a question exists, below the Add a question box
* test(ui): move the decision model form tests into an integration test file
They render the whole AddGuardrailForm, so they belong in a .integration.test.tsx file. Plain fields are now filled with fireEvent.change, and the two existing form test files go back to main apart from the modelHubCall mock the form now needs
* test(guardrails): build the decision model test params without mutating a local
* fix(guardrails): reject list checks outside decision_model and honor streaming scan settings
- LitellmParams now rejects a list `checks` for any guardrail other than
decision_model, so a Bedrock guardrail with list checks gets 422 again
instead of being stored and failing every request
- The decision model guardrail reads streaming_end_of_stream_only and
streaming_sampling_rate, so a streamed post_call response can be scored
once at end of stream instead of every 5 chunks
- max_concurrent_decision_calls is described as a per-worker limit
* fix(ui): match guardrail providers like the default picker filter
- The provider search uses the Base UI collator filter on the label and
the aliases, so searches such as presidio_pii or llm_as_a_judge match
again, as they did before the alias filter
- Model groups from /model_group/info are parsed defensively and after the
providers are populated, so a malformed response cannot empty the
provider list
* fix(ui): edit decision model questions in place and show passing test chips in green
Questions start as an Add question button. Each click adds a blank question
card with its name, text, action and threshold, editable until the guardrail
is created, mirroring the per-model budget editor. Create and Next reject a
blank or duplicate enabled question. Test run verdicts and chips use the
success tone for Pass, the error tone for Block and the warning tone for a
flagged Log only question
* fix(ui): drop the per-question enable checkbox from the decision model form
A question is either in the list or removed with its X, so every listed
question is sent and scored. The card now labels Name and Question the way
it labels Action and Threshold
* fix(ui): give the question name field the full card width
* fix(ui): show the provider logo on the selected decision provider
* fix(ui): show the full threshold slider track on decision model questions
* feat(ui): add decision model cards to the guardrail garden
Adds a Decision Model section to the guardrail garden with TypeSafe Jev,
Perplexity, Microsoft, OpenAI and Databricks cards. Each card opens the
create form scoped to that provider's decision models.
The create form drops the Decision Provider picker and lists every
decision model on the proxy with its provider logo, prompts to add a
model when none is available, and says that test runs are billed and
logged.
* fix(ui): shorten the decision model test hint to one line
* fix(ui): prefill the decision model from the garden card's provider
* chore(ui): regenerate the dashboard API types after merging main
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy/auth): expand all-team-models sentinel for project model access checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): use CI-served models and tighten project all-team-models denial check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): declare coverage registry cells for project all-team-models tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger e2e after ephemeral stack deploy timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): wait for team allowlist propagation before project denial check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): annotate written_at as Final per LIT010
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jesus <jesus@berri.ai>
* fix(sagemaker): keep AWS credentials out of the sagemaker_chat request body
sagemaker_chat and sagemaker_nova built the container body from every optional
param, so aws_access_key_id, aws_secret_access_key, aws_session_token and the
other aws_* auth params, plus model_id and sagemaker_base_url, were sent in the
JSON body to the endpoint container. The request is still SigV4-signed with the
same credentials, which stay in optional_params for signing
* fix(sagemaker): limit the sagemaker_chat body filter to aws_ params
model_id and sagemaker_base_url are not credentials, so they stay in the
body as before and the change sticks to what the fix is about
* test(sagemaker): assert the whole sagemaker_chat request body
* fix(sagemaker): filter the sagemaker_chat body with the shared AWS auth param list
* test(sagemaker): cover the sagemaker_chat body on every endpoint, credential source, and outage
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): streamed request bills the same SpendLogs row as the non-streamed request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): write the stream parity cells as StreamParityTestCase literals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): express stream parity cells as CostTrackingTestCase pairs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity asserts exactly one SpendLogs row per leg after a settle window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity forces a spend-log flush with a /v1/responses row instead of a settle sleep
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity covers messages -> chat and messages -> converse bridges
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stream parity bridges use xai/grok-4.7 and us.anthropic.claude-opus-5-5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): use latest frontier models in stream parity cells (gpt-5.6, claude-opus-5-5)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): switch OpenAI parity cells to gpt-6.1-sol and the chat -> responses bridge to gpt-5.5-pro
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): parse Vertex timestamps as UTC so created_at is not shifted by the host timezone
convert_vertex_datetime_to_openai_datetime parsed the trailing-Z Vertex and GCS
timestamps into a naive datetime, so .timestamp() read them as host local time
and every created_at it produced was off by the host UTC offset (about 7h ahead
on a PDT host). Attach UTC before converting, and accept the whole-second and
nanosecond forms Google's RFC 3339 timestamps also use, which used to raise.
Refs LIT-9383
* test(vertex_ai): type the UTC epoch test, guard tzset, build the rig config without mutation
* feat(sdk): add litellm.systemone and trim litellm.decisions to the OpenAI shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run tests/unit/systemone in the core-utils unit shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): keep litellm.decisions(state=...) working with a DeprecationWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make tests/unit/systemone a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): surface the litellm.decisions(state=...) deprecation as a UserWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): guardrail Bedrock Converse stream text per content block
The buffered converse-stream guardrail joined every text delta of the same
kind across all content blocks into one string, guardrailed it, then split the
result back over the frames by length. When a guardrail edit changed the
length (a mask), characters moved between blocks: text spilled from one text
block into the next, and parallel toolUse inputs became invalid JSON.
Group fragments per (contentBlockIndex, kind) so each block is guardrailed and
written back on its own.
* test(guardrails): run the Bedrock stream block tests through the real post-call hook
Drive de_anonymize_event_stream with a real ProxyLogging and pass-through logging object, with a fake guardrail at the boundary, and assert the texts the guardrail receives as well as the frames the client gets back
* test(guardrails): pin the logging start time in the Bedrock stream tests
* docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI
Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the
decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap _inlined schema walk depth for the recursion check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: gate undocumented POST routes in the proxy OpenAPI document
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: install the proxy extra before the OpenAPI docs gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop unused TypeIs import from proxy _types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only pass the JSON-RPC method to MCP admission auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): expect method-only MCP admission body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PR #45667 removed openrouter/typesafe/jev-1.13 from the cost map after #45640 moved OpenRouter decisions billing to usage.cost. The OpenAI-format wire test still read the removed row and raised KeyError on CircleCI integration-providers
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI
Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the
decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap _inlined schema walk depth for the recursion check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): move LitellmUserRoles and KeyManagementRoutes into litellm.types.proxy.auth.user_roles
Part of LIT-9423
* refactor(types): move is_jwt into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* refactor(auth): alias is_jwt import so JWTHandler.is_jwt is not read as recursive
* refactor(types): type UserAPIKeyAuth, its token view and hash_token in place
Part of LIT-9423
* refactor(types): move UserAPIKeyAuth, its token view and hash_token into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* fix(types): keep agent, budget snapshot and is_jwt names importable from proxy._types
Part of LIT-9423
* refactor(core_utils): move resource_ownership out of the proxy
Part of LIT-9423
* fix(types): restore main's _types.py content alongside the user_api_key_auth move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
* refactor(proxy): keep only user_api_key_auth re-exports in _types.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): carry main's UserAPIKeyAuth field into moved file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): move LitellmUserRoles and KeyManagementRoutes into litellm.types.proxy.auth.user_roles
Part of LIT-9423
* refactor(types): move is_jwt into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* refactor(auth): alias is_jwt import so JWTHandler.is_jwt is not read as recursive
* refactor(types): type UserAPIKeyAuth, its token view and hash_token in place
Part of LIT-9423
* refactor(types): move UserAPIKeyAuth, its token view and hash_token into litellm.types.proxy.auth.user_api_key_auth
Part of LIT-9423
* fix(types): keep agent, budget snapshot and is_jwt names importable from proxy._types
Part of LIT-9423
* fix(types): restore main's _types.py content alongside the user_api_key_auth move
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
* refactor(proxy): keep only user_api_key_auth re-exports in _types.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): carry main's UserAPIKeyAuth field into moved file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(types): drop blank lines between proxy._types re-export imports
Part of LIT-9423
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add ScaleDown model pricing
Registers the five ScaleDown models in the pricing map. ScaleDown bills on
input tokens only, so output_cost_per_token is zero for every entry. The
classify and decisions models bill at $0.04 per million input tokens, and
extract, summarize, and compress bill at $0.05 per million.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.04 per million and drop unverified context limits
Every ScaleDown model now bills $0.04 per million input tokens. The
32000-token max_input_tokens/max_output_tokens values were not confirmed
against the service, so they are removed rather than registered.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.05 per million input tokens
This is the standard rate in ScaleDown's billing and matches the usage dashboard.
The Decisions usage.cost field is not used for pricing.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add regression tests for the ScaleDown pricing entries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cite the source of the ScaleDown rate in the pricing tests and type them
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add ScaleDown as a chat provider
ScaleDown serves five models over two wire protocols. extract, summarize, and
compress speak OpenAI chat on /v1/chat/completions, while classify and
decisions implement the Jev Decisions API on /v1/scaledown, taking a state
object plus a questions map and returning one typed answer per question.
The provider authenticates with x-api-key rather than a bearer token, and the
decisions models need a non-chat request body, so it routes through the HTTP
handler instead of the OpenAI passthrough. Malformed decisions requests are
rejected locally with a message naming the problem, so callers do not have to
decode an opaque 422 from upstream. Decisions bills one model call per
question, so the cost upstream reports is passed through rather than
re-estimated from token counts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Route ScaleDown domain models to the native API and stop trusting usage.cost
The /v1/chat/completions and /v1/models routes return 403 on api.scaledown.xyz,
so extract, summarize and compress now call the native /extract,
/summarization/abstractive and /compress/raw/ endpoints. The default api_base is
the bare host; a trailing /v1 is stripped and re-added only for decisions.
The Decisions usage.cost field was about 600x the dashboard amount, so it is no
longer passed through; cost is computed from input tokens at the registered rate.
Native responses are returned unchanged as JSON, including the _value/_span_anchor
wrappers on nested extraction. They carry no output token count, so
completion_tokens is reported as 0 and documented as unmeasured.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Move ScaleDown translation into llms/ and address review findings
- Dispatch moves to llms/scaledown/chat/handler.py; main.py keeps a thin call
and get_llm_provider_logic.py no longer has a ScaleDown branch, since the
config resolves key and base URL itself.
- questions passed via extra_body are merged and validated after the shared
merge (sign_request), so the documented extra_body path works.
- extra_body can no longer change the upstream model, and decisions state may
carry a document but not text, so guardrails always see the text.
- A custom api_base requires its own api_key; the environment key is only sent
to the default host or SCALEDOWN_API_BASE.
- Image content parts become the decisions document; max_completion_tokens maps
to max_tokens; stream is served as one chunk for every model.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Refuse prompt-bearing extra_body keys and empty-key fallback to the env credential
extra_body is merged after guardrails inspect the messages, so each model now
accepts only option keys there (extract threshold/top_n, compress
compression_rate, decisions questions/state); text, instructions, prompt,
context and model are rejected. An empty api_key no longer lets the
SCALEDOWN_API_KEY from the environment go to a custom api_base.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Read compress token usage from the real response shape
/compress/raw/ nests the per-prompt counts under results and reports the total
as a top-level input_tokens; the provider read a top-level original_prompt_tokens
that is not there, so compress reported zero tokens and zero cost.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Accept the router's max_retries and word the extra_body error clearly
The proxy adds max_retries to every call, which map_openai_params rejected, so
every proxied ScaleDown request failed with a 400. It is now accepted and not
forwarded.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Return extracted fields from extract, follow schema refs, and meet the type-discipline gate
- extract content is now the extracted fields (span anchors dropped, wrappers
unwrapped), so it validates against the caller's response_format; the raw
payload stays on _hidden_params['scaledown_response'].
- Local $ref/$defs in the response_format schema are resolved.
- The adapter is rewritten with Final locals, read-only annotations, one raising
helper and a typed handler; check_type_discipline reports no violations.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Fix stream_options, schema-guided extract cleaning, document input and multi-image handling
- stream_options is accepted and not sent upstream, so include_usage streams work.
- Extract output is cleaned against the requested entities, so a field named
_value or *_span_anchor is no longer dropped.
- Extract and summarize send an image or PDF as the native document fields.
- More than one image is rejected instead of silently using the first.
- The cost test derives its rate from the cost map.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cut recursive schema refs at the cycle and cap the expanded entity count
A small response_format schema with a self-referencing $def expanded to 4^16
leaves before any request was sent. Refs that point back into themselves are now
cut off at that point, and the total expanded entities are capped at 1000 so
repeated acyclic refs are rejected with a clear error.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Bound ref chains and schema nesting so long schemas fail cleanly
Resolving a chain of thousands of distinct $refs recursed until RecursionError.
Ref resolution is now an iterative walk capped at 32 hops, and nested properties
are capped at 32 levels; both raise a clear 400 instead.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(scaledown): support native classify and preserve extraction results
* test(scaledown): include provider regressions in CI shard
* fix(scaledown): pass registry typing and bounded recursion checks
* docs(scaledown): capture provider setup before and after
* fix(scaledown): preserve nullable schemas and reject malformed requests
* fix(scaledown): isolate provider dispatch from existing providers
* fix(scaledown): preserve the shared dispatch return contract
* fix(scaledown): preserve failure logging and reject lossy schemas
* fix(scaledown): keep timeout typing annotation on its expression
* fix(scaledown): resolve escaped extraction schema references
* fix(scaledown): annotate the bounded reference walk
* chore(scaledown): remove unused loop suppression
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: moe-berri <moe@berri.ai>
* feat(otel): let a team strip captured message content from its own OTel v2 destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): split the shared callback option conflict check and sync dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): address review feedback on team capture_message_content
Classify capture_message_content as NotSecret in the credential slot registry, inject test exporters through register_exporter_factory instead of patching _exporter_from_spec, type the test helpers, and stop reassigning processor in build_tracer_provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): add integration cells for team capture_message_content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): cover weave_otel and newrelic team destinations in capture_message_content cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): stop rejecting client-sent capture_message_content at the request body boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): let a team capture_message_content override the global setting in both directions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep a team's capture_message_content on its own OTel v2 destination
The setting no longer rides the flattened team callback vars, so a sibling destination and a routed provider follow the global mode. Request bodies get the base 400 back, a failure-only entry rejects the setting, and entries for one backend must agree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): type the capture_message_content callback validation tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): type the remaining capture_message_content test signatures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): mark the new capture_message_content test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): default tenant capture to callback_settings.otel and keep CaptureMessageContent formatting bare
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep CaptureMessageContent a plain str like main
The str-to-Enum switch changed the public type: the config field's type,
repr and model_dump, a constructor that now raised on unknown values, and
unknown env values coerced to no_content. CaptureMessageContent is the plain
str subclass main ships again, the field is a str, and a
captures_span_content helper replaces the enum property. A test pins that a
published preset's capture mode stays the default for an unowned
destination, which is what main forwards.
* test(otel): drop the module-level CaptureMessageContent import the rewritten test no longer uses
* fix(lint): keep the merged branch within main's type budgets
Freeze the capture-mode settings model (LIT015) and read stored logging
entries through one typed sequence helper, so the conflicting-capture checks
add no unknown types.
* fix(otel): reject a destination capture_message_content while OTel v2 is off
The legacy OTel callbacks never read the per-destination setting, so a team
could register no_content and still export prompts and replies under the
global capture mode. Registration now refuses the var unless the proxy runs
with LITELLM_OTEL_V2=true, the same rule the Arize transport and New Relic
routing vars follow.
* test(otel): type the OTel v2 fixtures and the v2-off regression test
* fix(otel): accept only no_content and span_only on a team or key destination
OTel v2 writes captured content only to span attributes, so event_only
behaved like no_content and span_and_event like span_only on a
destination. The proxy-wide setting keeps all four values
* docs(otel): say an omitted capture setting follows the proxy's own callback for that backend
* test(otel): describe the general capture fallback in the unowned destination test
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* refactor(anthropic): move passthrough SSE stream assembly into litellm/llms
Part of LIT-9423
* refactor(anthropic): move raw SSE stream assembly helpers out of proxy guardrails
Part of LIT-9423
* refactor(anthropic): give the anthropic_sse re-exports reasoned noqa codes
Part of LIT-9423
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
* test(anthropic): cover the raw SSE helpers in stream_assembly
Part of LIT-9423
* refactor(anthropic): move prompt cache pricing helper out of the proxy package
Part of LIT-9423
* fix(sdk): resolve litellm.proxy lazily on attribute access
The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
