* test(e2e): out-of-process mock OpenAI, Anthropic, Bedrock and guardrail servers for the perf cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): coding-agent request corpus and real provider captures for the perf mocks
Request bodies come from a deterministic coding-agent session (system prompt, Read/Edit/Write/Bash/Grep/Glob tools, tool_use and tool_result turns carrying real source files picked by the request marker) in OpenAI chat, Responses and Anthropic formats. captures/ holds one real response per provider, route, shape and stream mode (gpt-6-luna, claude-sonnet-5, us.anthropic.claude-sonnet-5), cold prompt cache, body only, raw stream chunks with time offsets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): replay module for the perf mocks, captured provider responses picked by route, mode and request size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mocks replay the captured provider responses, knobs shrink to flake and speed
The OpenAI, Anthropic and Bedrock mocks answer with the captured body or stream chunks for their route, mode and nearest request size, ids swapped per request, on the captured clock scaled by the p knob. The synthetic reply builders, the per-token stream pacing and the hand-rolled Bedrock event-stream encoder are gone
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mock request log names its provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf guardrail mock remembers the request marker by call id for the response hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mock captures retaken cold, one marker and system-prompt prefix per cell
Run 2 shared one marker across cells, so every stream capture was served from the provider's prompt cache (cache_read_input_tokens up to 523946) and the json cells had small prefix hits. Every cell now carries its own marker, also prepended to the system prompt, and all 30 captures report zero cached tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf corpus reads tests/e2e files so the lane's runner image has them
* test(e2e): perf long_context json captures for Claude retaken with a 16k output cap, no longer cut mid-thinking
* test(e2e): perf mocks keep only what the replay path and the test read
* test(e2e): perf mocks interpolate the provider's time to first byte from the captures around the request size
* test(e2e): perf captures at 100k (short) and 500k (long) tokens with tools, the fixed-size set removed
* test(e2e): PerformanceTestCase schema, deterministic request plan and wire records for the perf cases
RequestShape names a corpus session (input_tokens, tools, output_tokens) and wire.request_body builds it with setup/corpus/session.py, so a 300k request is 85 file reads of real source in the endpoint's own format. The plan no longer predicts reply text, the mocks replay captured provider responses and the runner joins what the client saw against the mock's log by marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf Metric gains time_to_first_token_seconds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf User loses the unused role field, every JWT user is an internal_user
* test(e2e): perf RequestShape sizes requests from an input-token range, short and long stock shapes
* test(e2e): perf case runner (provision, Locust plan replay, observe) and the 50 rps mixed-provider case
provision.py checks the deployed config against the case and builds the callers, run.py replays the plan from a separate Locust process (locustfile.py) recording status, header, first-byte, first-token and total time per request, observe.py joins the records with what the mocks received and the workers' RSS to score every Limit, and compares each served body with the capture the mock replayed for it as a diagnostic. The case sends 15,000 requests at 50 rps over three model groups, two guardrails and the three request shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf case asserts time_to_first_token_seconds at percentile 50 and 99
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf provision creates every JWT user as internal_user
* test(e2e): perf observe reads only the mock log fields that remain
* test(e2e): 50 rps case sends short and long coding-agent sessions sized per request
* test(e2e): perf run keeps the per-request records as perf-records.jsonl under E2E_ARTIFACT_DIR
* test(e2e): the mixed small-gateway case runs at 25 rps (50 saturated one 4-worker pod on main and on v1.105.1)
* test(e2e): small_bodies_high_rate perf case, the github issue 46276 shape (11 to 17 KB chat bodies, keys only, 200 rps)
* test(e2e): the mixed small-gateway case runs at 10 rps for 6,000 requests (25 rps still saturated one pod on v1.105.1)
* ci: python lint job gets 20 minutes, the e2e harness phase (625 tests plus the replay harness suite) pushes a run past 15 when tests/e2e changes
* test(e2e): small-bodies perf case runs at 5 rps so gateway_added_seconds is per-request overhead, not queue wait
* test(e2e): mixed-provider perf case runs at 5 rps x 3,000 requests from 100 users
* test(e2e): mixed-provider perf case sends happy-path traffic only (no flaky deployment, no guardrail interventions or blocks)
* test(e2e): perf gateway model_list is three happy-path deployments, one per provider
* test(e2e): Postgres and Redis work per request as perf case limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mocks wait a case-level latency with jitter instead of replaying captured timings, cpu_seconds_per_request limit from metrics-server
* test(e2e): perf suite PR carries the small_bodies_5rps case only, the mixed-provider case moves to its own PR
* test(e2e): small_bodies_5rps limits from the ten v1.105.1 baselines (1.5x on datastore counts, 2x on latency and cpu, absolute on errors and memory)
* test(e2e): perf mocks keep only the short captures the small case reads, captured timings and the Bedrock invoke route removed
* test(e2e): perf mocks generate each reply from the request (provider schema, usage, ids, event order, chunk sizes) instead of replaying captures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): sanitize trailing slashes in provider get_complete_url
* test(llms): move trailing-slash url tests into each provider's mapped unit file
* fix(llms): strip trailing slashes in Perplexity search and Ollama embedding URLs and run the Topaz tests in CI
* fix(llms): strip trailing slashes in search, DashScope, VolcEngine, GigaChat and Gradient AI URL builders
* fix(fastcrw): strip a trailing slash from api_base and expect one slash in the GigaChat tests
* fix(gigachat): strip a trailing slash from the pass-through api_base
* test(integration): cover trailing-slash api_base across provider URL builders
Wire cells for chat, streaming, messages, responses, embeddings, search and pass-through with and without a trailing slash, an owned-proxy module for the flag-gated providers, env bases and a worker kill, two chaos bursts, and a Topaz SDK cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): price AgentCore web search at $7 per 1,000 queries
* feat(search): honor input_cost_per_query set on a search tool (#46560)
* feat(search): honor input_cost_per_query set on a search tool
* feat(search): record the search tool price on proxy-created logging objects
* chore(search): type the search tool price lookups and shard tests/unit/search
* chore(openapi): regenerate snapshot and ui types for search tool input_cost_per_query
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Kerry Lu <klu@berri.ai>
* fix(vllm): write spend log rows for /vllm passthrough calls to router models
The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call.
* ci: run the vllm passthrough unit tests in the llms-providers shard
* fix(proxy): log /vllm text completions and ask router models for stream usage
A chat stream relayed to a router model without stream_options now gets
include_usage added on the way out, and the usage chunk vLLM appends is
removed before the client sees it, so the spend row carries vLLM's own
token counts instead of a local estimate. /vllm/completions calls now
write a row too, stream and non-stream.
* fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams
Inject and strip stream usage on /vllm/completions too, hold every finish
frame for the fingerprint when n > 1, pass a forced-usage server's stream
through untouched, stop turning continuous_usage_stats alone into per-chunk
usage, close the upstream when the client hangs up so the row is written,
and assemble text choices with one sort instead of a scan per choice
* fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint
A held finish chunk now goes out as soon as any other chunk arrives, so a
choice that finishes while another still streams keeps its place. Only the
finish chunks right before the usage chunk get vLLM's system_fingerprint.
* fix(vllm): keep caller stream_options and log both success handlers
Forward a stream_options the proxy cannot read, or a non-boolean
include_usage, to vLLM as the caller sent it. Only a false
always_include_stream_usage changes the /vllm relay, so clients get the
same stream bytes under every setting.
Record the passthrough origin when logging starts so the sync and async
success handlers both parse the relayed body, hand callbacks a parsed
text completion, and read relayed messages and prompt separately so a
stray field no longer drops the local count.
Add the audit's integration tests for the relay, the settings, and the
Azure spend row.
* test(vllm): expect the caller's stream under always_include_stream_usage true
* fix(vllm): relay long SSE frames in linear time
* test(vllm): time the relay's usage and finish_reason prefilter on one large frame
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(telemetry): record proxy requests and provider attempts into the telemetry sink
* fix(telemetry): validate env settings, fix the window and only record the request's own provider calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): bind the call id from a typed local
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record a finished request into the sink current when it finishes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record every failed provider attempt, including retries and fallbacks on a shared logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): report timeouts and connection failures as their own provider status classes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record failed attempts from the per-deployment failure hook instead of a new core logging hook
Reverts the async_log_attempt_failure_event hook added to CustomLogger and
litellm_logging in 3421b38685. The router already calls
async_post_call_failure_deployment_hook once per failed attempt, retries and
fallbacks included, so every failed fallback still gets its own row without
touching the shared logging object.
* fix(telemetry): register the attempt logger only once a telemetry group is turned on
A proxy that never opts in now carries no telemetry callback. Once registered
the logger stays, returning early while every group is off.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): add litellm.telemetry records, consent groups and sink protocol
* docs(telemetry): reword TelemetryGroup docstring and drop module docstring
* fix(telemetry): add timeout and connection_error status classes, check relative imports in the package boundary test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors
Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy
modules with one pydantic validation at the point the payload enters the
module, so every value read afterwards carries a real type.
reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright,
ruff-strict or type-discipline rule is higher repo-wide or in any single
file, and no cast, ignore or suppression marker was added.
* chore(typing): replace 47 Any annotations that every caller already satisfies with object
Each annotation here is a parameter or return type whose value is only
stored, compared, logged or forwarded, so object describes it exactly and
no function body changes. A site was kept only if no basedpyright,
ruff-strict or type-discipline rule rose in any file that references it.
reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655.
* chore(typing): type upstream responses and loose annotations across providers, proxy and integrations
Second pass of the Any reduction. Decoded upstream JSON is validated once at
the provider boundary and read through typed views afterwards, annotations
that every caller already satisfies move from Any to a concrete type, and the
nine search transformations validate each result item on its own so error
text no longer carries the item index.
No cast, no type: ignore, no suppression marker and no budget file edit
* ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard
* test(e2e): rename the load suite to perf, one E2E_PERF_TESTS opt-in, and a mixed-provider Locust perf test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop the mixed-provider perf test, keep the rename and the E2E_PERF_TESTS consolidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop Greptile from the PR template, AGENTS.md and CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: require bot reviews to finish before ticking the review checkbox
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The redis-compat matrix and mcp-integration ran unit tests on five large
CircleCI machines per integration run. redis-compat now runs as a redis-py
matrix in the unit workflow with the same pinned redis-server 7.2.16 binary,
no reruns, and the redis-compat Codecov flag on the 5.3.1 leg. The MCP unit
directories not already claimed by mcp-elicitation and mcp-oauth run in a new
mcp-server shard whose report is also uploaded under the mcp-integration flag.
The Windows release wheel job installs Python from a checksum-verified
python.org installer with a bounded wait instead of Chocolatey, helm chart
testing waits on the rollout instead of sleeping, and roi-database no longer
checks out the repository twice.
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test: migrate proxy security, schema migration and MCP legacy tests
* test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp
* test: warm the baseline store before timing the locked projection
* test(e2e): hold the stack exclusively while the semantic filter flag is flipped
* test(mcp): scope the in-process proxy e2e fixtures to their module
The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite
* test(e2e): assert the semantic filter keeps listed tools instead of a ranking
Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k
* test(mcp): keep the in-process proxy e2e tests on one worker
loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out
* test(mcp): name the upstream requests that carried the wrong token
The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* ci: run every CircleCI check from the integration workflow
* test(ci): keep the startup test CircleCI-owned
* test(ui): find reset log rows by request id so concurrent workers cannot page them out
* test(ui): retry the key budget window spec until the draft refresh fix lands
* ci(ui): set Playwright workers and retries on the CircleCI command only
The shared playwright.config.ts also drives the Buildkite UI suite, where 3 workers
ran out of memory on one of two runs. Buildkite keeps main's 1 worker and 2 retries
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e
* test: fix disk cache collection, ungate saturation check, drop RSS memory tests
* test: assert one delivered log per completion in Datadog and LangSmith checks
* test: drop SDK-direct e2e copies and align cost-map path filter cases
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move guardrails, image, audio, batches, search legacy tests to unit
* test: restore the custom guardrail and remaining chunk 1 unit rows
* test: move the ocr matrix to e2e
* test: drop comments from moved tests and fix vertex ocr credentials
* test: move litellm utils and unified google legacy tests to their tiers
* test: move logging callback and pass-through unit legacy tests to their tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move store_model_in_db legacy tests and start the admin ui migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: finish the proxy job rows and drop their CircleCI jobs
Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI
* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names
* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist
* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in
* test: type the e2e beta header, prompt cache and secret manager values
* test: assert the missing model error message on the OpenAI routes
* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks
* test: make the langfuse fixture directory a package for the layout check
* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane
* test: keep the ocr e2e pdf fixture next to the test
* test: put meta below covers on the streaming tool search e2e
* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures
* test: count only OCR calls in the OCR matrix recorder
* test: pin OCR parse bodies and billed cost, group new e2e cases into classes
* ci: restore the python image anchor dropped in the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): add litellm.systemone and trim litellm.decisions to the OpenAI shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run tests/unit/systemone in the core-utils unit shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): keep litellm.decisions(state=...) working with a DeprecationWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make tests/unit/systemone a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): surface the litellm.decisions(state=...) deprecation as a UserWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI
Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the
decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap _inlined schema walk depth for the recursion check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: gate undocumented POST routes in the proxy OpenAPI document
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: install the proxy extra before the OpenAPI docs gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop unused TypeIs import from proxy _types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add ScaleDown model pricing
Registers the five ScaleDown models in the pricing map. ScaleDown bills on
input tokens only, so output_cost_per_token is zero for every entry. The
classify and decisions models bill at $0.04 per million input tokens, and
extract, summarize, and compress bill at $0.05 per million.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.04 per million and drop unverified context limits
Every ScaleDown model now bills $0.04 per million input tokens. The
32000-token max_input_tokens/max_output_tokens values were not confirmed
against the service, so they are removed rather than registered.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Price all ScaleDown models at $0.05 per million input tokens
This is the standard rate in ScaleDown's billing and matches the usage dashboard.
The Decisions usage.cost field is not used for pricing.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add regression tests for the ScaleDown pricing entries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cite the source of the ScaleDown rate in the pricing tests and type them
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Add ScaleDown as a chat provider
ScaleDown serves five models over two wire protocols. extract, summarize, and
compress speak OpenAI chat on /v1/chat/completions, while classify and
decisions implement the Jev Decisions API on /v1/scaledown, taking a state
object plus a questions map and returning one typed answer per question.
The provider authenticates with x-api-key rather than a bearer token, and the
decisions models need a non-chat request body, so it routes through the HTTP
handler instead of the OpenAI passthrough. Malformed decisions requests are
rejected locally with a message naming the problem, so callers do not have to
decode an opaque 422 from upstream. Decisions bills one model call per
question, so the cost upstream reports is passed through rather than
re-estimated from token counts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Route ScaleDown domain models to the native API and stop trusting usage.cost
The /v1/chat/completions and /v1/models routes return 403 on api.scaledown.xyz,
so extract, summarize and compress now call the native /extract,
/summarization/abstractive and /compress/raw/ endpoints. The default api_base is
the bare host; a trailing /v1 is stripped and re-added only for decisions.
The Decisions usage.cost field was about 600x the dashboard amount, so it is no
longer passed through; cost is computed from input tokens at the registered rate.
Native responses are returned unchanged as JSON, including the _value/_span_anchor
wrappers on nested extraction. They carry no output token count, so
completion_tokens is reported as 0 and documented as unmeasured.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Move ScaleDown translation into llms/ and address review findings
- Dispatch moves to llms/scaledown/chat/handler.py; main.py keeps a thin call
and get_llm_provider_logic.py no longer has a ScaleDown branch, since the
config resolves key and base URL itself.
- questions passed via extra_body are merged and validated after the shared
merge (sign_request), so the documented extra_body path works.
- extra_body can no longer change the upstream model, and decisions state may
carry a document but not text, so guardrails always see the text.
- A custom api_base requires its own api_key; the environment key is only sent
to the default host or SCALEDOWN_API_BASE.
- Image content parts become the decisions document; max_completion_tokens maps
to max_tokens; stream is served as one chunk for every model.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Refuse prompt-bearing extra_body keys and empty-key fallback to the env credential
extra_body is merged after guardrails inspect the messages, so each model now
accepts only option keys there (extract threshold/top_n, compress
compression_rate, decisions questions/state); text, instructions, prompt,
context and model are rejected. An empty api_key no longer lets the
SCALEDOWN_API_KEY from the environment go to a custom api_base.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Read compress token usage from the real response shape
/compress/raw/ nests the per-prompt counts under results and reports the total
as a top-level input_tokens; the provider read a top-level original_prompt_tokens
that is not there, so compress reported zero tokens and zero cost.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Accept the router's max_retries and word the extra_body error clearly
The proxy adds max_retries to every call, which map_openai_params rejected, so
every proxied ScaleDown request failed with a 400. It is now accepted and not
forwarded.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Return extracted fields from extract, follow schema refs, and meet the type-discipline gate
- extract content is now the extracted fields (span anchors dropped, wrappers
unwrapped), so it validates against the caller's response_format; the raw
payload stays on _hidden_params['scaledown_response'].
- Local $ref/$defs in the response_format schema are resolved.
- The adapter is rewritten with Final locals, read-only annotations, one raising
helper and a typed handler; check_type_discipline reports no violations.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Fix stream_options, schema-guided extract cleaning, document input and multi-image handling
- stream_options is accepted and not sent upstream, so include_usage streams work.
- Extract output is cleaned against the requested entities, so a field named
_value or *_span_anchor is no longer dropped.
- Extract and summarize send an image or PDF as the native document fields.
- More than one image is rejected instead of silently using the first.
- The cost test derives its rate from the cost map.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Cut recursive schema refs at the cycle and cap the expanded entity count
A small response_format schema with a self-referencing $def expanded to 4^16
leaves before any request was sent. Refs that point back into themselves are now
cut off at that point, and the total expanded entities are capped at 1000 so
repeated acyclic refs are rejected with a clear error.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* Bound ref chains and schema nesting so long schemas fail cleanly
Resolving a chain of thousands of distinct $refs recursed until RecursionError.
Ref resolution is now an iterative walk capped at 32 hops, and nested properties
are capped at 32 levels; both raise a clear 400 instead.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(scaledown): support native classify and preserve extraction results
* test(scaledown): include provider regressions in CI shard
* fix(scaledown): pass registry typing and bounded recursion checks
* docs(scaledown): capture provider setup before and after
* fix(scaledown): preserve nullable schemas and reject malformed requests
* fix(scaledown): isolate provider dispatch from existing providers
* fix(scaledown): preserve the shared dispatch return contract
* fix(scaledown): preserve failure logging and reject lossy schemas
* fix(scaledown): keep timeout typing annotation on its expression
* fix(scaledown): resolve escaped extraction schema references
* fix(scaledown): annotate the bounded reference walk
* chore(scaledown): remove unused loop suppression
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: moe-berri <moe@berri.ai>
* perf(ci): distribute unit cases and reuse security sweep connections
* revert(ci): reject first speed trial after new failures
* perf(ci): avoid repeated test discovery and parsing
* perf(ci): isolate parallel security workers
* revert(ci): restore measured security execution isolation
* perf(tests): reuse source scans and deduplicate cache cleanup
* perf(ci): restore parallel security with safe log batching
* test(ci): isolate fixture state and schema comparisons
* test(ci): make schema and scheduler fixtures deterministic
* test(ci): drain queued logging before fixture teardown
* test(ci): preserve logging delivery without idle worker restarts
* test(ci): keep guardrail sinks available during parallel bursts
* test(ci): isolate full proxy lifecycle from gateway imports
* test(ci): preserve first failure responses and sweep evidence
* test(ci): restore documentation URL settings between cases
* test(ci): isolate open MCP tools in private gateways
* test(ci): group private MCP cases and handle exited workers
* test(ci): retain proxy exits and MCP failure details
* test(ci): repair merge interactions with main
Key the schema shard's full clone on artifact-name, since main removed the
unit-flag input. Give each owned Redis its own log file again. Hand the
realtime transcription tests main added a logging object whose dispatch
handlers are awaitable, so the per-test logging drain no longer waits on a
MagicMock task forever
* perf(ci): shard owned integration cases and close lifecycle leaks
* fix(ci): qualify shard execution and harden interrupted cleanup
* fix(ci): bound worker supervision and own fault traffic
* perf(ci): reuse unparametrized fixtures across bounded case groups
* fix(ci): preserve boot fault allowances and complete exit records
* test(ci): tolerate startup retries and retain readiness details
* perf(ci): balance integration shards by recorded duration
* fix(ci): load migration database helpers only for workers
* test: move packaging and install tests to unit and integration
* test: move packaging and install checks to smoke and integration
* test: keep the boot check proxy on its scratch database in replica mode
* test: restore install imports and address review findings on packaging checks
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility
* fix: preserve HTTPX compatibility across OpenAI SDK versions
* refactor: trim OpenAI SDK compatibility patch
* refactor: simplify SDK client defaults
* test: cover SDK compatibility across provider routes
* fix: use shared names for SDK transport helpers
* fix: address SDK compatibility review feedback
* test(openai): send requests through the SDK API factory clients
* fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2
* ci(base-sdk): count httpx2 as base-only when a base dependency brings it in
OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure
* test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively
* test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10
---------
Co-authored-by: Marcus Wood <marcuswood@openai.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: delete legacy required-checks shim workflow
* ci: fold semgrep into lint and helm into unit, drop the duplicate UI build
semgrep runs the repo's own rules over the source, so it becomes lint / semgrep
behind lint passed. The helm chart tests become unit / helm behind unit passed.
UI Build Check ran the same docker build --target ui-builder as
smoke / dashboard-build, so it is deleted
* ci: merge the alphabetical llms shards into one llms-providers shard
The two shards were split by letter range, which says nothing about what they
run. Every provider without its own shard now runs in llms-providers, with the
same 137 paths as before
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange
* fix(ui): create a credential for credential-only auth types in Add Model
* fix(microsoft_365_copilot): accept max_tokens and list the chat model
* fix(microsoft_365_copilot): register provider model set
* fix(ui): hide other auth types' fields when credential-only types are filtered
* fix(health): do not inherit stored auth settings when a connection test brings its own api_key
* docs(health): restore connection-test inheritance docstrings
* fix(lint): suppress justified provider boundary casts
* fix(lint): satisfy M365 type-discipline gate
* fix(types): eliminate type-check gate regressions
* style: shorten suppression reasons so ruff format leaves them on one line
* refactor(types): drop shared-file type widening unrelated to the Copilot provider
* fix(health): pass caller headers as secret_fields to connection-test probes
* fix(types): type connection-test probe and token counter locally
* fix(lint): drop cast import from connection-test header getter
* chore: remove explanatory comments flagged by review
* fix(proxy): gate OAuth client credentials behind proxy admin
* fix(router): skip cooldown on caller-scoped OAuth auth failures
* fix(router): skip fallback cooldown on caller-scoped OAuth auth failures
* fix(router): type caller-scoped cooldown lookups
* fix(ci): include Microsoft 365 Copilot tests in unit shard
* fix(proxy): keep saved endpoint when test_connection retests a deployment by id
* fix(m365): collapse doubled Graph replies
* fix(m365): preserve trailing system messages
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(greptile): load nested AGENTS.md files as review context scoped to their directories
* refactor(greptile): use a stdlib dataclass so the files.json check runs without uv
* test(greptile): write fixture AGENTS.md files through a helper instead of rebinding a loop variable
* test: move offline router, latency-routing, retry and exception tests from local_testing into tests/unit
Moves 29 offline nodes into the tests/unit files that mirror the code they exercise.
Hosted-provider calls are mocked at the HTTP boundary with respx, and sleeps that only
built a start/end gap are replaced by explicit timestamps. Legacy files left empty are deleted.
* ci: record the live prompt caching cases left in tests/local_testing as unrun
Moving the router prompt caching test to tests/unit leaves only live Anthropic and Vertex cases in tests/local_testing/test_anthropic_prompt_caching.py, which every CircleCI -k already deselects
* test: freeze the clock in the moved lowest-latency routing tests
The latency logger buckets usage by the current minute, so a run that crosses a minute boundary could grow the cache and fail the memory check. The moved tests now use fixed timestamps and a frozen clock in the logger module
* ci: describe the mixed prompt caching file accurately in the coverage allowlist
Restrict 26 build_and_test jobs to main with job-level branch filters, delete litellm_router_unit_testing in favor of a router-unit-tests GHA shard, and narrow guardrails_testing to the license-dependent test while the rest of tests/guardrails_tests runs in a guardrails-tests GHA shard
Co-authored-by: yuneng <yuneng@berri.ai>
* ci: restructure lint, unit and smoke workflows
* ci: preserve source formatting in tier workflows
* ci: retire per-shard coverage flags and tighten the tier workflow guards
* test(ci): freeze the workflow startup safety models
* ci: keep the current required check names running until the ruleset moves to the tier collectors
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* ci(unit): add unit passed collector job and fold proxy-db shards into test-unit.yml
* test(ci): cover the unit passed gate's success, failure, cancelled and skipped results
* test(ci): run the unit passed gate test from the code quality workflow
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(mcp): support upstream OAuth client metadata identities
* fix(mcp): load client metadata on cold OAuth requests
* fix(mcp): discover client metadata with configured OAuth endpoints
* fix(mcp): preserve configured endpoints when discovery fails
* fix(mcp): retain CIMD identities across refresh and catalog reloads
* fix(mcp): reuse saved CIMD identity for token endpoint refresh
* fix(mcp): keep dynamic client registration ahead of CIMD unless the deployment opts in
A provider that advertises both a registration endpoint and client ID
metadata documents now gets the registration flow the gateway used
before, so a gateway on a private network keeps working against it;
the metadata document identity applies when the provider offers no
registration, or when general_settings.mcp_prefer_client_id_metadata_document
is true. The saved CIMD refresh identity compares tokens as bytes so a
non-ASCII refresh token cannot crash the token route.
* chore(ui): regenerate dashboard API types for the new MCP general setting
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs
* test: restore lost coverage in litellm_utils offline replacements
Bound live health-check tasks to the concurrency limit so eager task
creation behind a semaphore fails, route the Langfuse trace-id check
through completion -> Logging.get_trace_id with all four metadata
combinations, run the callback dedup checks through acompletion, pass
the dynamic key via litellm_params, cover the env-inferred default
model list, and add the per-provider audio transcription config lookup
* test: restore router coverage lost in the offline move
Add the missing non-stream LIT-3058 header/count-once test, pin the UTC minute on every
usage-counter test, assert router-level client reuse for transcription, and replace the
private selector-attribute checks with routing outcomes per strategy. Assert outbound bodies
for speech, rerank, image, assistants and moderation, add timeouts to event waits, and drop
the router_unit_tests husk files that no longer hold tests
* test: restore sync stream, router sync stream, error event and field-type coverage for migrated responses tests
Add the sync streaming logging and sync Router.responses streaming cases the
offline replacements dropped, port the legacy per-event and response field-type
validation, assert raw headers, the float created_at conversion, MCP tool headers,
the search_context_size input, and add an offline replacement for the in-stream
context-window error event. Remove helpers left dead in the legacy file.
* test: run numpydoc-backed unit tests in GHA and drop a test that pinned a crash
* test: restore sequence number, item id and content part checks in the responses stream validator
* test: match router logging events to the test's own deployment so late events from other tests are ignored
* test: restore the s3 cold storage history test to its original assertions
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* ci(lint): derive gate ceilings from merge-base counts and drop the budget files
The four lint gates (ruff strict, type discipline, basedpyright, test quality)
now fail a branch only when a rule's codebase count grows past its count at the
merge-base with litellm_internal_staging plus a fixed per-rule headroom, which
is zero everywhere except the LIT010/LIT011 and reportAny/reportExplicitAny
seeds. Base counts come from a disk cache, then the CI artifact the renamed
publish-lint-base-counts workflow uploads for every staging push (all four
checkers, one artifact per checker and sha), then a scan of the base worktree.
The four *-budget.json files, make lint-budget-update, budget_ratchet_check.py,
and the unratcheted check are gone, so no PR carries a budget edit again.
* test(lint-gates): assert checker identities by behavior and type the new gate tests
* test(lint-gates): resolve base points over an injected git so unit tests stay in-process
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(lens): isolate trace storage and investigation in a Rust service
* fix(lens): include Rust sources in the image build context
* feat(lens): wire service setup, scoped delivery receipts and lease attempts
* fix(lens): complete service routing and reject stale investigation results
* fix(lens): retry key propagation and validate isolated Compose setup
* fix(lens): seed through isolated ingestion and preserve upstream queue fixes
* chore: sync schema.prisma copies from root
* fix(lens): bind nullable due timestamps as text for Prisma
* chore(ui): remove stale lint suppressions
* fix(lens): address CI failures and review findings
* refactor(lens): remove retired Python worker and run evaluations in Rust
* fix(lens): reuse control connections and satisfy review checks
* test(lens): install and upgrade both Helm charts on Kubernetes
* test(lens): run connection reuse coverage as an integration test
* fix(ui): upgrade Next.js to 16.3.8 security release
* fix(lens): fence stale attempts and preserve reviewed evidence
* Revert "fix(ui): upgrade Next.js to 16.3.8 security release"
This reverts commit 2f79a51b25.
* fix(lens): stop failed investigations and stream history excerpts
* test(lens): cover model tool and result contracts
* test(lens): fix retired routes and reuse installation build artifacts
* test(lens): use portable grep in Helm installation smoke
* test(lens): wait for migrations before forwarding Helm services
* fix(lens): keep failed evidence reads retryable
* fix(lens): preserve sandbox output during process exit
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* test(e2e): move the harness self-tests out of tests/e2e
The nightly Buildkite run copies tests/e2e into the runner image and runs
bare pytest, so the 672 tests of the harness itself (fixture parsing, JUnit
properties, the stack lock, the load aggregators, the Claude Code driver)
counted as e2e tests on the status page even though none of them reaches a
proxy. They now live in tests/e2e_harness, mirroring the tests/e2e layout,
and run in the GitHub Actions lint job and the CircleCI
provider_replay_harness job instead
* fix(ci): point the providers replay controls at tests/e2e_harness
The providers integration job still selected the four replay-control
tests under tests/e2e/test_provider_edge.py, so pytest exited before
they ran. The raw-HTTP check's file walk also drops to one loop per
comprehension
* style(tests): mark the raw-HTTP check's bindings Final
* fix(lens): use async-timeout on python 3.10 for budget reservation timeouts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): keep uv.lock diff to the async-timeout entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(lens): cover real request deadline expiry in reserved_budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): finish Python 3.10 timeout coverage and dependency checks
* test(lens): control event-loop time for deadline regressions
* test(tracing): include priced call count in trace fixture
* fix(lens): limit timeout compatibility changes to PR scope
---------
Co-authored-by: Moe Khalil <moe@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
