46 commits

Author	SHA1	Message	Date
Kerry Lu	081cfa2a62	fix(proxy): reset end-user spend in committed pages so the budget cascade survives a large LiteLLM_EndUserTable (#46502) ... * fix(proxy): reset end-user spend in committed pages so the budget cascade survives large LiteLLM_EndUserTable Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): collapse _enduser_ids_by_tier return to one line for ruff format Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): mark the end-user page IN list as bounded for the unbounded-IN check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 19:47:12 -07:00
rameswaran-mohan	e65e5abc4e	feat(proxy): name the successor model for deprecating deployments (#41670) ... * feat(proxy): name the successor model for deprecating deployments * chore(ui): add successor_model to the generated deprecation api types * fix(proxy): keep the cost map successor when a deployment overrides only the date * fix(proxy): read the cost map successor from metadata.successor The cost map schema rejects a top-level successor_model key, and curated entries already carry the successor under metadata.successor. Read that key, and look it up even when the cost map entry has no deprecation date, so a deployment that sets only its own date still names the successor. * fix(proxy): keep the first deployment when deduping, let a blank successor hide the cost map one, and name a served successor by its model group * fix(proxy): name only the deployment's own successor_model, since cost map successors are LiteLLM ids rather than served model names * fix(proxy): leave out a successor_model the deployment's callers cannot call, with a warning * fix(proxy): report successor_model as the admin set it and only warn when no model matches it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 14:50:14 -07:00
yucheng-berri	d9b5d9e19f	Revert "feat(proxy): add LITELLM_FIPS_MODE startup gate with provider assertion and loud password migration failure (#42700)" (#46740) ... This reverts commit 1238cfe90f. Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-15 12:39:14 -07:00
devin-ai-integration[bot]	30ada9d26a	feat(batches): store each batch line's request and response in S3 and GCS (#41691) ... * feat(batches): emit per-line JSONL batch records to callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): only emit line items for final batches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(batches): satisfy ruff format for line item logging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): cover line-item edge shapes and general settings propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): keep deployment credentials out of callback-visible params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(batches): declare credential transport attribute on Logging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): integration contract for per-line batch callback events Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): assert per-line events beside the aggregate and create events Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): assert line item events carry no credentials Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): reset batch line-item flag when the setting is absent Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): map anthropic batch lines through the anthropic message transformer Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: tag the anthropic result cast with a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: shorten cast-ok reason so ruff format passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): skip line-item callbacks for providers without a line reconstruction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: wrap widened provider literals so ruff format passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): pair bedrock lines by recordId and type them from the reconstructed response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): keep line item events out of proxy limiters and strip credentials from child params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): skip failed line item events in parallel request limiter failure hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): tolerate lifecycle spend rows when asserting no line item rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): place mutable-ok tags on the lines the discipline gate flags Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): skip router budget accounting for batch line item events Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): keep anthropic line errors and skip native vertex batch rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): restore test-quality tags on line item tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move batch line-item unit test to tests/unit after merging main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): emit batch line-item callbacks once per batch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): release line-item claim when fan-out emitted nothing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): make the line-item claim release ownership-safe Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): keep bodyless failed batch lines reaching callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): seed batch line hidden params in get_hidden_params defaults Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(batches): reuse fetched batch result files for line-item logging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): release line-item claim when nothing was emitted Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): type batch line-item callback sites to satisfy basedpyright budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: wrap batch hidden-params cast for ruff format Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): reason suppressions for batch line-item lint deltas Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop stale mutable-ok suppressions after LIT002 removal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486) * fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id) bypassed the call-type-only batch guards: - ClickHouseSpendLogger wrote one spend row per line on top of the aggregate aretrieve_batch row, billing every batch twice with the flag on. - Router deployment callbacks and the routing-strategy usage handlers charged the historical batch's tokens/rpm to the live TPM/RPM counters, so after Router.aretrieve_batch the next chat raised RouterRateLimitError. Skip these events wherever the aggregate batch retrieve is already skipped. * fix(logging): keep internal litellm_params authoritative in model_call_details A request body key named litellm_params lands in optional_params, and update_environment_variables merged **optional_params after the internal entry, so a caller could overwrite the logging kwargs' litellm_params and forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose body carried {"litellm_params": {"batch_parent_id": ...}} returned 200 with no ClickHouse spend row. Re-assert the internal litellm_params after the merge so every sink (ClickHouse, Postgres, router quota counters) reads the same trusted value, and assert the aggregate aretrieve_batch row still writes in the ClickHouse line-item test (review finding). * fix(clickhouse): skip failed batch line items in the spend sink too A failed JSONL line is part of the aggregate aretrieve_batch row's request counts; writing it its own spend_logs row inflated request counts with the line-item flag on. Extend the batch line-item guard to the failure handler (review finding) and pin both directions: a failed line item writes no row, a genuine live failure still does. * fix(batches): declare batch_line_items service family on claim calls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(integrations): keep batch line items out of the built-in metering sinks With store_batch_line_items_in_callbacks on, per-line batch events carry a real response_cost and reached six metering sinks that lacked the is_batch_line_item_event guard, so a completed batch metered aggregate + per-line (roughly 2x true spend): prometheus (litellm_spend_metric and request counters), openmeter, lago, datadog cost management (FOCUS BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost metrics. Skip line-item events in each sink's success/failure metering entry points, matching the guards already on the Postgres and ClickHouse spend sinks. OTel spans for line items are still emitted; only their cost/token metrics are skipped. Tests drive real sink instances on the real dispatch lists through a completed 2-line batch and assert each sink meters exactly the aggregate. * style: format metering-sinks test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test Follow-ups from review on the sink guards: record_failure() now skips batch line items too (a failed line's interval measures retrieval and emission, not its model call), and the prometheus guard test snapshots and restores the process-wide registry collectors instead of leaving them unregistered. The OTel v2 test is now a flag on/off A/B over the whole metric snapshot: cost and usage values must match exactly and no metric may gain datapoints with the flag on. * chore: finish main merge conflict resolution for batch line items Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: sort imports in litellm_logging after main merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(batches): use the public hook class names main now exports * fix(batches): only the batch writer can mark a callback event as a line item amoderation, transcription and speech build litellm_params from every request kwarg, so a body field named batch_parent_id reached Logging.litellm_params and every spend, budget and rate-limit hook skipped the request. update_environment_variables now drops an incoming batch_parent_id; the line-item writer sets it through Logging.mark_batch_line_item. Batch line hidden params are emitted only when set. * chore(batches): keep the lint gates within main's ceilings after the merge Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the banned cast with a literal match, dropped isinstance checks main made redundant, and moved test-quality reasons onto the lines the gate reads. * fix(anthropic): read the message id off the validated message The anthropic_messages logging path can hand over an already parsed message object, which has no .get; main read its id through getattr, so take it from the validated AnthropicResponse instead. * fix(batches): an unreadable error file no longer drops the output line events Costing counts an error file it could not fetch as no failed lines; line logging refetched it and the raise aborted the whole fan-out, so the successful lines were never emitted. The error file now reads as empty with a warning. * test(batches): pin billing for batch_parent_id on every endpoint and value shape Chat completions, embeddings, responses and messages join moderations, speech and transcriptions, and speech is driven with an empty, int, list, object and 5 KB value. * test(batches): bill every request of a concurrent burst mixing forged and plain bodies * revert: keep the line-item fan-out all-or-nothing on an unreadable error file 85fc4498cc emitted the output lines and kept the batch claim when the error file could not be read, so its failed lines were never retried. Raising releases the claim and the next retrieve retries the whole fan-out. * test(batches): pin the all-or-nothing retry and warm the burst model first An unreadable error file emits no lines and the next retrieve emits all of them. The burst cell drives one speech model after a warm-up request so a cold worker cannot fail it, and waits for the batched key spend. * fix(batches): run the line fan-out outside the aggregate event's deadline The logging worker cancels a callback coroutine after its per-callback deadline, and the line fan-out ran inside the aggregate aretrieve_batch event before its callbacks, so a large batch or a slow sink could cancel the aggregate billing event. The fan-out now runs in its own task, built from a copy of the parent's litellm_params taken before the aggregate callbacks write their request-local markers. Anthropic keeps no input file and reports input_file_id "None", which the fan-out fetched and failed on, so no Anthropic line ever arrived. That id is now treated as no input file. * fix(batches): let running line fan-outs finish before proxy shutdown closes their clients A started fan-out keeps its batch claim, so a shutdown that cut it short skipped the unsent lines until the claim expired. proxy_shutdown_event now waits up to MAX_TIME_TO_CLEAR_QUEUE for them first. Also replace a test's type: ignore with an isinstance check. * fix(batches): keep batch line fields off callback kwargs and skip fan-outs at shutdown Line bodies were spread into the callback kwargs, so a line's metadata could settle router rate limit reservations; they now stay in optional_params, and the router rate limit and prompt caching hooks skip line events. A retrieve logged after the shutdown drain, or by the logging worker's exit flush, no longer starts a fan-out its loop would destroy while it holds the batch claim. The output file is fetched before the error file again. * fix(batches): log batch lines under the retrieve's callback settings and keep them out of alerting and shadow eval Line events inherit the retrieve request's key and team callback settings, so turn_off_message_logging and per-team logging credentials apply to them, and they keep the system prompt in the logged messages. Shadow eval, Datadog metrics, Slack daily reports and PagerDuty skip line events, since the batch API served them and the aggregate event already counts the batch. * test(batches): line events follow the key's message logging opt-out * test(batches): line events follow team logging settings, keep the line system prompt and survive a restart once * fix(batches): keep in-memory line claims for 10,000 batches and pin managed batch emission across poller and retrieves * Keep batch line unit tests in-process and drop repeated per-sink comments * Share one batch provider alias across batch_utils and line item logging * Report Bedrock line status codes, skip unknown Bedrock lines without raising, fetch the input file after the native Vertex check * Read a swallowed Redis claim failure as unavailable, price unpriceable lines at zero, keep the OTel failure docstring * Call handle_completed_batch when line items are off, keep a cache test from leaking Redis into the line claim cache, pin an unpriceable line's zero cost on the callback kwargs * Hand the cost poller's credential snapshot to line item file reads, so a managed Bedrock batch's lines read its input from the deployment bucket * Log batch lines through the sync callbacks a direct async call runs, keep failed lines out of router cooldown counts, and emit an error line whose body is text * fix(batches): log failed Bedrock embedding lines as embeddings with their input A Bedrock batch input record carries no url, so a failed Titan v2 line fell back to acompletion and its inputText never reached the event messages. The failed line now takes aembedding from the deployment model and the record's inputText is the line input * fix(batches): keep a Bedrock line's system prompt in its logged messages Bedrock batch input records hold the system prompt as content blocks, which logging only reads off the kwargs as text, so Claude and Nova lines dropped it * feat(batches): store per-line batch requests and responses in S3/GCS via batch_line_item_callbacks Replace the per-line logging path with a dedicated hook. When the hourly batch cost poller finishes a managed batch, it pairs every output and error line with its input line, prices each line, and hands one payload to every logger listed in litellm_settings.batch_line_item_callbacks. s3_v2 and gcs_bucket write it as one NDJSON object per batch under batch_line_items/<day>/<batch id>.jsonl. Bodies are dropped and the header kept when the creator key or team, the global setting, or the logger itself turns off message logging, and when the creator can no longer be found. A failed upload is logged and never blocks spend tracking. * fix(batches): store batch line items after the batch is billed and retired Send a billed batch's lines only after its row is marked complete, so a slow or failing sink never holds a claimed batch open. Skip reading the input file when the bodies are dropped anyway, still send the error lines of a batch whose output file is gone, and end every NDJSON record with a newline. * test(batches): cover team-level and deleted-key redaction of batch line items * fix(batches): store batch line items in the background so a hung bucket never holds up billing The poller now retires a batch first and hands its line items to a per-worker background store that runs one batch at a time. A slow, hung or failing bucket no longer delays billing or retiring the other batches in the poll cycle. While a poll cycle's worth of batches is still waiting to store, the next batch's lines are dropped with an error. * fix(batches): store an unbilled batch's line items only once it is retired When retiring an expired, cancelled or all-failed batch fails, the batch stays on the poll page, so its lines are no longer read and stored again on every poll cycle until it is retired. The poller also releases a billed batch's output bytes before reading the next batch's output, as it did before line items existed. * fix(batches): price and write batch line items off the event loop Pricing every line of a large batch and serializing its object ran on the proxy's event loop, so a 50k-line batch stalled the worker for seconds. Both now run in a worker thread. An object the bucket refuses for good is no longer kept in s3_v2's drop marks, which only a log-queue flush clears. Vertex AI input rows carry the custom_id in their key or request labels, so their lines now get the request too. * fix(batches): pair no request for a shared custom_id and wait for an unclaimed batch's retry A Vertex AI embeddings entry whose input is a list is split into one row per item, all with the same custom_id, so its line now gets no request instead of the last row. On a database without batch_processed nothing claims a billed batch, so when marking it complete fails it stays in the poll page and its lines now wait for the next poll instead of being written on every cycle. * fix(batches): store batch line items only once the poll has billed every batch Pricing a large batch's lines on a worker thread still competes with the poll for the GIL, so the next batches in the same poll were billed and retired seconds later. The background stores now wait until the poll cycle that queued them has finished. * fix(batches): hash line-item objects off the event loop and keep lines with a lone surrogate A batch's whole line-items object was hashed for its S3 PUT on the event loop, and a line holding a lone surrogate from a provider's JSON escapes made the object impossible to encode as UTF-8, so the batch's object was never written. The object is now prepared in a worker thread, and only the line that cannot be encoded is written with escaped characters. * fix(batches): keep request and response for batches created without a virtual key A batch created with a JWT has no API key on its row, and its line items were stored without request and response as if its creator had been deleted. Such a batch now follows its team's and the proxy's message logging settings, the same as a master key batch. A deleted key or team or a failed lookup still drops the bodies. * fix(batches): pair line items with the right request on every deployment and skip empty objects A batch retired with nothing billable now reads its input file by the id the provider reported, so a batch that ran on the second deployment of a model group keeps each line's request. A Vertex AI line read as custom_id "unknown" pairs with no input row, and a batch whose files hold no readable line writes no object * fix(batches): let OAuth2 batches follow team and proxy message logging settings The line-item redaction check looked up every saved creator credential other than the master key or a CLI session as a virtual key. An OAuth2 batch saves the raw token, so it read like a deleted key and lost its request and response It now looks up only a saved value shaped like a virtual key's hash, reusing is_valid_sha256_hash, so OAuth2 batches follow their team's logging settings and the proxy-wide setting while deleted keys still fail closed * fix(batches): hash the creator credential and write batch line items in parts The creator credential reaches line metadata only as a sha256 hash, so an OAuth2 token or the master key alias is never written to the bucket. A queued store keeps only whether the batch was billed and downloads the output file again at its turn, logging an error naming the batch if that fails. Lines are built and uploaded one part at a time, each closing once its raw request and result bytes reach BATCH_LINE_ITEMS_MAX_PART_BYTES (16 MiB by default), with the input file indexed by byte span instead of parsed up front. * docs(batches): say async_log_batch_line_items is called once per part * fix(batches): log an error when a batch's deployment is gone before its line items are stored --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng <yucheng@berri.ai>	2026-10-15 11:32:56 -07:00
devin-ai-integration[bot]	fbc2e358bc	fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703) ... * fix(responses): bill a background response once, from the poll job Reading a completed background response with GET /v1/responses/{id} wrote a full-cost spend row on every read, on top of the row the background cost poll writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and the poll job is the single biller. The poll job now bills with the creating request's attribution (key hash, team, org, user, aliases, tags) read off the managed object row, through a helper it shares with the batch cost poll. The background create persists the creating key and request tags so that attribution survives until the poll runs. * test(proxy): a background response read reports zero cost * fix(responses): bill a background response once, on the first read or poll that sees it finished A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll * fix(responses): keep the caller's response id out of the billing failure warning * fix(responses): keep the creating client's User-Agent tags on the one background charge * fix(responses): keep managed background reads free and bill each response once The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404. * fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts * fix(responses): give the background billing event its own spend log row and bill responses the create left untracked * fix(responses): narrow the managed files hook with a protocol instead of an enterprise import * fix(responses): leave a read the caller retargets with client-side credentials to the cost poll * fix(responses): bill background responses on the creator's end user and report the charge on the settling read The read that settles the bill returns the charge in x-litellm-response-cost, the billed event carries the creator's end user and the key, team, and user budget fields, and its spend row is named after the id the client got. Override reads, cancels, and deletes settle from the configured deployment, a streamed create marks its row billed, stored rows keep no model output, and model insights counts the billed row's spend without a request. * test(responses): cover background response billing across proxies, reads and chaos * fix(responses): settle background billing only for callers allowed to address the response A delete now settles the bill only after the response id access check passes, a streamed create is marked billed when it sets background inside extra_body or its completed event arrives after the last chunk, the billing event keeps the deployment's litellm_params base_model, a key with no user and no team bills no user instead of its owner scope, and the enterprise cost check import is optional * fix(responses): record the bound on the pending status IN filters * fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was The read that settles a background response now sends the whole cost header family from the billed event, so the breakdown headers add up to the charge. Batch cost events go back to main's attribution (creator id as the user, no budget fields); only background responses get the end user and budgets. Integration tests expect the charge on the settling read (or none when the poll billed first), one charge for a streamed create, and SDK flows that poll with the id each read returns. * fix(responses): bill each background response once across upgrades, streams, and failed settles Rows stored before this billing model are found by the issued id and rekeyed by the provider id, and the cost poll retires one a read already billed. A streamed background create is free on its own and bills the finished response through the same managed row before the client sees response.completed, so a racing read cannot bill it again. A settle that fails retries in the background, a retargeted read with no row tracks and bills from the configured deployment, and a delete keeps its billed cost header. Reverts the unrelated json.dumps narrowing in the completion transformation. * fix(responses): bill cache-polled background streams and match poll rows by the issued id * test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing Adds integration cells for the background response billing paths the audit inventory lists: a DELETE that settles the bill and refuses a stranger, a settle the database refuses that retries once it recovers, the Prometheus remaining budget, base_model pricing from model_info and litellm_params, the daily model usage rollup, the end user, the settling read's cost split, stored rows without model output, a forged managed marker on the create, streamed creates with background in extra_body, and the cache-polled create --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 11:00:14 -07:00
Mateo Wang	622daf3cdf	fix(scim): write a group's roster in one locked transaction instead of one member at a time (#45197) ... * fix(scim): write a group's roster in one locked transaction instead of one member at a time POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the membership rows and the users' team tags in one transaction under the team's advisory lock, so a 500-member push runs a fixed number of statements instead of thousands. PUT now answers with the roster it just wrote * test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers * fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock * fix(scim): evict the cached team on a group rename and read a no-op roster from the writer A rename-only PUT or PATCH wrote the new alias and left the team cached under its id and old alias on every worker, so keys kept logging spend against the old team alias until the TTL expired. The group write now evicts the cached team (and broadcasts the eviction) right after the alias or metadata update. sync_team_roster reads the team from the writer when the plan names nobody, so a replica cannot hand back the pre-rename row; a plan already matching the roster returns inside the lock without writing, evicting, or emitting the members metric; the response is built from the written row, so lastModified reflects the roster write; and detaching a removed member bumps the user's updated_at like every other user write. * fix(scim): bump detached members' updated_at in UTC The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts into Prisma's naive timestamp column in the session timezone, so on a database whose session is not UTC the removed member's row landed hours away from the UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and have the bulk roster cell assert the bumped row sits between two reads of the database's own UTC clock taken around the replace. * fix(scim): chunk the group write's member lookups under the bind cap Every Prisma "in" filter the group write issues bound one parameter per member with nothing bounding the list, which check_unbounded_in_lists reports as 11 new findings. The single-field lookups now go through litellm.repositories.chunked_in (caller_transaction inside the roster transaction, per_chunk_ok for the role recompute), and the one cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two slices, and a member named only in the last slice still resolves. * fix(proxy): send bulk cache evictions in slices so the broadcaster drops none Veria pointed out that the invalidation publisher caps its pending tasks at 1,024 and drops the excess with a warning, and that a group push now detaches every removed member's keys in one call instead of one member at a time, so a removal of more than roughly 340 members (three cache keys each) silently left stale key objects on the other replicas. The merge base never hit the cap because each member was a separate request. Bulk evictions (team, user, organization, and session key deletes, plus the roster settle) now publish in slices of 256 and wait, bounded at two seconds, for the backlog already handed to the event loop before each slice. A Redis that never answers costs a bulk caller at most that wait per slice instead of wedging the request, and single-key callers are untouched. Regression tests push 1,500 keys and a 500-member removal through a slow publisher and assert every eviction reaches Redis; both fail against the previous tip. * fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts Greptile pointed out that pacing the broadcasts in slices also paced the local eviction: a bulk delete waited for the publish backlog before dropping the next slice from this worker's own memory, so on a Redis that accepts connections but never answers a key deleted from the database kept authenticating on the handling worker for up to two seconds per slice. The merge base cleared every key locally at once. Bulk evictions now drop every affected entry from this worker's memory first, through the cache's own partition lookup since key objects live apart from the other auth objects, and only then publish in paced slices. The roster settle does the same for the deleted tokens, the JWT mapping keys and every changed member's user and membership entries before any step that can wait. Regression tests run both paths against a publisher that never answers and check that nothing is still cached on this worker before the first bound elapses; both fail with the local eviction removed. * fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts * docs(proxy): drop the module docstring that repeated the PR description * fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps Bulk key evictions now drop entries from memory, delete them from Redis in one chunked DEL, and broadcast every key in one pipelined publish, so a large roster removal no longer overflows the publisher's pending cap or costs a Redis round trip per member. A push waits for in-flight membership loads before evicting. User rows are locked in byte order to match the spend writer, a bulk user delete takes every team lock before its first user write, and a POST whose roster write fails deletes the empty team so the IdP retry creates it again. * test(team): observe delete broadcasts at the Redis layer * fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently * fix(team): chunk the email lookup behind the member row lock * fix(scim): create a member deleted mid-push again instead of failing the group write * fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch A member deleted mid-push is created again only when scim_upsert_user allows it, under the verb's own created_via. New group members are checked against the license seat limit as one batch before any is created. Auth cache publish drops log the key count instead of the key. * fix(scim): lock member rows while recomputing admin roles after a group push * fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member * fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member * fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time. The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team. A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup. * fix(user-delete): keep the team alias label on the team members gauge after removals * test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them * fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404 * fix(scim): answer 404 and roll back when the team is deleted during a roster write * test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast	2026-10-15 10:57:00 -07:00
Hayden Moulds	de0ed1557e	fix(proxy): resolve model aliases to their target deployment for listing metadata (#41041) ... * fix(proxy): resolve model aliases to their target deployment for listing metadata A model alias (a key's `aliases`, a team's `model_aliases`, or the router's `model_group_alias`) is rewritten at request time and owns no deployment row, so `get_model_listing_info` returns None for it and the alias name is not a cost-map key either. Both metadata candidate sources come up empty and the listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`, while the deployment it routes to reports them on the same response. Resolve a listed alias to the deployment it routes to for the metadata lookup, keeping the alias as the response id. Maps are applied in order, each against the result of the last, mirroring how `litellm_pre_call_utils` applies the team map and then the key map, so a chained alias reports the deployment the caller actually reaches rather than an intermediate one. An alias shadowing a same-named deployment follows the alias, because the request rewrite does too. Fixes the model_aliases half of #40553. * fix(proxy): look listed aliases up through the shared alias chain Drop the separate alias resolver and retarget a listed alias in alias_listing_entries and alias_target instead, so /v1/models and /v1/models/{id} report the model /chat/completions rewrites it to, and only when the caller can list that model * fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval * fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing * fix(proxy): leave an alias to an undiscoverable model bare in model listing A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them. * fix(proxy): leave an alias to a model a listing callback hides bare in model listing * fix(proxy): keep aliases bare for models a listing callback never judged or another team owns --------- Co-authored-by: haydster7 <haydster7@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 00:38:50 -07:00
devin-ai-integration[bot]	e136e91273	fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) ... * fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise. * fix(router): give prompt cache pins their own tier and honour any declared ttl Pins now live in a DualCache of their own instead of the router's 200-slot in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows. The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not only the literal "1h". * fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache Compute the cache_control ttl scan through offload_token_count beside the prefix hash, clear the dedicated pin cache in Router.flush_cache, and point the order-fallback and priority-group pin tests at the pin cache they read * fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots * fix(router): match the 1h pin gate to what Bedrock forwards Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock, the same check that decides whether the 1h ttl reaches Bedrock, so an application inference profile ARN keeps a 5 minute pin whatever its base_model prices * test(router): give the anthropic_messages pin fixture its provider * fix(router): let each pin write set its own lifetime and require a positive 1h price In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support. * test(router): integration cells for the one-hour prompt cache pin lifetime Cover every surface the pin lifetime touches: chat and /v1/messages through both SDKs, the responses bridge, response-cache hits, each provider family's 1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed and cross-group routing, concurrency, every Redis attach path, the in-memory slot pressure without Redis reads, a Redis outage and a killed worker * fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes * test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap * fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry * fix(caching): keep the in-memory cache bounded when entries have no usable expiry * test(caching): cover the response cache bound when cached entries have no usable ttl --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 21:10:09 -07:00
devin-ai-integration[bot]	058c84476a	fix(mcp): leave out tool texts too long to embed instead of failing tool search (#43554) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 14:53:00 -07:00
yuneng-jiang	1587faeeb9	test(unit): wait on the work itself instead of fixed sleeps (#46261) ... * test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files Replace sleeps that raced background logging, cache writes, refreshes and thread work with either awaiting that work directly (logging worker flush, the batch logger's flush_queue, the tasks the call spawned, thread joins) or an Event the test's own fake sets when it runs. Each wait keeps a 5s hang guard that never decides pass or fail. Tests that need a fake clock, real sockets, or measure latency are left unchanged, and no production code moves * test(unit): await the tasks each completion spawns instead of yielding once The logging tests yielded once with asyncio.sleep(0) so the task that enqueues the success log could run before the worker flush. That relied on the enqueue happening before the task's first await. Snapshot the running tasks around each acompletion call and await the ones it spawned, so the guarantee holds however many steps sit before the enqueue	2026-10-13 12:41:06 -07:00
mrinal-berri	231721f20a	feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination (#45880) ... * feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: ruff format otel excluded_services changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: use raw regex for excluded_services match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): reject non-string destination callback vars Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): harden destination callback var validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): compare parsed values in callback var conflict check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-12 19:25:52 -04:00
devin-ai-integration[bot]	bc2cc42340	docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI (#45838) ... * docs(proxy): describe /v1/decisions and /v1/systemone in Swagger UI Add summaries, descriptions, inlined request body schemas with examples and typed 200 responses to the decision routes and their aliases, so the OpenAPI document and the lazy snapshot show both request formats Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): cap _inlined schema walk depth for the recursion check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 21:04:59 +00:00
nate-berri	c582bc6e68	refactor(core_utils): move resource_ownership out of the proxy (#45805) ... * refactor(types): move LitellmUserRoles and KeyManagementRoutes into litellm.types.proxy.auth.user_roles Part of LIT-9423 * refactor(types): move is_jwt into litellm.types.proxy.auth.user_api_key_auth Part of LIT-9423 * refactor(auth): alias is_jwt import so JWTHandler.is_jwt is not read as recursive * refactor(types): type UserAPIKeyAuth, its token view and hash_token in place Part of LIT-9423 * refactor(types): move UserAPIKeyAuth, its token view and hash_token into litellm.types.proxy.auth.user_api_key_auth Part of LIT-9423 * fix(types): keep agent, budget snapshot and is_jwt names importable from proxy._types Part of LIT-9423 * refactor(core_utils): move resource_ownership out of the proxy Part of LIT-9423 * fix(types): restore main's _types.py content alongside the user_api_key_auth move Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(sdk): resolve litellm.proxy lazily on attribute access The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main * fix(sdk): resolve litellm.proxy lazily on attribute access The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main * style(types): drop blank lines between proxy._types re-export imports Part of LIT-9423 * refactor(proxy): keep only user_api_key_auth re-exports in _types.py Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): carry main's UserAPIKeyAuth field into moved file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(types): drop blank lines between proxy._types re-export imports Part of LIT-9423 --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 21:02:53 +00:00
devin-ai-integration[bot]	dc95de5209	feat(otel)!: let a team capture_message_content override the global setting for its OTel v2 destination (#44264) ... * feat(otel): let a team strip captured message content from its own OTel v2 destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): split the shared callback option conflict check and sync dashboard API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): address review feedback on team capture_message_content Classify capture_message_content as NotSecret in the credential slot registry, inject test exporters through register_exporter_factory instead of patching _exporter_from_spec, type the test helpers, and stop reassigning processor in build_tracer_provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): add integration cells for team capture_message_content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): cover weave_otel and newrelic team destinations in capture_message_content cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): stop rejecting client-sent capture_message_content at the request body boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(otel): let a team capture_message_content override the global setting in both directions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep a team's capture_message_content on its own OTel v2 destination The setting no longer rides the flattened team callback vars, so a sibling destination and a routed provider follow the global mode. Request bodies get the base 400 back, a failure-only entry rejects the setting, and entries for one backend must agree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): type the capture_message_content callback validation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): type the remaining capture_message_content test signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): mark the new capture_message_content test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): default tenant capture to callback_settings.otel and keep CaptureMessageContent formatting bare Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep CaptureMessageContent a plain str like main The str-to-Enum switch changed the public type: the config field's type, repr and model_dump, a constructor that now raised on unknown values, and unknown env values coerced to no_content. CaptureMessageContent is the plain str subclass main ships again, the field is a str, and a captures_span_content helper replaces the enum property. A test pins that a published preset's capture mode stays the default for an unowned destination, which is what main forwards. * test(otel): drop the module-level CaptureMessageContent import the rewritten test no longer uses * fix(lint): keep the merged branch within main's type budgets Freeze the capture-mode settings model (LIT015) and read stored logging entries through one typed sequence helper, so the conflicting-capture checks add no unknown types. * fix(otel): reject a destination capture_message_content while OTel v2 is off The legacy OTel callbacks never read the per-destination setting, so a team could register no_content and still export prompts and replies under the global capture mode. Registration now refuses the var unless the proxy runs with LITELLM_OTEL_V2=true, the same rule the Arize transport and New Relic routing vars follow. * test(otel): type the OTel v2 fixtures and the v2-off regression test * fix(otel): accept only no_content and span_only on a team or key destination OTel v2 writes captured content only to span attributes, so event_only behaved like no_content and span_and_event like span_only on a destination. The proxy-wide setting keeps all four values * docs(otel): say an omitted capture setting follows the proxy's own callback for that backend * test(otel): describe the general capture fallback in the unowned destination test --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-10 13:52:32 -07:00
devin-ai-integration[bot]	5325c4a236	feat(proxy): per-model team member budgets with router fallback (#43887) ... Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 20:33:55 +00:00
nate-berri	c397b57f2b	refactor(anthropic): move prompt cache pricing helper out of the proxy package (#45753) ... * refactor(anthropic): move prompt cache pricing helper out of the proxy package Part of LIT-9423 * fix(sdk): resolve litellm.proxy lazily on attribute access The layer cleanup stopped import litellm from loading any proxy module, so litellm.proxy was no longer bound and litellm.proxy._types.X raised AttributeError. The module __getattr__ now imports litellm.proxy on first access, which keeps import litellm free of proxy._types and fixes the docs and code-quality checks on main --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 20:32:34 +00:00
nate-berri	9d559d8579	refactor(core): move path_utils from proxy common_utils to litellm_core_utils (#45775) ... Part of LIT-9423	2026-10-10 19:07:22 +00:00
Mateo Wang	883e3210ee	feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider (#45200) ... * feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider * fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/ * fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor * fix(ui): name every OSS classifier provider in the classifier radio copy * feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry * fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry * fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router A /model/new for an auto router lands on one replica. A request routed to that auto router on a sibling replica missed it, the model read-through loaded only the auto router's own row, and the pre-routing hook then picked a tier the sibling had not loaded, so the caller got a 400 "no healthy deployments for <tier>" until the next periodic reload. An auto router miss now runs the full add_deployment reconcile, tiers included. Found by the audit's two-worker rig on this PR's routing cells; CircleCI's one-worker proxy never opens the window. * test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing cells for the complexity router's databricks classifier on all three endpoints with the Postgres rows read back, management cells for the classifier's spend writes and for the evaluation-mode health check of a Databricks serving endpoint.	2026-10-09 16:19:23 -07:00
moe-berri	22ad3c5fff	refactor(lens): connect LiteLLM to the independent Lens service (#45529) ... * feat(lens): consume the independent Lens service and shared UI * fix(lens): keep embedded setup stable and vendor UI before builds * chore(lens): pin embedded UI provenance to published Lens source * fix(lens): complete embedded UI extraction across CI builds * docs(lens): record latest main and removal-boundary validation * fix(lens): wire external chart credentials and ingestion modes * test(lens): cover adapter failures and fix extraction CI * docs(lens): refresh bundled chart setup instructions * test(lens): restore gateway adapter test package marker * build(lens): pin clean-cut UI and supported storage chart * test(ui): await guardrail scope menu before checking options * perf(lens): omit unused trace-team lookup from product requests * docs(lens): record final-image adapter permission checks * test(lens): assert delegated investigation authorization * refactor(lens): remove copied trace runtime and preserve spend logging * test(lens): verify independent gateway image upgrades and outages * test(lens): pin the chart qualification database image * fix(lens): update embedded UI metadata filters * test: wait for Lens chart pod identity convergence * fix(e2e): select Lens chart pods by deployment ownership * fix(e2e): select Helm migration ownership for Lens upgrades * test(lens): retain migration Jobs through rollout assertions * docs(helm): require existing database for migration hooks * test(lens): qualify release boundaries with embedded navigation update * docs(lens): record browser qualification for embedded tabs * fix(lens): remove unrelated CI and guardrail test changes * test(lens): give qualification tenants unique key aliases * feat(lens): embed the latest canonical Lens interface * fix(lens): qualify split inference through the gateway * fix(lens): preserve scoped auth and isolate delegated credentials * chore(lens): remove unrelated documentation and lint changes * fix(lens): bound concurrent forwarding buffers through response delivery * fix(lens): preserve OAuth2 dispatch without inference policies * chore(lens): adopt current shared onboarding UI * fix(lens): refresh shared setup UI and review context * fix(lens): bound uploads before gateway authentication * docs(lens): remove extraction evidence from the gateway repo * docs(lens): refresh shared setup prompt for paired releases	2026-10-09 16:17:05 -07:00
yuneng-jiang	5e1c5c0bd1	test: move offline proxy auth, hook, spend and pass-through tests to tests/unit (#45568) ... * test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit Relocate 53 legacy tests that pass with no network or keys into the tests/unit files that mirror the code they exercise, drop 2 llm_guard tests already covered by test_llm_guard_call_type_aliases, and delete the emptied legacy files. * test: freeze the reset clock and run the real reset methods in the moved budget reset tests Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await the service-hook tasks the job spawns instead of sleeping. Drive the per-user failure with a malformed budget_duration row from the prisma mock so the real _reset_budget_for_key/user/team run instead of patched replacements. * test: use fixed timestamps in the moved pass-through and spend log tests * test: mock the moderation HTTP call and type the moved moderation and spend log tests Serve the OpenAI moderation response through an httpx MockTransport so the real async_make_request runs, annotate the moved tests' locals with Final, and pass get_logging_payload typed kwargs instead of an untyped input_args dict. * test: drive the moved proxy tests through boundaries and annotate their locals with Final Serve GCS downloads and the Router moderation call through httpx mocks with fake Google credentials, let the real pass-through success handler fail on the mocked upstream response, and fold the Vertex live route endpoint check into the existing route test. Annotate moved locals with Final, build the budget reset rows without rebinding, and type the remaining fakes without **kwargs	2026-10-09 09:49:13 -07:00
devin-ai-integration[bot]	390bea6a53	fix(proxy): save file details for every batch output file so they list and retrieve (#41761) ... * fix(proxy): register bedrock batch output files with a file object so they list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): guard output file size when content is missing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover managed batch output file listings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): persist metadata for batch output files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover batch output file listing and retrieval end to end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve provider metadata for managed batch files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update managed batch output registration mocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): inject fake prisma client in managed batch output file tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retry provider file details and refresh fallback batch output entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover fallback batch output file details refresh after provider recovers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover model-name managed batch file retrieval Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(managed_files): bound the batch output file lookup so a slow provider cannot stall GET /v1/batches * fix(managed_files): skip the provider lookup for a fallback entry written moments ago * fix(proxy): safely refresh managed batch file details Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep batch listing provider-free Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): return metadata for empty S3 objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): strengthen batch fallback regression assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): describe provider reads and DB writes for batch output files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): sanitize caller-derived ids in managed file logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): simplify batch and file endpoint docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate API types from proxy OpenAPI spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(test): resolve managed files lint violations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): save refreshed managed file details through ManagedFileRepository Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): classify retrieved file purpose by its own bucket prefix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): move batch and file endpoint behavior notes to litellm-docs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): resolve strict lint violations in file retrieval Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): mark retrieval request metadata handoffs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve managed batch type-check diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): avoid duplicate final response bindings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Yucheng He <yucheng@berri.ai>	2026-10-08 18:51:45 -07:00
devin-ai-integration[bot]	b7ecb05340	fix(proxy): mark background responses stale_expired when provider returns 404 (#41724) ... * fix(proxy): mark background responses stale_expired when provider returns 404 Responses deleted upstream (store=false / ZDR rows dropped after provider retention) now move to stale_expired on the poll instead of being retried every cycle until the 7 day stale cleanup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop inline comment in check_responses_cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive _get_response seam in 404 stale_expired tests Assigning AsyncMock on the instance keeps the new tests inside the TQ002/TQ008 test-quality ceilings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only expire 404 responses polled through a resolved deployment A NotFoundError from the bare SDK fallback can mean a missing or misconfigured deployment, which a config fix inside the staleness window can still recover. Rows whose poll went through their router deployment are the ones the provider actually dropped, so only those move to stale_expired. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve the deployment before fetching so via_router binds once Drops the reinitialised loop flag flagged by review and modernizes the moved annotation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive the real response fetch in the poll 404 regression tests * fix(enterprise): expire background response rows on any provider 404 status The GET path maps the provider's 404 body to litellm.BadRequestError that still carries status_code 404, so an except on NotFoundError never fired and the poll job kept retrying the row every cycle. Key the terminal decision on the status code instead of the class * fix(enterprise): expire a background response only on a 404 naming it and skip a bad deployment entry per job * chore(enterprise): record the poll-cycle bound on the managed-object IN lists * test(integration): cover background response poll retirement on the real proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 10:35:52 -07:00
Mateo Wang	fbbb54c841	fix(proxy): answer a rejected websocket handshake without crashing the HTTP exception handler (#45251) ... * fix(proxy): answer a rejected websocket handshake without crashing the HTTP exception handler * fix(proxy): read the OTLP route check's scope keys with .get and drive the websocket auth tests through the real auth path * test(proxy): build the websocket auth test keys without reading the clock	2026-10-07 22:27:46 -07:00
Mateo Wang	58258409c9	refactor(proxy): expose public names for private proxy helpers (#45170) ... * refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 18:59:27 -07:00
moe-berri	e9cfba2c17	feat(lens): isolate ingestion and investigations in a Rust service (#45148) ... * feat(lens): isolate trace storage and investigation in a Rust service * fix(lens): include Rust sources in the image build context * feat(lens): wire service setup, scoped delivery receipts and lease attempts * fix(lens): complete service routing and reject stale investigation results * fix(lens): retry key propagation and validate isolated Compose setup * fix(lens): seed through isolated ingestion and preserve upstream queue fixes * chore: sync schema.prisma copies from root * fix(lens): bind nullable due timestamps as text for Prisma * chore(ui): remove stale lint suppressions * fix(lens): address CI failures and review findings * refactor(lens): remove retired Python worker and run evaluations in Rust * fix(lens): reuse control connections and satisfy review checks * test(lens): install and upgrade both Helm charts on Kubernetes * test(lens): run connection reuse coverage as an integration test * fix(ui): upgrade Next.js to 16.3.8 security release * fix(lens): fence stale attempts and preserve reviewed evidence * Revert "fix(ui): upgrade Next.js to 16.3.8 security release" This reverts commit 2f79a51b25. * fix(lens): stop failed investigations and stream history excerpts * test(lens): cover model tool and result contracts * test(lens): fix retired routes and reuse installation build artifacts * test(lens): use portable grep in Helm installation smoke * test(lens): wait for migrations before forwarding Helm services * fix(lens): keep failed evidence reads retryable * fix(lens): preserve sandbox output during process exit --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-10-07 17:18:58 -07:00
devin-ai-integration[bot]	fa2c8984ba	test: move the unit half of 126 mixed legacy files into tests/unit (#45090) ... * test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-07 14:07:43 -07:00
devin-ai-integration[bot]	dfe4df8c8a	test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807) ... * test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-07 05:02:28 +00:00
devin-ai-integration[bot]	c3a23fe499	refactor: expose core private helpers under public names (#44871) ... * refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 00:05:27 +00:00
devin-ai-integration[bot]	26a9b02f7b	feat(otel): let team and key Arize callbacks choose the OTLP transport (#44492) ... Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 14:46:50 -05:00
devin-ai-integration[bot]	837c6a7481	fix(security): remove the publicly known master key from the repo (#44718) ... * fix(security): hash the publicly known master key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: replace weak master key examples and regenerate artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: replace weak key fixtures with generated test keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: generate master keys for proxy startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve lens dev key entropy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore proxy key compatibility in scrub examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: scrub merged SSO fixture and refresh dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): stabilize test keys and metadata collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 10:55:24 -07:00
moe-berri	2190003bb8	fix(lens): reconstruct native coding agent conversations (#44711) ... * fix(lens): reconstruct native coding agent conversations * fix(lens): display timestamps used for conversation ordering * fix(lens): preserve coding trace identity and message provenance * fix(lens): complete capture checks after trace pagination * fix(lens): show loaded replies during trace pagination	2026-10-05 17:51:05 -07:00
devin-ai-integration[bot]	1238cfe90f	feat(proxy): add LITELLM_FIPS_MODE startup gate with provider assertion and loud password migration failure (#42700) ... * feat(proxy): LITELLM_FIPS_MODE startup gate with provider assertion, TLS verify guard and loud password migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): match ssl_verify off detection to runtime str_to_bool semantics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop tautological fips probe test and satisfy CodeQL return checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): inject the fake Prisma client through the module boundary instead of patching _setup_prisma_client Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-05 01:11:52 -07:00
devin-ai-integration[bot]	0ed1c08f02	feat(anthropic): workload identity federation and pluggable identity sources (#44448) ... * feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * fix(proxy): let a deployment write name a federated credential reject_federated_credential_reference runs from is_request_body_safe, which pre_db_read_auth_checks calls on every route, so it also fired on POST /model/new, /model/update, /model/{id}/update and /health/test_connection. A proxy admin could no longer attach a federated credential to a deployment over the API or the Admin UI, leaving a static config.yaml entry as the only way to configure the feature the rejection told the caller to go configure, and _reject_non_admin_wif_write never got to make the call it exists to make. is_request_body_safe now takes the route and skips only the credential-reference check on the routes that reach can_user_make_model_call. Federation fields typed inline into a body stay refused everywhere, and a call naming a federated credential still cannot pick the identity it mints as. * refactor(proxy): derive health display policy from the federation key sets The health check module hand-copied the five workload identity fields whose value is a credential, so a shared proxy surface named provider-specific parameters and a newly added secret-bearing field would have gone on being displayed until someone remembered both places WIF_SECRET_BEARING_KEYS now sits beside the key sets it splits out of, types/utils derives secret_bearing_wif_litellm_params from it, and the health layer splats that tuple the same way it already splats the admin-only one * fix(anthropic_wif): treat blank identity-source fields as unset * test(proxy): classify the federation params in the credential slot registry main's registry test (#43298) now fails the build for any credential-named deployment param without a classification. The five federation fields that carry a token, a token file path, or a signing or client secret reference are Unplanted, matching WIF_SECRET_BEARING_KEYS; the four remaining Keycloak settings name a URL, a client id, an auth method, or a scope and are NotSecret * fix(anthropic_wif): declare federation params as owned connection leaves and chart their metrics Register the 18 Anthropic and 3 OpenAI federation params as frozen ConnectionSettings leaves so the owned-kwarg registry, the kwargs funnel and the request-body ban list read one declaration. Pass the deployment api_base through to the count-tokens handler instead of a pre-suffixed URL, which doubled the /count_tokens path on main's prompt-cache predictor. Add the five litellm_anthropic_wif_* families to the all-metrics Grafana dashboard. * fix(credentials): gate PATCH on WIF fields resolved from model_id The credential PATCH handler checked server-owned workload identity federation fields only on the values the caller sent, while a body that named a deployment through model_id had its credential values resolved after that check. A non-admin could therefore copy a federated deployment's WIF fields onto an ordinary credential. Resolve the incoming values first and run the non-admin gate on them, matching the POST path * fix(anthropic): count tokens with ANTHROPIC_AUTH_TOKEN through the shared auth header Count-tokens walked its own credential ladder: a static key, else skip minting when ANTHROPIC_AUTH_TOKEN is set, else mint a federated token. With only the auth token set it forwarded nothing and the proxy silently fell back to its local tokenizer while chat on the same deployment authenticated with that token. The handler now takes the auth header that AnthropicModelInfo.aget_auth_header resolves, the same ladder chat, files, batches and skills use, and merges the oauth beta a minted or consumer token carries with the token-counting beta --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> Co-authored-by: mateo-berri <happymvw@gmail.com>	2026-10-03 17:08:30 -07:00
devin-ai-integration[bot]	fe683ea139	feat(proxy): serve a Codex-native model catalog with per-model service_tiers from /v1/models (#44136) ... * feat(proxy): serve a Codex-native model catalog with per-model service_tiers from /v1/models GET /v1/models and /models answer Codex CLI's catalog fetch (the request carrying its client_version query parameter) with Codex's own {"models": [...]} shape: a model Codex knows keeps the metadata of its bundled 0.159.3 catalog (vendored), any other model gets Codex's fallback entry, and model_info.service_tiers becomes each entry's service tiers so Codex offers them as slash commands that send service_tier upstream. Without the parameter the OpenAI list shape is unchanged. The CLI's litellm agents codex catalog shares the same builder. * fix(proxy): offer a Codex service tier only when every deployment of the model lists it * fix(codex-catalog): an invalid service_tiers value offers no tier for the model * fix(codex-catalog): read service tiers off the deployments the key's team can route to A tier is offered to Codex only when every deployment of the model name a request from the key's team can route to lists it, so another team's deployment of the name and a deployment an admin paused via model_info.blocked no longer withhold or add tiers for requests that never reach them The catalog's always-null fields are annotated NoneType so the module imports under pydantic 2.12.0 on Python 3.14, the lowest pin the MCP resolve job installs, which rejects a None annotation with a None default * test(codex-catalog): drop the redundant module docstring and sort the imports * test(integration): add the Codex catalog audit cells and the multi-worker convergence note * test(integration): clean up every catalog test model and answer the refresh GET * fix(proxy): keep tiered models under Codex's catalog cut and resolve alias tiers Under Codex's 1 MiB catalog limit the entries offering a service tier are kept ahead of those offering none, each group in model_list order, with every kept entry at its listing position, so the model an operator configured tiers for survives a wide key's long listing. A model_group_alias row reads its target's deployments, so it carries the target's tiers and stock metadata under the alias name. * fix(proxy): pick Codex catalog metadata per team and skip entries too large for the cut The upstream model that selects Codex's stock entry was read off the first deployment of a name without checking the key's team, so a team whose requests route to a different deployment could be handed another team's prompt, reasoning levels, and tiers. The upstream model and the tiers now come from the same team-aware selection routing uses, and a caller with no team reads the deployments no team owns The byte cut kept a prefix of the tier-first order, so one entry larger than the whole limit emptied the catalog. An entry too large for the bytes left is now passed over and the smaller ones after it are still kept --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-03 20:42:55 +00:00
devin-ai-integration[bot]	4c6c84afb7	perf(proxy): stop prompt-cache eligibility from tokenizing the whole conversation (#44221) ... is_prompt_caching_valid_prompt ran the full Python token_counter over every message to compare against the deployment's prompt cache minimum, 500 to 1000 ms at 440k to 740k tokens on every request that reaches the prompt_caching pre-call check, Rust on or off. messages_reach_token_count does the same arithmetic as token_counter(...) >= threshold and stops at the first message that reaches the threshold. Groups with one healthy deployment skip the prefix hash and pin lookup, which cannot change the result for them Four fixed name span events make the pre-LLM phases measurable with OTel v2: litellm.request.body_received (with body_bytes) once per body read before parsing, on the JSON, binary and form branches, body_parsed, pre_call_completed, and deployment_selected emitted once per pick inside Router.async_get_available_deployment and get_available_deployment with attempt, reason and model group, so every router surface, retry and fallback is covered. Measured locally on /v1/chat/completions, /v1/messages and /v1/responses at 440k tokens with Rust on and off against a fake upstream Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 09:20:28 -07:00
devin-ai-integration[bot]	797353f13a	fix(otel): name postgres service spans by operation and table (#44240) ... Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 09:20:28 -07:00
devin-ai-integration[bot]	e340e546e2	feat(traces): tracing development seed (#44363) ... * feat(dev): seed linked tracing and spend fixtures * chore(dev): use OpenAI model in tracing config * chore(dev): align tracing credentials with UI E2E * fix(dev): update fixture seeder query scope * feat(dev): seed linked tracing and spend fixtures * chore(dev): use OpenAI model in tracing config * chore(dev): align tracing credentials with UI E2E * fix(dev): update fixture seeder query scope * wip * wip * wip * chore(trace): checkpoint ongoing Rust migration * refactor(trace): group Python bridge under trace package * refactor(traces): read span conventions through a Convention trait Each span format (Claude Code, LangSmith, OpenInference, gen_ai) now lives under normalize/convention/ as a unit struct implementing Convention, owning both its detection and its extraction. Precedence is one ordered registry instead of an if-chain in mod.rs that reached into each module differently. The modules now share one way to read attributes: present() for the first non-empty key and Payload for a text that also reports the key it consumed, replacing three different idioms and the &mut Vec threaded through payload readers. Instrumentation::adjust returns a new Extraction instead of mutating one, with each SDK rule as its own function, and the LangChain middleware suffix list exists once. * feat(trace): export Rust-owned wire schemas and enforce contract bounds * fix(trace): bound quoted counts in ClickHouse wire schemas * feat(trace): generate Python wire contracts with datamodel-code-generator * test(trace): validate migrated callers and generated contracts at the native boundary * refactor(traces): rename normalization convention to format * fix(traces): reconcile spend evidence and preserve unknown costs * feat(traces): normalize additional telemetry formats * test(traces): cover captured normalization fixtures * refactor(traces): isolate SDK normalization rules * feat(tracing): seed all trace exports for local dashboard * fix(clickhouse): preserve custom LiteLLM request metadata * docs(traces): define normalization module boundaries * docs(traces): define resolution and OTLP boundaries * fix(ui): normalize nullable trace message names * refactor(traces): split resolver modules and cover resolution behavior * test(traces): replace normalization snapshots with behavior assertions * fix(ui): align dashboard API contracts with generated types * refactor(traces): type normalization and storage boundaries * fix(traces): seed captured SDK spend and preserve provider identities * wip * test(traces): verify guide discovery and content ordering --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 09:59:51 +00:00
yucheng-berri	5d42cb7cfa	fix(guardrails): encrypt guardrail litellm_params secrets at rest (#43627) ... * fix(guardrails): encrypt guardrail litellm_params secrets at rest * fix(guardrails): keep salt-key encryption on master key rotation and retry rows edited mid-rotation - rotate guardrail params under LITELLM_SALT_KEY when set, matching the key reads decrypt with - re-read and retry a row whose updated_at moved during rotation, up to GUARDRAIL_ROTATION_ATTEMPTS - build decrypted Guardrail rows and the rotation count without mutating locals * refactor(guardrails): retry guardrail rotation by bounded recursion instead of a rebound cursor - each attempt re-reads the row and recurses with attempts_left - 1, so no loop variable is rebound - cover the give-up path after GUARDRAIL_ROTATION_ATTEMPTS writes * test(guardrails): drive the real guardrail rotator from the master key rotation test - inject an encrypted guardrail row through the prisma client instead of replacing the GuardrailRegistry method - assert the written params decrypt under the new master key * Annotate guardrail param encryption collections for type-discipline gate * Type guardrail param recursion through validated JSON containers * Type guardrail registry test helpers and drop section comment * Reject client-supplied encrypted values in guardrail litellm_params * Allow depth-bounded contains_encrypted_marker in the recursion detector * Keep a loaded guardrail when its DB params do not decrypt with the current key * Apply other DB edits while keeping loaded values that do not decrypt, including PATCH models * Keep the loaded guardrail when an undecryptable param has no loaded value * Drop suppressions the type discipline gate on main now reports as unused * Assert what the reinitialized guardrail holds after an edit to an undecryptable one * Drive the rotation sync tests through a registered guardrail instead of patching reinitialize * Type the rotation test helpers and drop the new test docstrings * fix(guardrails): refuse to approve a submission whose params do not decrypt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-02 22:49:41 -07:00
yujonglee	677205b3f5	fix(proxy): share ownership permissions for spend logs and traces (#44239) ... * refactor(proxy): extract shared spend log read policy * test(proxy): use named bindings for spend scope regression * test(proxy): reuse existing spend log query harness * test(proxy): cover spend log permission lookup adoption * chore(proxy): relocate existing spend query baseline * refactor(proxy): make scope query returns explicit * refactor(proxy): inject deferred log permission lookup * test(proxy): cover teamless management compatibility lookup * refactor(proxy): compose user and team log grants * refactor(proxy): share generic authorization composition * refactor(proxy): compose trace read permissions * refactor(proxy): centralize spend and trace authorization * refactor(proxy): strengthen spend and trace scope types * refactor(proxy): flatten log read scope into owned logs Replace the AnyOf grant tree with a flat OwnedLogs(user_id, team_ids) scope, and OwnedTraces(logs, api_key_hash) for traces, since every consumer flattened the tree back into that shape. A caller with no user id now gets an empty scope instead of matching ownerless rows through Prisma's IS NULL. The dead request_id guard in ui_view_spend_logs is removed, and the management facets inject the log team lookup and reuse read_scope_sql instead of the list shim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): run spend scope tests through one SQLite emulator Replace the string-matching payload emulator and the hand-rolled Prisma where interpreter with one SQLite helper that runs the real scope SQL. Session scope tests now go through the endpoint, including the no-user caller that must not match ownerless rows. Drop duplicated lookup-failure and trace mapping cases. load_permitted_log_team_ids returns no teams without a database instead of relying on the resolver's broad except. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): unify log and trace ownership permissions * test(tracing): align fixtures with ownership read scopes * refactor(tracing): align query scopes with row ownership * refactor(spend): make ownership SQL predicates explicit * test(spend): validate ownership SQL against PostgreSQL * docs(traces): drop key-row visibility from query help guide Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): reach the empty-memberships branch in team lookup test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate dashboard API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 01:38:54 +00:00
yuneng-jiang	b877a38e5f	fix(proxy): stop queued registry read-throughs spending the resync budget (#44277) ... * fix(proxy): stop queued registry read-throughs spending the resync budget RegistryReadThrough.attempt serializes misses behind one lock, but every request that was queued behind the first one still spent a unit of the 20-per-5s resync budget and re-ran the DB resync, even though the first request had already loaded the object. A burst of more than 20 requests for a model created on another worker therefore exhausted the budget and the rest got 400 "Invalid model name". attempt now checks whether the key is already loaded once it holds the lock and returns early without touching the budget. Models check the router's model names and deployment ids, guardrails and agents reuse their existing registry lookups. * test(proxy): gate the queued read-through test on events and cover each registry's loaded check The queued-requests test now holds the first resync on an asyncio.Event instead of a timed sleep and records calls in a recorder with tuple and frozenset state. New tests show the model, guardrail and agent read-throughs each answer an object that is already loaded without reading the database, so rewiring any registry's loaded check now fails a test * test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture	2026-10-02 18:12:56 -07:00
yujonglee	ec605826d4	feat: improve trace ingestion and trace details (#43975) ... * refactor: separate OTLP HTTP decoding from trace codec * feat: complete trace ingestion and read paths * fix: encode OTLP protobuf errors in Rust * fix: raise OTLP body limit to 16 MiB * test: cover OTLP auth body parsing boundary * refactor: parse OTLP media type into enum * fix: enforce OTLP body size at HTTP boundary * perf: preserve shared OTLP metadata across ingestion * bench: compare owned and shared trace resource fanout * refactor: extract shared storage and Python conversion caches * refactor: keep shared storage owned by traces * test: keep trace loopback coverage in Rust * test(proxy): adapt trace coverage to injected access context * fix(tracing): satisfy stacked branch lint checks * refactor(tracing): use immutable ingestion payloads * fix(tracing): declare native error encoder export * test(proxy): resolve trace access through dependency * fix(tracing): align merged normalizer types and bridge tests * fix(tracing): address ingestion and diagnostic review findings * fix(proxy): preserve body parsing for partial request scopes * test(proxy): use valid HTTP scopes in request fixtures * test(proxy): complete auth request flow scopes	2026-10-01 13:45:33 -07:00
devin-ai-integration[bot]	a76b59db9f	test(proxy): move middleware, spend_tracking, pass_through, common_utils and root proxy tests into tests/unit/proxy (#44015) ... Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-01 18:23:31 +00:00
tin-berri	de06c93767	feat(router): opt in to prompt-cache cost routing (#43232) ... * feat(router): opt in to prompt-cache cost routing * fix(router): address prompt-cache routing review * ci: include cache-routing regressions in coverage	2026-09-26 18:13:13 -07:00
devin-ai-integration[bot]	e0fb89bc82	fix(proxy): keep the submitted body out of 422 validation errors (#43231) ... Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-09-25 17:05:53 -07:00
Mateo Wang	25fb7810c2	fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token (#40541) ... * fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token A CLI session token is a fresh random secret on every login, so since v1.99 each login's spend rows carried a different sha256 hash as api_key and the usage APIs could resolve neither key_alias nor user_email for them. Spend rows and logging callbacks now attribute a session request to its stable alias, cli-session-<user_id>, and the usage endpoints derive that alias and owner from the key itself instead of scanning for a matching digest * fix(spend): resolve the CLI session team from the user's first team in usage metadata A cli-session key carries no team of its own in the DB, so the usage breakdown showed team_id None for it and the export grouped it as Unassigned. The login attaches the user's first team to the session, so the recovery mirrors that rule for cli-session keys only. * fix(spend): claim the session team only for a single-team user The CLI login attaches a team on its own only when the user has exactly one; a user in several teams picks one per login, so usage metadata for the alias would otherwise name a team the login may not have used. * test(pass_through): mark the mocked auth object as a plain key The logged key follows the alias only for a session token; a bare MagicMock reads as one, so the test names the field it relies on. * fix(spend): attribute CLI session pass-through, queue, and managed batch spend to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): only treat the exact cli-session-<created_by> value as a batch key alias A managed object row written by an older build can still carry the raw per-login session token, which shares the cli-session- prefix. Matching on the prefix alone would have surfaced that token as a trusted alias and persisted it verbatim in the batch cost spend log, so the alias check now requires the exact per-user value and every other prefixed value keeps going through redaction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): log proxy executed batch rows under the cli-session alias instead of the session token _row_metadata set user_api_key from the raw bearer token while user_api_key_hash carried the alias, so the spend log redaction rejected the alias as untrusted and hashed the random session token instead Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): attribute semantic search embedding spend to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): scope /key/spend/report for a CLI session to the cli-session alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): use the cli-session alias for websearch spend, prometheus failure labels and the parallel limiter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(spend): drop explanatory docstrings on get_logged_api_key and attach_user_details Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): only recover cli-session usage keys whose suffix is a known user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-24 18:21:47 -07:00
devin-ai-integration[bot]	248f0eb159	ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) ... * ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * build: point the local proxy unit targets at the nested tests/unit/proxy tree * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-09-24 22:59:11 +00:00