mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-25 05:54:53 +00:00
94 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
alberto
|
bbd446b476
|
fix(proxy): report requested model in streamed Responses response.model (#46662)
* chore: Add failing test case Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Restamp nested response.model in streamed Responses events Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Add tests Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * chore: Run format and lint, and fix Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Type nested Responses restamp to satisfy basedpyright gate Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(proxy): restamp the nested model on streamed response.queued events Background streams send response.queued as a GenericEvent, so its nested response.model still carried the backend name. Also drop the new comments * fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events * test(proxy): drive typed events through the nested restamp no-op test * fix(proxy): restamp the nested model on guardrail block frames in streamed responses * test(proxy): assert a cache-polled background response reports the requested model * fix(proxy): restamp nested model on dict events and on every SSE data line --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
4a58784fa2
|
fix(cost): price exact cost-map names before price-free capability rules, and fix SCIM and MCP integration tests on CircleCI (#47037)
* fix(test-harness): preserve sibling auth modules in isolated worker configs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect servers in MCP connect-flow descriptions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect ptu_hours in daily activity metrics after #43043 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): kill every serving worker once the SCIM push blocks on the row lock * fix(cost): price exact cost-map names before price-free capability rule matches * test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): price an explicit base_model under its own provider after candidate reordering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): retain priced generalized candidates and skip explicit-pricing lookups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): assert custom pricing makes no remote model-info requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): keep the per-candidate pricing provider final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments * fix(cost): sort cost candidates from the static cost map and rules, without provider lookups * test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
yuneng-jiang
|
910c30381b
|
test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001)
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets * test(integration): wait for the team spend flush before killing a worker * test(integration): poll for the S3 attempt the batch sinks write concurrently * test(integration): queue the terminal object and its sibling into the same s3_v2 flush * test(integration): time the long Bedrock version against an equally long unversioned name * test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it * test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries |
||
|
Deepanshu Pal
|
d5ce7dbb8c
|
fix(proxy)!: honor caller litellm_session_id in spend log session_id (#44199)
* fix(responses): scope session history lookup to the referenced key * Implement SQLite-backed DB for session tests Added a SQLite-backed database class for testing and a new test to ensure session isolation between different API keys. * fix(proxy): honor caller litellm_session_id in spend log session_id Updated docstring to clarify session ID resolution logic for spend log rows. * test(proxy): caller litellm_session_id wins over the trace id * style(proxy): fix blank lines around new tests * fix(proxy): read the caller session from the logging payload for spend logs * fix(responses): match user and team too when replaying session history * fix(proxy): keep bridged Responses chains whole and scope session history by owner * fix(proxy): keep a metadata session next to a trace header and let session headers win * fix(proxy): send the trace-header session only to spend logs and let session headers win beside a trace header * fix(proxy): drop a client-sent caller session and keep it out of requester metadata * fix(proxy): keep the trace id as the spend log session when the caller session contains a NUL byte * fix(responses): keep a continued bridged chain's session under request_correlation_in_logs With the flag on, the caller session outranked the chain session the bridge sets for a continued Responses chain, so a chain whose session changed or first appeared on a later turn split. The continuation now logs under the chain's session and drops the turn's caller session key, as on main. Adds the /audit integration cells for the spend log caller session. * fix(responses): log a continued bridged chain under its session through the spend-log key only The previous commit set the chain session as the turn's litellm_session_id, which also moved the OTel conversation id of a continued turn away from its trace header. The bridge now writes the chain session into the internal spend-log session key of the metadata the spend log reads, so only the spend log session changes, as on main. Adds an OTel cell for a continued chain. * test(spend): cover bridged chains that share one caller session * test(spend): cover the caller session on configured pass-through routes * test(spend): type the SQLite fake and split stacked comprehensions in the caller session tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4e2dd65549
|
fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys (#45743)
* fix(interactions): keep background interaction bills through crashes, late restarts, and deployment keys A settlement claim is now a lease, so a row whose settler died between claim and outcome is claimable again after the lease and boot resumes it. A resumed poll gets the full poll window and fetches with the serving deployment's credentials from the live router. The stored create context keeps the request tags and the request metadata that the creating worker's own spend row used * fix(interactions): bill a lapsed claim's takeover only when no spend row exists A settler that billed and then died, or failed to record the outcome, left the row unsettled, so the replica that took over its lapsed claim billed the interaction a second time. A takeover now looks up the spend row keyed by the interaction id and records the row as billed when it finds one. * fix(interactions): settle Vertex AI and edited deployments, never double bill with spend logs off A takeover with spend logs off records the interaction unsettled instead of billing it again, and a release records its outcome before freeing the budget reservation. A resumed or live fetch carries the deployment's Vertex AI project, location, and credentials, and a stale deployment id falls back to the model group only when one same-provider deployment serves it. A poll stops at the first 400, 401, 403, or 404, string request tags no longer fail the create, and settled_at gets an index built concurrently. * fix(interactions): leave a live replica's interaction to it when a resumed poll cannot read it A replica that boots with credentials unable to read an interaction got a 404 on its resumed poll and settled the row as unsettled at once, so the replica that created it, still polling, found it settled and never billed it. A resumed poll that gets a 400, 401, 403, or 404 now stops fetching and keeps checking the row until its poll window ends, and settles it as unsettled only when no other replica did. * fix(interactions): keep the creator's metadata on a resumed bill and retry outcome writes inside the claim lease A bill another replica writes now carries the metadata the creating worker's own spend row is built from, so a client's metadata.model_group no longer relabels it. A failed outcome write is retried for half the claim lease before the budget reservation is left to a takeover, and the creating replica retries every fetch error for its whole window again, as main does. The resume integration case now checks that a resumed fetch uses the deployment's own key. * fix(interactions): resume background interaction polls only after database models load Boot resume ran before the proxy loaded the models stored in the database, so a resumed poll for a model added through /model/new or the Admin UI fetched with env credentials. A 4xx there stops a resumed poll, so the bill was lost. The settlement now installs after the database models load. The integration test resumes a database model with an eager first fetch and fails with the old startup order. * fix(interactions): renew a settling claim so a slow settle is never taken over and billed twice A settle runs every success callback in turn, and one slow callback could hold it past the 5 minute claim lease. A booting replica or a DELETE then took the live claim over, found no spend row yet, and billed the interaction a second time. The settler now renews its claim every third of the lease until it records the outcome, so only a settler that stopped renewing is taken over. A takeover also read the spend log after writing its claim, so a failed read left the row under the taker's own fresh claim, which its poll then took for another settler's and left unbilled until a reboot. The spend row is now read before the claim is written, so a failed read takes nothing and the claim is retried on the next poll. * test(spend): cover lapsed claim takeovers, rotated deployment keys, late restarts, and resumed bill tags --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
5ed3629242
|
fix(responses): write one spend-log row per native WebSocket turn (#46768)
* fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection logs and bills as its own call when its terminal event arrives, keyed by the resp_ id the client saw, instead of one row at socket close that sums every turn * test(responses): mock the websocket slot release and mark the per-turn start time naive * style(responses): keep the per-turn start time line within ruff format * refactor(responses): type websocket turn events as read-only mappings * fix(responses): settle the websocket budget reservation on the first turn only Every per-turn logging copy carried the connection's admission reservation, so two turns whose cost callbacks overlapped could both reconcile it. The first turn keeps the reservation and later turns charge their cost directly * fix(proxy): keep the legacy limiter's websocket slot until disconnect A websocket turn callback no longer decrements current_requests in the legacy limiter. The socket's slot is released once from arelease_max_parallel_requests_on_disconnect, which now dispatches to the legacy handler as well --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d25b58f744
|
fix(count_tokens): count Anthropic-format requests for OpenAI the way /v1/messages inference converts them (#46759)
* fix(count_tokens): convert Anthropic-format messages for the OpenAI counter the way inference does * test(count_tokens): check /v1/messages/count_tokens sends OpenAI the input /v1/messages inference sends * fix(count_tokens): build the Anthropic count request without a cast and sync dashboard API types * test(count_tokens): compare tools by value in the public api test * fix(count_tokens): count Anthropic bodies the way /v1/messages inference sends them to OpenAI Unwrap affinity-encoded reasoning, count the structured-output schema, flatten replayed web search results, join a list system prompt, and answer a message without a role with a 400 before calling OpenAI. messages_format is now internal to the Anthropic count route instead of a public /utils/token_counter field. * test(count_tokens): expect the image detail the adapter now sends after merging main * fix(count_tokens): type normalize_native_messages without the unimported Any On Python 3.10 to 3.13 the annotation is evaluated at import, so the proxy failed to start with NameError * test(count_tokens): expect the converted Responses input in the budgeted-key count integration test --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> |
||
|
devin-ai-integration[bot]
|
67bfb7a148
|
fix(ollama): prevent reused response ids from dropping spend logs (#39227)
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): narrow the provider response id with isinstance instead of cast Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_tracking): validate callback fields for current type gates * test(gcs_pubsub): assert provider response id in spend metadata * fix(ollama): normalize reused response ids automatically --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
7090588bdf
|
feat(gemini)!: auto-continue gemini-4-argon long decodes with continuationToken (#46240)
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions * fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel * feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing early-access row, carrying the same limits and capabilities. The AI Studio row mirrors the early-access entry; the Vertex row differs only in provider and pricing source. Prices follow once Google publishes them. Extend the mapped metadata test to cover all three ids, that each routes to its provider, that its backup copy matches, and that the public rows carry the same limits as the early-access row. * feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops * chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap * fix(gemini): follow continuation hops in a loop instead of recursion The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls * fix(gemini): bill Google Search grounding from every continuation request Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain * fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch * chore(gemini): name the type errors the dual-mode stream iterator trips The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up * fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment * fix(gemini): keep one response id across a continued answer A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key * test(gemini): add integration cells for long decode continuation Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once * fix(vertex_ai): fetch the access token again before each continuation request A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged * fix(vertex_ai): start each continuation stream with an empty parser buffer When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async * fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way * fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes * fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not * fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0 A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer Adds the audit's wire tests for both, which fail 24 cells before this change * test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed * fix(vertex_ai): log a continued answer's raw response as one object A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch * test(integration): check a continued answer logs one merged raw response A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch * test(vertex_ai): cover a follow-up tool call and its thought signature in long decode * test(gemini): pin that a one-request answer logs Google's body unchanged |
||
|
Dushyant Acharya
|
06efa5de85
|
fix(proxy): honor disable_end_user_cost_tracking in spend logs and end-user tables (#40945)
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs * fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off * fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off * test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e344dbd8db
|
feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043)
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
|
||
|
devin-ai-integration[bot]
|
fbc2e358bc
|
fix(responses): bill a background response once, on the first read or poll that sees it finished (#45703)
* fix(responses): bill a background response once, from the poll job
Reading a completed background response with GET /v1/responses/{id} wrote a
full-cost spend row on every read, on top of the row the background cost poll
writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and
the poll job is the single biller.
The poll job now bills with the creating request's attribution (key hash, team,
org, user, aliases, tags) read off the managed object row, through a helper it
shares with the batch cost poll. The background create persists the creating
key and request tags so that attribution survives until the poll runs.
* test(proxy): a background response read reports zero cost
* fix(responses): bill a background response once, on the first read or poll that sees it finished
A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll
* fix(responses): keep the caller's response id out of the billing failure warning
* fix(responses): keep the creating client's User-Agent tags on the one background charge
* fix(responses): keep managed background reads free and bill each response once
The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404.
* fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts
* fix(responses): give the background billing event its own spend log row and bill responses the create left untracked
* fix(responses): narrow the managed files hook with a protocol instead of an enterprise import
* fix(responses): leave a read the caller retargets with client-side credentials to the cost poll
* fix(responses): bill background responses on the creator's end user and report the charge on the settling read
The read that settles the bill returns the charge in x-litellm-response-cost, the billed event
carries the creator's end user and the key, team, and user budget fields, and its spend row is
named after the id the client got. Override reads, cancels, and deletes settle from the
configured deployment, a streamed create marks its row billed, stored rows keep no model
output, and model insights counts the billed row's spend without a request.
* test(responses): cover background response billing across proxies, reads and chaos
* fix(responses): settle background billing only for callers allowed to address the response
A delete now settles the bill only after the response id access check passes, a streamed
create is marked billed when it sets background inside extra_body or its completed event
arrives after the last chunk, the billing event keeps the deployment's litellm_params
base_model, a key with no user and no team bills no user instead of its owner scope, and the
enterprise cost check import is optional
* fix(responses): record the bound on the pending status IN filters
* fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was
The read that settles a background response now sends the whole cost header
family from the billed event, so the breakdown headers add up to the charge.
Batch cost events go back to main's attribution (creator id as the user, no
budget fields); only background responses get the end user and budgets.
Integration tests expect the charge on the settling read (or none when the
poll billed first), one charge for a streamed create, and SDK flows that poll
with the id each read returns.
* fix(responses): bill each background response once across upgrades, streams, and failed settles
Rows stored before this billing model are found by the issued id and rekeyed by the
provider id, and the cost poll retires one a read already billed. A streamed background
create is free on its own and bills the finished response through the same managed row
before the client sees response.completed, so a racing read cannot bill it again. A
settle that fails retries in the background, a retargeted read with no row tracks and
bills from the configured deployment, and a delete keeps its billed cost header.
Reverts the unrelated json.dumps narrowing in the completion transformation.
* fix(responses): bill cache-polled background streams and match poll rows by the issued id
* test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing
Adds integration cells for the background response billing paths the audit
inventory lists: a DELETE that settles the bill and refuses a stranger, a
settle the database refuses that retries once it recovers, the Prometheus
remaining budget, base_model pricing from model_info and litellm_params, the
daily model usage rollup, the end user, the settling read's cost split, stored
rows without model output, a forged managed marker on the create, streamed
creates with background in extra_body, and the cache-polled create
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
yucheng-berri
|
1f4ef4b7d1
|
feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs (#45804)
* feat(proxy): general_settings.disable_requester_ip_address_logging keeps caller IP out of spend logs
When the setting is on, spend log rows store requester_ip_address as null in
the column, the metadata and the stored request (which also drops client-IP
forwarding headers), and IP addresses in stored error messages are masked, so
the Logs page shows no caller IP. The setting is read the way the settings API
validates it. Capture is unchanged, so guardrails, logging callbacks and
metrics still receive the IP. Toggleable from the Admin UI General Settings.
Resolves LIT-7442
* fix(proxy): mask IPv6 addresses that end or start with :: across eight groups
The masking pattern allowed seven colons, so valid forms like 1:2:3:4:5:6:7::
and ::1:2:3:4:5:6:7 stayed in stored error text with
disable_requester_ip_address_logging on. Also lists the request metadata walker
in the recursion check, since it only walks the tree the spend log sanitizer
already returned.
* fix(proxy): mask IPs in every stored header value and after labels or underscores
With disable_requester_ip_address_logging on, client IPs carried in headers outside the forwarding list (do-connecting-ip, incap-client-ip, custom names) and addresses written as host:2001:db8::2, peer_2001:db8::2, socket_192.0.2.1 or an unbracketed IPv6 with a port were still stored in spend logs.
* fix(proxy): mask request header IPs before long values are truncated
A header value longer than the spend log string limit kept the part of an IP that fell before the truncation cut. Request headers in metadata and requester_metadata are now masked before the body is truncated, like error text already was. IPv4 followed by a dotted port (192.0.2.1.443) is masked again.
* fix(proxy): mask the IP piece a header truncation cut leaves instead of pre-masking headers
Masking header values before truncation scanned the whole client value,
including text the cut drops, and walked hook-made header cycles. Mask after
truncation instead, then mask the hex and digit run touching each side of the
truncation marker when it is short enough to be part of an IP address. Every
nested headers mapping gets the same treatment.
* docs(proxy): say the requester IP setting masks IPs in other stored header values
* fix(proxy): mask stored error text after truncation and harden the cut-piece pass
Error messages and tracebacks were regex-scanned in full before the DB cap,
which costs seconds on multi-megabyte provider errors. Mask them after the
cap like header values, with the same pass over the run beside each
truncation marker. That pass now handles every marker, skips long hex tokens
so an address beside one is still masked, keeps at most one address length,
and also masks colon-only IPv6 groups.
* fix(proxy): drop the requester IP from the originating request an auto router classifier row stores
* fix(proxy): drop the requester IP from the classifier input an auto router classifier row stores
* refactor(proxy): mask stored IPs with the content filter's ipv4 and ipv6 patterns
The requester IP setting now reuses the content filter guardrail's prebuilt
ipv4 and ipv6 patterns instead of its own address regexes, so spend logs mask
what that guardrail masks. Addresses glued to a label or underscore stay as
the content filter leaves them.
* fix(proxy): mask stored IPs with the spend-log matcher again
The content filter's ipv4 and ipv6 patterns left labelled IPv6 (host:2001:db8::2),
unbracketed IPv6 with a port and IPv4 after an underscore in stored error text and
header values, and masked parts of text like std::io::Error. Restore the
address-validating matcher and its regression cases.
This reverts commit
|
||
|
Mateo Wang
|
d81a5719ad
|
fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359)
* test(integration): follow count_tokens error surfacing and decisions pricing in CircleCI suites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore Responses input_file PDF pricing coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep integration AGENTS guide unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cache): preserve provider field presence on cache hits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect credential display name in response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): align OpenInference refusal field expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect Akto request ingestion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): expect missing tool error wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect no refusal field in cursor chat wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Oliver Jensen
|
14e820ef7c
|
fix(proxy): scope /global/activity/cache_hits analytics to the caller's own and permitted-team logs (#46210)
Non-admin callers previously received deployment-wide cache activity, error breakdowns, every key alias on the proxy, and could filter the aggregates by any other tenant's key alias. All four queries now take the caller's read scope as bind parameters, which also shrinks the key-alias and model filter options to the caller's own visibility. The three-way scope resolution (admin / internal user / fallback) now lives in resolve_spend_log_read_scope, shared with /spend/tags and /spend/logs/session/ui instead of being duplicated per endpoint. |
||
|
devin-ai-integration[bot]
|
cfa98e2f40
|
test(integration): cover pass_through endpoint contracts (#44690)
* test(integration): cover pass_through endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): run configured endpoint contracts on one proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove pass-through streams, SDK spellings, endpoint renames and bedrock stream bodies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover pass-through wire behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Vertex and Gemini passthrough authentication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove pass-through runtime method, credential-less Anthropic, stream barriers and Vertex strip layers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(pass_through): gate unquoted Bedrock stream targets and assert configured endpoint part types and methods after rename Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert pass-through query pairs so duplicate params fail Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
feb92d3255
|
test(integration): cover orgs_users_customers endpoint contracts (#44725) | ||
|
devin-ai-integration[bot]
|
5f7de8d6ac
|
test(integration): cover keys_auth endpoint contracts (#44674)
* test(integration): cover keys_auth endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover hashed, path and token spellings for keys_auth contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen JWT mapping lifecycle coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover reset_spend over plaintext and hashed path spellings with reset_to 0 and 0.02 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop Final in reset loop and use drain-then-fresh-refusal in warmed-peer delete proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): disable model info refresh in JWT lifecycle rig Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop unused discovery constant Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover cache invalidation and key paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
devin-ai-integration[bot]
|
8822e086ec
|
test(integration): cover budgets_spend endpoint contracts (#44736)
* test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover budgets_spend endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): harden budgets_spend contracts per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): make spend report window midnight safe and drop redundant comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin organization spend report scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close budgets and spend review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop streamed cost header absence assertion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover streamed spend on linked budget tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): wait for async spend logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0f0c1bc80b
|
test(integration): cover teams endpoint contracts (#44705)
* test(integration): cover teams endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address review on teams contract tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drain shared upstream observations at fixture setup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove team block, member and bulk permission contracts at runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert email admin adds and whole membership rows for team members Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drive key update grant and revoke, Z-suffixed temp budget expiry, link LIT-9257 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close review gaps in teams endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): harden team member endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
148ca66ecb
|
test(integration): cover prompts_tags_caching endpoint contracts (#44686)
* test(integration): cover prompts_tags_caching endpoint contracts * test(integration): assert prompt lifecycle requests per environment * test(integration): pin non-latest prompt versions, label/path spellings and tag header/key spellings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover production PATCH, streamed and root /responses prompts, and memory create scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): ignore proxy model discovery probes in prompt upstream doubles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove tag update, peer prompt resync, memory team rows and Responses prompt spellings at runtime Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover memory metadata and key_prefix listing, multi-tag spellings for tag cache invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> |
||
|
devin-ai-integration[bot]
|
32f0d680d2
|
test(integration): cover google_genai endpoint contracts (#44680)
* test(integration): cover google_genai endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten google_genai contract assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover non-admin agent api_base guard and exactly-once spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expand Google GenAI route contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover adapter responseJsonSchema, tool-loop history, /models alias and anthropic stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin raw SSE id/retry framing and agent list/versions key guard Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): strengthen Gemini endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
90a503d98c
|
test(integration): cover videos_containers endpoint contracts (#44691)
* test(integration): cover videos_containers endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): end the first container page on an ownerless id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert container upload parts, SDK video extend, bare video ids, cost from map Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): catch a stale container list cache, cover video edit JSON, bracketed video refs and object input_reference Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin container optional fields, video client extras and the /videos alias on the wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin multipart character id decoding on video create as a BUG skip Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
117684cce5
|
test(integration): cover agents_a2a endpoint contracts (#44650)
* test(integration): cover agents_a2a endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin agent daily spend to the logged request day and assert cleanups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect agent daily activity newest day first Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover A2A 1.0 task aliases, agent id headers, declared access groups and per-token spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert full A2A stream, header, alias, task and spend contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert forwarded a2a-version header on A2A sends Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): release A2A stream gate only after a full SSE frame Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin A2A trace id, stream headers, denied alias and task methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin A2A card protocol version Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert the whole accepted push config response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
jesus-berri
|
fead47fa7b
|
fix(proxy): skip failure rows for expired Admin UI session keys on dashboard info routes (#45881)
* fix(proxy): skip failure rows for expired Admin UI session keys on dashboard info routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): mark DB-boundary write assertion as test-quality-ok Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
02feb67c20
|
test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037)
* test(integration): build worker stacks concurrently and record proxy boot phases The controller built every xdist worker's database, Redis, upstream and gateway proxy one after another before any test ran, about 95 s per extensions shard, and since every multi-worker suite is now isolated the same serial wait hits mcp, security and cost too. The stacks now start on a thread pool bounded by the CPU count. Each stack reads its endpoints and results directory from an explicit environment instead of a patched os.environ, every stack that started is registered for cleanup, and a failure in one stops the others before the error is raised Owned proxy startup.json now records the wall-clock spawn time, the worker count and when the Prisma CLI child ran, when the port started accepting and when the first worker answered readiness, and exit.json records when the stop started and ended, so the boot time can be split and overlapping gateway boots can be seen in the artifacts * test(integration): boot one uvicorn worker where a test does not assert cross-worker behaviour A 2-worker owned proxy takes about twice as long to become ready as a 1-worker one, because the uvicorn parent imports the app and each spawned worker imports it again. The passthrough upstream error visibility cases, the single-request S3 dedupe cases and the single-request spend log payload cases send one or two requests and assert on the response, the proxy log and the spend row, none of which depends on a second worker, so they now boot the default single worker Cases that do rely on two workers keep them, the concurrent S3 burst through a sink outage, previous_response_id replay, response cache hits, the concurrent spend bursts and upstream stop and recovery * test(observability): settle OTel excluded-services traces by a quiet period instead of a full window _trace_spans slept through its whole 15 or 30 s window on every call even when the trace had long stopped growing. It now polls and returns once the trace has spans and has not gained one for five tenant export intervals, and the module pins the tenant batch exporter's schedule delay to 1 s so that bound is set by the test rather than by the SDK default. Absence checks still run only after the trace stopped growing, and against a build that leaks postgres spans to the tenant they still fail These cases also boot one uvicorn worker now. None of them asserts anything about a second worker --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
7efce7f8e0
|
test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)
* test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
780c504e00
|
test: migrate tests/proxy_behavior into integration and unit tiers (#45939)
* test: migrate tests/proxy_behavior into integration and unit tiers (partial) * test(integration): run settings-dependent management tests on an owned proxy * test(integration): assert stored values in the rewritten management rows * test(integration): pin sibling team allocations against the org guaranteed throughput cap * test(integration): keep Prisma clients out of the shared integration conftest import path --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Yassin Kortam
|
3f16213906
|
feat(usage): add top users by spend and user activity views (#45310)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
9b53ee878f
|
perf(ci): isolate and shard integration suites (#44146)
* perf(ci): distribute unit cases and reuse security sweep connections * revert(ci): reject first speed trial after new failures * perf(ci): avoid repeated test discovery and parsing * perf(ci): isolate parallel security workers * revert(ci): restore measured security execution isolation * perf(tests): reuse source scans and deduplicate cache cleanup * perf(ci): restore parallel security with safe log batching * test(ci): isolate fixture state and schema comparisons * test(ci): make schema and scheduler fixtures deterministic * test(ci): drain queued logging before fixture teardown * test(ci): preserve logging delivery without idle worker restarts * test(ci): keep guardrail sinks available during parallel bursts * test(ci): isolate full proxy lifecycle from gateway imports * test(ci): preserve first failure responses and sweep evidence * test(ci): restore documentation URL settings between cases * test(ci): isolate open MCP tools in private gateways * test(ci): group private MCP cases and handle exited workers * test(ci): retain proxy exits and MCP failure details * test(ci): repair merge interactions with main Key the schema shard's full clone on artifact-name, since main removed the unit-flag input. Give each owned Redis its own log file again. Hand the realtime transcription tests main added a logging object whose dispatch handlers are awaitable, so the per-test logging drain no longer waits on a MagicMock task forever * perf(ci): shard owned integration cases and close lifecycle leaks * fix(ci): qualify shard execution and harden interrupted cleanup * fix(ci): bound worker supervision and own fault traffic * perf(ci): reuse unparametrized fixtures across bounded case groups * fix(ci): preserve boot fault allowances and complete exit records * test(ci): tolerate startup retries and retain readiness details * perf(ci): balance integration shards by recorded duration * fix(ci): load migration database helpers only for workers |
||
|
moe-berri
|
22ad3c5fff
|
refactor(lens): connect LiteLLM to the independent Lens service (#45529)
* feat(lens): consume the independent Lens service and shared UI * fix(lens): keep embedded setup stable and vendor UI before builds * chore(lens): pin embedded UI provenance to published Lens source * fix(lens): complete embedded UI extraction across CI builds * docs(lens): record latest main and removal-boundary validation * fix(lens): wire external chart credentials and ingestion modes * test(lens): cover adapter failures and fix extraction CI * docs(lens): refresh bundled chart setup instructions * test(lens): restore gateway adapter test package marker * build(lens): pin clean-cut UI and supported storage chart * test(ui): await guardrail scope menu before checking options * perf(lens): omit unused trace-team lookup from product requests * docs(lens): record final-image adapter permission checks * test(lens): assert delegated investigation authorization * refactor(lens): remove copied trace runtime and preserve spend logging * test(lens): verify independent gateway image upgrades and outages * test(lens): pin the chart qualification database image * fix(lens): update embedded UI metadata filters * test: wait for Lens chart pod identity convergence * fix(e2e): select Lens chart pods by deployment ownership * fix(e2e): select Helm migration ownership for Lens upgrades * test(lens): retain migration Jobs through rollout assertions * docs(helm): require existing database for migration hooks * test(lens): qualify release boundaries with embedded navigation update * docs(lens): record browser qualification for embedded tabs * fix(lens): remove unrelated CI and guardrail test changes * test(lens): give qualification tenants unique key aliases * feat(lens): embed the latest canonical Lens interface * fix(lens): qualify split inference through the gateway * fix(lens): preserve scoped auth and isolate delegated credentials * chore(lens): remove unrelated documentation and lint changes * fix(lens): bound concurrent forwarding buffers through response delivery * fix(lens): preserve OAuth2 dispatch without inference policies * chore(lens): adopt current shared onboarding UI * fix(lens): refresh shared setup UI and review context * fix(lens): bound uploads before gateway authentication * docs(lens): remove extraction evidence from the gateway repo * docs(lens): refresh shared setup prompt for paired releases |
||
|
yuneng-jiang
|
cddc7cde97
|
test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530)
* test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone |
||
|
devin-ai-integration[bot]
|
f09408dd37
|
test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362)
* test: replace 61 live logging and otel tests with offline unit and integration coverage * test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts. * test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request * test: isolate the prometheus latency test on its own deployment so counts do not depend on order * test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only * test: drain the redis test's logging and scope otel span checks to the test's own trace * test: drive the periodic slack flush without a wall-clock interval * test: scope router hook events to the test, script the db clock, split a nested comprehension --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
406514fcaf
|
feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs (#44659)
* feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs Adds general_settings.spend_logs_metadata_fields with mutually exclusive include and exclude lists. The filter runs on a copy of the row right before it is queued for Postgres, so daily spend rollups, budgets and callbacks still see every key. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): keep excluded auto-router savings keys out of published spend log metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): cover metadata retention across endpoints, failures, cache hits, batches, runtime updates and auto-router publication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for spend_logs_metadata_fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): filter metadata at DB write so guardrail usage sees full rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): poll guardrail daily metrics instead of reading once Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): drop timeout comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): read spend_logs_metadata_fields through typed general settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b7ecb05340
|
fix(proxy): mark background responses stale_expired when provider returns 404 (#41724)
* fix(proxy): mark background responses stale_expired when provider returns 404 Responses deleted upstream (store=false / ZDR rows dropped after provider retention) now move to stale_expired on the poll instead of being retried every cycle until the 7 day stale cleanup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop inline comment in check_responses_cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive _get_response seam in 404 stale_expired tests Assigning AsyncMock on the instance keeps the new tests inside the TQ002/TQ008 test-quality ceilings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only expire 404 responses polled through a resolved deployment A NotFoundError from the bare SDK fallback can mean a missing or misconfigured deployment, which a config fix inside the staleness window can still recover. Rows whose poll went through their router deployment are the ones the provider actually dropped, so only those move to stale_expired. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve the deployment before fetching so via_router binds once Drops the reinitialised loop flag flagged by review and modernizes the moved annotation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive the real response fetch in the poll 404 regression tests * fix(enterprise): expire background response rows on any provider 404 status The GET path maps the provider's 404 body to litellm.BadRequestError that still carries status_code 404, so an except on NotFoundError never fired and the poll job kept retrying the row every cycle. Key the terminal decision on the status code instead of the class * fix(enterprise): expire a background response only on a 404 naming it and skip a bad deployment entry per job * chore(enterprise): record the poll-cycle bound on the managed-object IN lists * test(integration): cover background response poll retirement on the real proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d5173e3d70
|
test: move 76 live top-level tests to offline unit and integration coverage (#45352)
* test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Hektor Jacynycz Garcia
|
157d96dcd0
|
fix(logging): log spend once for large non-streaming requests on the chat to Responses bridge (#44508)
* fix(logging): claim the async success dedup flag before the first await Nested @client wrappers on the chat -> Responses bridge schedule two async success tasks on one logging object. The dedup flag was checked at the top of _async_success_handler_body but only set after several awaits; the worker-thread base64 offload for messages >= 256 KiB yields there, so the second task passed the same check and the request was logged twice (two SpendLogs rows, key/team/daily spend counted twice). Fixes #44500 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(logging): serialise async success handlers per logging object instead of claiming early Claiming the dedup flag before the first await (previous commit) let a first handler that is cancelled before its callbacks run (e.g. by the logging worker's per-coroutine timeout) suppress the second one, losing the request's success accounting. Serialise non-streaming handlers on the same Logging object with an asyncio.Lock kept in a module-level WeakKeyDictionary: the second handler waits, then skips if the first logged, or logs itself if the first raised or was cancelled first. Streaming chunk logging is not locked. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(integration): pin one spend row and one key charge for large bridged chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the large request's spend row before asserting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): use fixed timestamps in async success dedup tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(logging): trim the async success dedup lock comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): reset background interaction dedupe under the success dedup lock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): use a fixed start time in the interactions logging helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: claim the async success log at schedule time instead of locking the handler Nested @client wrappers (chat over the Responses bridge, Anthropic Messages over the chat adapter) exit with one shared Logging object and each queued their own async success handler. The handler checked has_logged_async_success before its awaited work and set it after, so both could pass the check and the request was logged and billed twice. _schedule_async_success_logging now claims the log on the Logging object synchronously through claim_async_success_log, and a later wrapper returns without enqueuing. There is no await between the check and the decision, so no lock is needed. The per-object asyncio.Lock, the WeakKeyDictionary that held it and the locked block in async_log_background_interaction_completion are gone. Tests: the regression test moves to tests/unit/test_utils.py and drives two _dispatch_success_logging exits with one Logging object through the logging worker, asserting one log with the inner result. It fails without the claim. The cancelled-handler test is dropped since a second wrapper no longer retries. The background interaction completion test stays. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): pin the background completion event order and name the innermost wrapper in the claim docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send one bridged request and assert the inner Responses result is the logged one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
5147aefa8b
|
fix(lens): reuse trace reviews and consolidate findings across runs (#44778)
* fix(lens): reuse trace reviews and consolidate findings across runs * chore: sync schema.prisma copies from root * fix(lens): preserve partial reviews and bound budget admission * fix(lens): resolve CI regressions and clarify reused runs * test(lens): exercise review reuse in worker container smoke * fix(lens): show actual reviews when a run finishes * fix(lens): distinguish reuse plans and stop blocked scans * fix(lens): retain partial findings when model requests stop * docs(lens): clarify budget edits during active runs * fix(lens): publish findings only after reconciliation completes * fix(lens): guide insufficient-budget runs to budget settings * fix(lens): renew budget holds and bound admission waits * fix(lens): preserve provider errors during budget cleanup * fix(ci): update PgBouncer and sharp for current builds * fix(lens): serialize settlement and fence review checkpoints * fix(lens): defer generated Prisma client type import * test(lens): verify checkpoint and progress rollback together --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
nate-berri
|
b8239a9873
|
fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image (#44673)
* fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image Google prices Gemini 3.1 Flash Image at 1.1x on non-global endpoints for input, text output and image output, but the cost map row had no regional_endpoint_uplift_multiplier, so regional calls billed at the global rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(integration): cover regional uplift spend for gemini-3.1-flash-image * test(integration): read the proxy salt from the environment in the uplift spend cells --------- Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
nate-berri
|
922ea71fe5
|
fix(vertex_ai): apply regional endpoint uplift on image generation cost path (#44679)
* fix(vertex_ai): apply regional endpoint uplift on image generation cost path completion_cost had vertex_location but never passed it to the image generation cost router, so a regional_endpoint_uplift_multiplier on a Vertex image row would be ignored. No image row carries the multiplier yet, so nothing is misbilled today. Pass the location through to the Vertex image calculator for both the token-based price and the per-image fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(vertex_ai): cover regional image cost through the proxy logging path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(images): hand image_edit's vertex_location to its cost resolver * test(integration): audit Vertex image regional uplift billing * test(integration): require every spend row after a proxy restart * test(integration): reject extra spend rows after a proxy restart --------- Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moe-berri
|
d232b008bc
|
test(lens): use current worker protocol in billing integration (#44760) | ||
|
devin-ai-integration[bot]
|
a1a42768c1
|
fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560)
* fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(spend-tracking): share the short-lived miss cache write Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover key alias recovery after spend log lookup failures across usage routes * test(spend): bound outage alias lookups per miss window instead of a fixed count * fix(spend-tracking): treat any spend-log lookup failure as a short-lived miss The Prisma client raises a plain AttributeError when the database drops the connection mid-query, so the PrismaError catch let it through and the whole usage call answered 500. Any failure now keeps the 30 second backoff only, and the integration proxy patches its test entitlement at import so uvicorn's spawned workers inherit it * test(integration): audit spend-log metadata recovery under timeouts and dropped connections Cover the daily activity routes, the usage AI chat, the Vantage and CloudZero dry runs and exports under a locked spend-log table and under a database connection dropped mid-lookup, on a two-worker proxy, with the recovery after the outage asserted through the proxy's own miss TTL. Add a dropped_connection_relay that closes only the connection whose bytes carry a trigger, so a cell can drop the one connection the recovery query runs on while the rest of the pool keeps serving. Rewrite the sweep and JWT cells for the merged main: the export route reads metadata by SQL join and never calls the recovery, the search routes answer key rows and find deleted keys by alias, and the daily-spend owner recovery names the user while the alias stays blank. The sweep cell now times out a second lookup under the same lock, which pins the keys blank on the merge base and recovers on this branch. * test(integration): match a dropped-connection trigger split across two reads The dropped-connection relay checked each TCP read on its own, so a SQL marker that straddled two reads never tripped it and the outage cells would run without the outage they meant to exercise. Carry the tail of the previous read into the next check, as the held-statement relay already does, and pin that with a unit test that splits the trigger across two writes. * test(integration): scan relay triggers through an in-process helper The dropped-connection relay now matches its SQL trigger through a TriggerScanner that carries the previous read's tail, and the unit test exercises that scanner directly instead of opening loopback sockets, which tests/unit forbids. The relay's end to end behavior stays covered by the integration cells --------- Co-authored-by: gabriele <gabriele@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3286782dea
|
fix(caching): key response cache by router model group in litellm_metadata (#44542)
* test: align integration fixtures with current behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the spend flush before asserting its trace placement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): key response-cache entries by model group from litellm_metadata /v1/responses routes through the router with model_group in litellm_metadata, which the cache key ignored, so identical requests to different model groups sharing one underlying model hit each other's cached responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1d52985d03
|
fix(ci): repair the security sweep and Lens billing integration tests for ROI, JWKS, and release identity changes (#44529)
* fix(ci): keep the security sweep off the observed ROI GitHub route and give Lens integration tests a release identity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): expect the credential JWKS export to 404 for non-federation credentials in the security sweep Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
1a7023366f
|
feat(roi): measure shipping velocity, quality, and recorded spend (#44426)
* feat(ui): prototype observed engineering ROI dashboard * feat(roi): replace effort estimates with measured repository metrics * fix(roi): finish connection recovery and generated API contracts * fix(roi): show merged changes before accounts are linked * fix(roi): preserve selected report tab across refreshes * fix(roi): recover app authorization and keep detail values readable * fix(roi): reuse the shared OAuth HTTP client * feat(roi): combine providers and compare equal reporting periods * docs: explain ROI metrics for first-time readers * fix(roi): preserve connections and scheduled reports during setup * ci(roi): assign database contracts to the active Postgres shard * fix(roi): preserve issue counts and normalized connections * fix(ui): compact ROI dashboard header and metrics * fix(ui): show ROI repository count with expandable list * fix(ui): wrap ROI controls within narrow panels * fix(roi): restore sample report preview and simplify setup |
||
|
devin-ai-integration[bot]
|
7b432d78d2
|
fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model (#44341)
* fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): streamed alias matching a capability rule bills the deployment price Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): assert every streamed chunk carries the client alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): log the client alias on the priced streamed response, the same as non-streamed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a76ba8c01e
|
revert(cost): revert "fix(cost): price rule-only model names at the deployment's rate" (#44144) (#44335)
This reverts commit
|
||
|
moe-berri
|
caee45fed4
|
feat(roi): add GitLab sources and branch cost attribution (#44324)
* feat(roi): support GitLab and tagged branch costs * fix(roi): count tagged branches independently of estimation status * test(roi): capture live GitHub and GitLab report validation * fix(roi): open estimate details at the start * fix(roi): clarify cost views and unify report layout * feat(roi): showcase per-PR costs in the sample report * fix(roi): separate report tabs and preserve branch cost attribution * fix(roi): preserve demo previews and align progress spacing * fix(roi): isolate demo loading and parallelize fork lookups Preserve active sync status when source changes finish saving, keep live reports available when demo requests fail, and cover each review regression * fix(roi): separate demo and live loading states Clear the demo URL on fallback, wait for live requests on exit, and retain request errors until the corresponding operation recovers * fix(roi): ignore refreshes from a previous source * fix: trust gateway context for ROI estimator exclusion * fix: preserve historical ROI estimator exclusion |