Commit graph

8 commits

Author SHA1 Message Date
devin-ai-integration[bot]
514bc181d6
test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)
* test(integration): streamed Bedrock Messages usage cost equals the recorded spend (Pylon #6667)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): echoed cost-map model info is not persisted as deployment overrides (Pylon #6844)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): reset sweep runs on one pod per tick while replicas share the lease (Pylon #6521)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bedrock post-call guardrail scans streamed Anthropic Messages tool use without 500 (Pylon #6503)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): realtime cached audio tokens bill at the audio cache-read rate (Pylon #6704)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): legacy GET /spend/logs returns at most the 10000 most recent rows and flags truncation (Pylon #6752)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): itemize Responses API cache write tokens as cache creation cost (Pylon #6454)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): migration entrypoint deploys pending migrations before proxy startup (Pylon #6649)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): opted-in team keys stop at the owner's personal budget (Pylon #6641)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): guardrail information stays in the spend log when the caller sends metadata on /v1/messages (Pylon #6614)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): key model allowlist is enforced on Bedrock passthrough routes (Pylon #6419)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): JWT mapped key backfills a null user email from token claims (Pylon #6266)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): scheduled budget reset recovers from a transient DB transport failure (Pylon #6582)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): team key lists models granted through a team access group (Pylon #6044)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): JWT subject without team claim lands in the configured default team (Pylon #5895)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): end-user spend lands for a key without user_id when the auth cache is Redis (Pylon #6021)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): stale-low redis counter still blocks team member over budget (Pylon #5824)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): prompt-carrying spend rows are written in byte-bounded statements (Pylon #6083)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): logs UI session_total_spend sums every round of a multi-round session (Pylon #5928)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): config.yaml guardrails are served by the guardrail usage detail and overview (Pylon #5813)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): plain chat request skips the object permission lookup (Pylon #5965)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): register july accounting regression contracts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): isolate cost map override clear on owned proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): make reset lease claim and db relay refusal deterministic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): poll pg_stat settle, bound unbanned relay refusals, clear reset lease on teardown

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bound relay refusals so the budget sweep can reconnect

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 09:51:56 -07:00
devin-ai-integration[bot]
65468219db
test(integration): regression tests for August cost tracking and budgeting bugs (#42622)
Some checks are pending
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Terraform Modules / fmt, validate, test (aws) (push) Waiting to run
Terraform Modules / fmt, validate, test (gcp) (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Waiting to run
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Waiting to run
* test(integration): saving echoed model_info never persists cost map pricing as a deployment override (Pylon #6870)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): budget_duration change on /budget/update recomputes budget_reset_at (Pylon #6913)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): count_tokens on a budgeted key reserves no budget and a later completion still succeeds (Pylon #6966)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): /cost/estimate reports configured prices for a deployment absent from the cost map (Pylon #7014)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): cache the team member default budget in Redis as JSON (Pylon #7180)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): aggregated team daily activity reports whole-range team spend in one page (Pylon #7224)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): failed daily user rollup commits are retried so spend report and daily activity agree (Pylon #7268)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): CLI session token without org_id is charged to and capped by the team organization budget (Pylon #7291)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): gemini passthrough success releases its budget reservation from the spend counter (Pylon #7295)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): batch retrieval spend row sums reasoning tokens and counts output and error file failures (Pylon #7341)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): uncostable batches retire from the cost poll page so newer batches are costed (Pylon #7342)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): charge a team member added without any budget on its membership row (Pylon #7363)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): failed dispatched requests keep estimated input tokens in spend logs (Pylon #7519)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bedrock passthrough converse guardrail ignores tool definitions (Pylon #7524)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): explicit null budget_duration on /team/new is not replaced by default_team_params (Pylon #7536)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): PATCH /organization/update with a null limit clears it (Pylon #7577)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): ultrafast service_tier bills ultrafast rates without leaking pricing fields upstream (Pylon #7587)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): keep the selected model in the response and spend log for an Azure Model Router alias (Pylon #7636)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): disconnected Bedrock /v1/messages stream still bills terminal usage (Pylon #7685)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): databricks cached prompt tokens bill at cache rates (Pylon #7738)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): completed batch spend row records reasoning tokens and error file failures (Pylon #7928)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): bill OCR annotation pages at annotation_cost_per_page (Pylon #7958)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): in-flight count tokens request reserves no key budget so a completion still reaches the provider (Pylon #7307)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): fail-closed key rejects known estimate over remaining budget before provider (Pylon #7691)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): streamed /v1/responses success callbacks keep provider response headers (Pylon #7775)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "test(integration): fail-closed key rejects known estimate over remaining budget before provider (Pylon #7691)"

This reverts commit 910348be7a.

* test(integration): reconcile contracts manifest for bundled regression tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): isolate proxy config writes in bundled regression tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): address review feedback on budget reset bounds and callback batch accumulation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): avoid rebinding the cache identity accumulator

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): assert forwarded messages per cache identity call

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): make budget reset and team default tests deterministic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 04:13:03 +00:00
Joshua Valluru
479360ae87 test(mcp): restore scoped execution and credential isolation regressions 2026-09-19 16:41:31 -07:00
Mateo Wang
57c757e9d7
Revert "test(mcp): verify scoped execution and OAuth credential isolation" 2026-09-19 16:10:47 -07:00
Joshua Valluru
daff22a884 test(mcp): grant the guardrail control team its server 2026-09-17 17:50:01 -07:00
Joshua Valluru
640b0e5fa9 test(mcp): discover concrete tools through a catalog key 2026-09-17 17:41:57 -07:00
Joshua Valluru
15b45839e1 test(mcp): enforce security regression contracts through live gateway 2026-09-17 17:37:40 -07:00
Yuneng Jiang
c00f1b4a5c
test: add extension and browser integration contracts
Some checks failed
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
2026-09-16 13:15:53 -07:00