litellm/tests/e2e
devin-ai-integration[bot] ca4169d20a
feat(providers): add CoralBricks as a native provider (#44700)
* feat(providers): add CoralBricks as a native provider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): harden cost map consistency tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): type cost helpers, read cost maps from disk, cover streaming spend

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): drop Final on loop-rebound names

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): add the provider audit cells, the Add Model browser cell and the README row

* test(coralbricks): derive UI spend from model info, type browser rows, always release held upstream

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): release held upstream before the wire server joins its threads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): number forwarded chats with itertools.count instead of a mutable list

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against deepseek-v4.1-flash-fast-fp4

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(coralbricks): pin retries, await the respawned worker, read the UI rate from the catalog

The retry cells now set num_retries on the deployment, since the CI rig posts
router_settings.num_retries 0 after readiness and the router default of 2 no
longer applies there. The SIGKILL chaos cell waits for the respawned worker's
startup before the owned proxy is stopped and derives its timeout from the
harness budget. The Add Model spec waits for the catalog the form loads, reads
the expected rate from it, and finds the deployment through /v1/model/info,
which does not paginate

* fix(coralbricks): price the current model names and the new DeepSeek input rate

CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4
to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut
DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read
from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache
write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input
plus cacheWritePerM.

The test config builders now build new dicts instead of editing the parsed
YAML, and the cost-map test locals carry Final.

* fix(coralbricks): keep pricing for the deprecated fp4 slugs CoralBricks still serves

* fix(coralbricks): list the catalog rows for wildcard deployments and derive the test's input limit from the row

* fix(coralbricks): price the deprecated glm-5.3 and deepseek-v4.1-flash slugs

---------

Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-12 21:56:59 +00:00
..
a2a test(e2e): tag a2a, access_control, other, secret_manager and migrations tests with Subject metadata (#44965) 2026-10-07 12:11:55 -07:00
access_control fix(proxy/auth): expand all-team-models sentinel for project model access checks (#43734) 2026-10-10 15:49:27 -07:00
batches test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
claude_code test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530) 2026-10-09 11:40:14 -07:00
coverage_registry test: migrate proxy security, schema migration and MCP legacy tests (#45937) 2026-10-11 16:17:24 -07:00
gateway test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur (#42503) 2026-09-22 18:02:32 -07:00
guardrails test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
llm_translation fix(bedrock_mantle): send max_completion_tokens for gpt-5.6 and gpt-6 chat completions (#43982) 2026-10-11 18:28:48 +00:00
logging test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
management feat(teams): let proxy admins grant team admins the right to raise their team budget (#45404) 2026-10-09 13:39:08 -07:00
mcp test: migrate proxy security, schema migration and MCP legacy tests (#45937) 2026-10-11 16:17:24 -07:00
migrations test(migrations): assert lock-timeout retry and exhaustion instead of an unbounded wait (#45955) 2026-10-11 17:36:47 +00:00
other test(e2e): tag a2a, access_control, other, secret_manager and migrations tests with Subject metadata (#44965) 2026-10-07 12:11:55 -07:00
perf test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) 2026-10-12 13:54:03 -07:00
quota_management ci: run every CircleCI check from the integration workflow (#45938) 2026-10-11 07:39:16 -07:00
router test(e2e): tag router, batches and mcp tests with Subject metadata and record client steps (#44964) 2026-10-07 10:32:58 -07:00
secret_manager test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
ui feat(providers): add CoralBricks as a native provider (#44700) 2026-10-12 21:56:59 +00:00
AGENTS.md test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) 2026-10-12 13:54:03 -07:00
conftest.py test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) 2026-10-12 13:54:03 -07:00
CONTRIBUTING.md ci: run every CircleCI check from the integration workflow (#45938) 2026-10-11 07:39:16 -07:00
e2e_config.py test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) 2026-10-12 13:54:03 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): record steps for raw transport calls and poll helpers (#45150) 2026-10-07 15:21:26 -07:00
e2e_metadata.py test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) 2026-10-06 17:03:35 -07:00
fixture_bundle.py test: add strict stateless provider replay identity 2026-09-14 16:55:43 -07:00
fixture_canonical.py feat(e2e): key the provider cache per test and mount Bedrock behind it 2026-09-16 02:15:46 -07:00
fixture_mode.py fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
fixture_profile.py test: preserve strict replay numeric spelling 2026-09-14 17:11:56 -07:00
idp.py test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) 2026-10-06 17:03:35 -07:00
idp_realm.json test(e2e): harden JWT fixtures and cover management lifecycles 2026-09-11 16:36:12 -07:00
junit_properties.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
lifecycle.py test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) 2026-10-06 17:03:35 -07:00
memory_readings.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
models.py fix(bedrock_mantle): send max_completion_tokens for gpt-5.6 and gpt-6 chat completions (#43982) 2026-10-11 18:28:48 +00:00
otel_client.py test(e2e): record steps for raw transport calls and poll helpers (#45150) 2026-10-07 15:21:26 -07:00
PROVIDER_CACHE.md test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
provider_cache.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
provider_cache_redis.py chore(e2e): report the key components behind a mount that never converges 2026-09-16 15:01:08 -07:00
provider_cache_routing.py revert(e2e): unmount Gemini, its api_base means two things 2026-09-16 08:37:25 -07:00
provider_edge.py test(e2e): record steps for raw transport calls and poll helpers (#45150) 2026-10-07 15:21:26 -07:00
provider_edge_bedrock.py feat(e2e): cache the responses and embeddings endpoints behind the edge 2026-09-16 02:34:09 -07:00
proxy_client.py feat(e2e): record each e2e test's steps, starting with ProxyClient (#42393) 2026-09-30 19:33:53 -07:00
pytest.ini test(e2e): rename load to perf and consolidate the opt-in into E2E_PERF_TESTS (#46077) 2026-10-12 13:54:03 -07:00
stack_lock.py test(e2e): run the memory cell alone on the shared stack (#42518) 2026-09-22 15:13:37 -07:00
transport.py test(e2e): record steps for raw transport calls and poll helpers (#45150) 2026-10-07 15:21:26 -07:00