mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
67643606ab
|
test(e2e): add reproducers for passthrough and model budget gaps (#34657)
* test(e2e): add failing reproducers for two open gateway bugs
Both tests assert the behavior a customer expects and both are red today. They
are reproducers, not regressions: the product is wrong, not the tests.
Native passthrough returns almost none of the operational headers the managed
route does. A /gemini/ generateContent call comes back with three x-litellm-*
headers and no x-ratelimit-* at all, against sixteen and four on
/v1beta/models/{m}:generateContent for the same prompt, and critically it omits
x-litellm-response-cost. Customers front provider-native traffic through this
route and read those headers to reconcile spend and pace themselves, so native
traffic is currently invisible to the tooling that covers every other route.
/budget/update rejects any model_max_budget with a 500. The reported symptom was
model ids containing dots, and that reproduces (prisma raises "Unexpected
`-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL
query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the
plain name gpt4o fails too, on a separate "model_max_budget should be of any of
the following types: Json" type mismatch at budget_management_endpoints.py:173.
Omitting the field returns 200. The test drives both names so the failure says
whether per-model budgets are broken outright or only for punctuated ids; today
it stops on the plain name, which is the wider bug.
* test(e2e): add reproducer for unenforced end-user per-model rate limits
model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new
stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1,
"max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end
user, drive three calls as that user, and all three return 200. The limit is
accepted, persisted, and then ignored.
The same shape already works when the budget hangs off a key, which is what
makes this quietly dangerous: the API gives every indication the cap is in
force. A customer using it to hold one end user to a slow rate on a shared key
gets no throttling at all.
Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit
the route already accepts, BudgetNewBody and create_budget carry
model_max_budget, and create_customer can attach an existing budget_id rather
than only an inline max_budget.
Red today, for the reason in the assertion message.
* test(e2e): tighten model_max_budget reproducers and drop in-loop closure
Trim the reproducer docstrings to the contract they assert, keeping the
failure messages that document each red-by-design bug. Replace the nested
per-model closure in the /budget/update test with a module-level predicate
and a per-model helper so nothing closes over a loop variable, and fix the
import order the merge left unsorted.
* test(e2e): skip the three reproducers while their gateway bugs stay open
The passthrough header contract, /budget/update model_max_budget, and
end-user per-model rpm enforcement reproducers all still fail against
staging by design. Skip each with the product gap named so the combined
suite can gate merges on green while the collector keeps reporting the
cells as uncovered.
* test(e2e): validate model budget response contracts
* refactor(e2e): unify model budget schema
* refactor(e2e): reuse shared model budget type
|
||
|
mubashir1osmani
|
ec8088f064
|
test(e2e): vendor API testing coverage (#34557)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly |
||
|
Yuneng Jiang
|
472dd2716f
|
revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)"
This reverts commit
|
||
|
mubashir1osmani
|
dcb4e5033c
|
test(e2e): vendor API strategy coverage across endpoints (#34649)
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly * fix(e2e): address greptile and veria review on vendor strategy suite Store search tool keys as os.environ refs and resolve them in SearchAPIRouter. Tighten validation helpers and assertions so 5xx/empty/unrelated failures no longer pass coverage cells * fix(e2e): drop search_api_router os.environ expansion from vendor suite Keep the PR test-only. Search tools register without an api_key so the proxy falls back to its own PERPLEXITY/TAVILY env, same pattern as a2a. * test(e2e): drop search e2e suite from vendor strategy PR Remove the /v1/search coverage file and its registry rows so this PR no longer carries search endpoint testing. |
||
|
yuneng-jiang
|
fcec1488e2
|
feat(proxy): add GET /management/v1/budgets (#35310)
* feat(proxy): add a generic list contract for management/v1 entity lists Paging, sorting, filtering and search for an entity collection, declared once as a ListSpec and served by handle_list. The route injects a ListExecutor that owns its table, so this module never imports Prisma. The caller's scope is derived from the caller alone and ANDed with whatever they filtered on, so a query parameter can only narrow what they may read. This is the shared half of the budgets list; it lands here so the endpoint has something to register against, and drops out when the framework arrives on its own branch. * feat(proxy): add GET /management/v1/budgets The Budgets page reads /budget/list, which returns the whole table as a bare array with no way to page, sort or filter it. A customer with enough budgets to fill the page has no way to find one. Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order newest-first with budget_id breaking ties, search on budget_id, and filters for budget_duration, max_budget and created_at. budget_duration is deliberately not sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts "30d" ahead of "7d". tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a pydantic model on the way out and serialize as JSON numbers. A caller without admin view is refused 403 as a problem document rather than served an empty page. /budget/list is untouched. * fix(proxy): rework the budgets list onto the merged list contract PR #35308 landed a different shape than this branch was written against: `where` is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec` carries both the row and the wire type, and `where_sql` / `order_by_sql` render for a raw-SQL executor. The budgets executor now queries through `query_raw` the way the spend logs facet does, selecting only the columns it serves. Also casts datetime binds in `where_sql`. They cross into the query engine as JSON, so an uncast placeholder arrives as text and Postgres refuses `timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering 500. The cast reads the bind as an instant and drops it to naive UTC to match Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies. * refactor(proxy): fold the predicate renderer instead of recursing recursive_detector flags `_render_all`, and the flag is fair: it recursed once per predicate, so the stack grew with the number of filters on the request for no reason. Walking a predicate list is a running bind index, which is a fold. `_render` still re-enters for `AnyOf`, but its clauses are plain comparisons built by `?q=`, so that nesting is one level deep and no caller can drive it deeper. |
||
|
mubashir1osmani
|
64fc19d61a
|
fix(e2e): stop tests from breaking the shared proxy for every suite after them (#34664)
* fix(e2e): stop the cache-settings test from persisting a degraded Redis config TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache settings and wrote them back, intending a no-op. Its capture modelled only type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and `redis_startup_nodes`. That is not recoverable on its own. `/cache/settings` persists what it receives into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and init_cache_settings_in_db re-applies it on a timer, so a restart does not clear it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext standalone node and every Redis call blocks to socket timeout. On the affected deployment that took out rate limiting entirely (the v3 limiter is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag, per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`, whose last statement syncs guardrails and never ran. 60 of 72 failures in one run traced back here. The settings blob is now round-tripped verbatim via a RootModel over an exhaustive value union, so a subset cannot be written. Two guards make a regression fail loudly at this test instead of silently downstream: - refuse to write when GET reports redis_type=cluster but omits redis_startup_nodes, which is the exact precondition for persisting a downgrade. GET resolves the stored row overlaid with REDIS_* env and never reads YAML, so a cluster configured only in YAML cannot round-trip here - compare /cache/ping before and after, so a write that breaks connectivity fails this test rather than every suite that follows The underlying product defect is filed as LIT-4816: GET cannot express the effective config, and a partial POST is allowed to downgrade transport. This change only stops the suite from triggering it; the Admin UI can still do so. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param Two failures that had nothing to do with the guardrail or route under test. create_bedrock_guardrail registered with default_on=True, which applies the guardrail to every request the proxy serves. The upstream ApplyGuardrail call was answering 403, and that came back to unrelated traffic as `403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough headers test alongside the bedrock one. The harness already supports the per-request `guardrails` selector, so the guardrail is now registered opted out of default_on and selected by the test that wants it. A broken upstream guardrail fails its own test instead of whatever else is running. Note this only contains the blast radius; the 403 itself still needs the bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its own until that is sorted. The OpenAI passthrough body sent `max_tokens`, which newer models reject with "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead." Passthrough forwards the body untranslated, so drop_params does not apply and the body has to satisfy OpenAI's contract directly. vllm_chat keeps max_tokens, which vLLM accepts. basedpyright clean (0 errors) under the e2e gate. * fix(e2e): drop the pinned a2a api_key that broke every message/send #34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent. The a2a bridge forwards the agent's litellm_params straight into litellm.acompletion() without expanding "os.environ/" indirection, so that literal string was sent upstream as x-api-key and every message/send failed with `AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`. Omitting api_key restores the normal provider resolution: litellm reads ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what the agent-owner flow depends on and what the suite did before #34512. Verified against a live proxy, same agent shape each time: api_key omitted -> message/send 200 api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key api_key <literal key> -> message/send 200 and the key itself is valid (direct call to api.anthropic.com returns 200), so this was indirection that never got expanded rather than a bad credential. This accounts for four failures (test_semver_protocol_version_registers_and_serves, test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape, test_pinned_v1_0_serves_nested_message_shape). They were previously reported as `403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail short-circuited the request before it ever reached the bridge and hid this. The bridge silently ignoring "os.environ/" in agent params is a product defect in its own right, filed separately; anyone configuring an agent credential that way through the UI hits the same wall. basedpyright clean (0 errors) under the e2e gate. * test(e2e): make the load suite less aggressive against a shared proxy 750 users at spawn rate 50 saturated the request path hard enough to distort the latency-sensitive suites sharing the same proxy, and it spends real provider money at that rate. Drop to 200 users at spawn rate 20. The RPS floor moves with the user count rather than staying put, so the assertion keeps its meaning instead of becoming a formality: 355 RPS over 750 users is ~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a similar pass margin. A request-path regression still trips it. All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE, E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run. Note the recorded failure for this test was "no requests completed in 60s", which was the gateway wedged on unreachable Redis rather than a throughput regression; this change is about not perturbing its neighbours, not about that failure. * fix(e2e): make the reasoning-tokens assertion exercise a request that reasons test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then asserted reasoning_tokens > 0. The model answers that directly without reasoning, so 0 is correct behavior and the assertion was testing the model's discretion rather than litellm's reporting. Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how the test provisions its model: reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0 reasoning_effort=high, the prompt used here -> reasoning_tokens=114 Raised to high effort with a prompt that requires a proof plus a search, so the field under test is actually populated and the assertion fails only if litellm stops surfacing it. While confirming this I also checked prompt caching, which needed no change: cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix against a dedicated deployment. An earlier reading of 0 was an artifact of probing a fan-out alias whose requests land on different deployments, not a caching defect. * test(e2e): skip the files-list test while LIT-4820 is open GET /v1/files does not include a just-uploaded file. The upload returns 200 and GET /v1/files/{id} resolves it, but the listing never contains it: the returned set stays fixed at 27 entries whose newest created_at is roughly ten hours older than the upload, on both the managed (/v1/files?model=) and provider-scoped (/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window. Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the suite red and masking a new regression somewhere else in the same test. The assertion is left exactly as it was on purpose. It encodes the contract we actually want, that a file retrievable by id is also enumerable, and anything that lists files (a UI picker, cleanup tooling that lists then deletes and would therefore leak provider-side files) depends on it. Relaxing it to get green would delete the signal. The skip reason says so and links the ticket, and the ticket records that removing this marker is part of its definition of done. Matches the existing pattern in this file, where test_unified_file_and_batch_create skips with a reason citing LIT-3266. While skipped, the registry cell llm.files.openai.list.nonstream.works has no passing covering test, so files-list coverage reports as uncovered rather than passing, which is the honest state. * fix(e2e): parse Sentinel node lists in the cache-settings model The value union covered scalar lists and lists of mappings, but not lists of lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on a Sentinel deployment pydantic rejected the response: sentinel_nodes.list[dict[str,...]].1 Input should be a valid dictionary [input_value=['localhost', 26380]] The round-trip test reads GET /cache/settings before it writes anything, so that rejection failed the test at the read, before any assertion ran. A Sentinel deployment would have looked like a broken cache-settings route rather than a model too narrow to parse a documented shape. A list element may now be a scalar, a list or a mapping, which covers both node shapes without special-casing either and tolerates a heterogeneous list instead of rejecting the whole response. Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain node, url mode with a null discrete field) plus transport() key selection. Confirmed it fails on the previous union and passes on this one: old union -> 1 failed, 4 passed (the sentinel case) new union -> 5 passed * test(e2e): remove the cache-settings round-trip test The test could not fail for the thing it claimed to test, and could break the deployment it ran against. Both halves of that are worth stating. It read the live settings, wrote back identical values, and asserted the read-back matched. If POST /cache/settings were a complete no-op that returned 200 and touched nothing, GET would still return the values read a moment earlier and the test would pass. It verified that GET is stable, not that the route persists anything. Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig, that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a plaintext standalone client and every later Redis call blocks to socket timeout. On 2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing, Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved, and guardrail sync never ran. Guarding the previous shape was not sufficient. Writing the blob verbatim plus a cluster precondition and a /cache/ping check narrowed the hazard but did not remove it, because GET cannot express the effective config: it resolves the stored row overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row written that drops it. No round-trip through this route is safe on a shared proxy. Removed with the models and helpers it owned, and TestCacheSettingsModel with them since it existed only to protect that parsing. The registry row mgmt.cache_settings.update.happy_path stays, now carrying the rationale for why it is deliberately uncovered and what a safe test would require (an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade transport). Coverage therefore reports this cell as a gap, which is the honest state. Collector passes --strict; the module still collects 11 tests. |
||
|
mubashir1osmani
|
28f012bb52
|
test(true_rabbit): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping (#33843)
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly. * test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL, TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path, Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm chat, and Nova Sonic realtime. Registry cells updated for the new markers. * test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and Langfuse driver models now use Anthropic haiku so local runs stay green when Gemini daily quota is exhausted. * test(e2e): drop Langfuse spend suite; feature is being deprecated Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the langfuse-only conftest driver/credentials fixtures. * test(e2e): fold provider/batch feature tests into their endpoint suites Keep the e2e layout endpoint- and suite-scoped instead of one file per provider or feature Move the virtual-key auth case into access_control/test_access_control_e2e.py as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone test_virtual_key_auth_e2e.py Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py. The hosted_vllm batch case is skipped for now since it needs a live vLLM server (HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant Merge the cohere, gemini and hosted_vllm chat cases into llm_translation/test_chat_completions_regression_e2e.py so /chat/completions coverage lives in one endpoint file, and repoint the coverage_registry source fields to the new homes Move the shared CacheControl / TextBlock / RichMessage request blocks into the root models.py (re-exported from endpoints_client) so quota_management can use them without a cross-suite import, which also clears the basedpyright errors in test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed json.loads path * test(e2e): address review feedback and re-home virtual-key coverage Replace the tautological Bedrock assume-role batch id assertion (`startswith(...) or batch.id`, always true) with a managed-id shape check, since the unified target_model_names path re-encodes the id rather than returning a raw ARN Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no longer consume the key's sole request unit before batch create runs; the batch create then clears the generic per-request limiter and the batch limiter is what returns the "Batch rate limit exceeded" body the assertions check Set exercised_on to [] on the pass-through header test; it drives a pass-through endpoint, not /chat/completions Move the virtual-key valid_allows / invalid_denied cells from other.yaml to mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management, and point its covers marker at the new ids |
||
|
devin-ai-integration[bot]
|
b83c60b9b7
|
test(e2e): cover credential-backed /v1/messages request (#33863)
* test(e2e): cover credential-backed /v1/messages request Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): use runtime Anthropic credential Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
mubashir1osmani
|
a05a1eef94
|
fix(ui): scope key models dropdown options to the key's team (#32382)
* fix(ui): scope key models dropdown options to the key's team A teamless key no longer offers the all-team-models option in the create and edit forms; the backend expands that sentinel to the full proxy model list when no team is attached, which is rarely what the user intended. A team key no longer surfaces the all-proxy-models sentinel that leaks in verbatim when the team's own model list carries it; the dropdown keeps All Team Models plus the team's individual models. Adds browser coverage to the management e2e suite: playwright (an optional dependency behind importorskip) drives the proxy-served dashboard at /ui, asserts the dropdown options a real user sees for teamless and team keys on both create and edit, and walks the create modal end to end, reading the persisted key back through /key/info. * fix(ui): offer all-proxy-models on teamless keys in the models dropdown A teamless key has no team allowlist to inherit, so the dropdown now offers All Proxy Models in place of All Team Models on both the create and edit forms, with the same exclusive-selection handling. Component and browser e2e tests updated to pin the swapped option pair; the teamless create case now also walks the modal end to end and reads the persisted key back through /key/info. * test(ui): update no-team key creation spec to pick All Proxy Models The create modal no longer offers All Team Models without a team; the teamless path now offers All Proxy Models, which is what this spec exercises * fix(ui): gate All Team Models on the team object being loaded When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): filter all-proxy-models from teamless model fetch in key edit form The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
a43f128a74
|
test(e2e): add coverage registry and collector (#32304)
Introduce the e2e coverage denominator: 282 behavior cells across the six tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance, Logging & Guardrails, Other), one validated YAML row each, plus a collector that diffs the registry against @pytest.mark.covers markers and reports coverage per module. The registry rows validate against a pydantic discriminated union so a row cannot carry a field from another module. The collector is static: a collect-only pass reads the markers, so it runs no test and needs no live proxy. Register the covers marker suite-wide so that pass works under --strict-markers. This is a draft for review. Tiers are proposed rather than signed off, and a few cells still need a support check or a prune. |