a05a1eef94
* fix(ui): scope key models dropdown options to the key's team A teamless key no longer offers the all-team-models option in the create and edit forms; the backend expands that sentinel to the full proxy model list when no team is attached, which is rarely what the user intended. A team key no longer surfaces the all-proxy-models sentinel that leaks in verbatim when the team's own model list carries it; the dropdown keeps All Team Models plus the team's individual models. Adds browser coverage to the management e2e suite: playwright (an optional dependency behind importorskip) drives the proxy-served dashboard at /ui, asserts the dropdown options a real user sees for teamless and team keys on both create and edit, and walks the create modal end to end, reading the persisted key back through /key/info. * fix(ui): offer all-proxy-models on teamless keys in the models dropdown A teamless key has no team allowlist to inherit, so the dropdown now offers All Proxy Models in place of All Team Models on both the create and edit forms, with the same exclusive-selection handling. Component and browser e2e tests updated to pin the swapped option pair; the teamless create case now also walks the modal end to end and reads the persisted key back through /key/info. * test(ui): update no-team key creation spec to pick All Proxy Models The create modal no longer offers All Team Models without a team; the teamless path now offers All Proxy Models, which is what this spec exercises * fix(ui): gate All Team Models on the team object being loaded When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): filter all-proxy-models from teamless model fetch in key edit form The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| collector.py | ||
| guardrail.yaml | ||
| llm_conversational.yaml | ||
| llm_nonconversational.yaml | ||
| logging.yaml | ||
| mcp.yaml | ||
| mgmt.yaml | ||
| other.yaml | ||
| README.md | ||
| registry.py | ||
| reliability.yaml | ||
| schema.py | ||
| test_collector.py | ||
e2e coverage registry
This directory is the denominator for e2e test coverage: the set of behaviors we
want covered, one row per behavior, checked into the repo so coverage is a number we
can track instead of a guess. It implements the plan in the "E2E Coverage Tracking"
note; the naming grammar lives in tests/e2e/CLAUDE.md.
The model
A cell is one customer-noticeable behavior a single e2e test can assert pass/fail
on, for example llm.chat_completions.bedrock_converse.tool_use.stream.works. Cells are
grouped module > feature > test, six dashboard modules in all. Each cell carries a
tier (P0/P1/P2), a source, and a fail_before_fix flag.
The rows live in per-prefix YAML files (llm_*.yaml, mgmt.yaml, mcp.yaml,
reliability.yaml, logging.yaml, guardrail.yaml, other.yaml) and validate against
the discriminated union in schema.py, so an LLM row cannot carry a guardrail field and
vice versa. logging and guardrail are two id-prefixes that roll up into the single
"Logging & Guardrails" dashboard module.
A test declares what it covers with a marker:
@pytest.mark.covers("llm.chat_completions.openai.tool_use.stream.works")
def test_openai_streaming_tool_calls(self) -> None:
...
The number
collector.py diffs the registry against those markers and reports coverage per module.
It is static: a collect-only pass reads the markers, so it runs no test and needs no live
proxy. Whether a covered cell currently passes or fails is a separate, live concern.
cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
The headline is P0 coverage. The collector also lists markers that point at ids not in the registry, so a typo or an unenumerated behavior surfaces instead of being silently dropped.
Status: this is a draft for review
The cells were enumerated from the codebase and the tiers are a first proposal. Known things to settle before treating the set as final:
- tiers are proposed, not signed off; 125 P0 is a lot to prove fail-before-fix, so P0 may want tightening
- a few cells need a support check or a prune (for example
llm.embeddings.anthropic.*andreliability.perf.throughput.under_slo) - auth is covered in two places (
other.auth.*and the mgmt authz assertions); the boundary needs a decision, and the auth cluster may deserve promotion to its own module - the P2 "niche" cells each stand in for a large tail of integrations/providers by design, so the denominator is deliberately P0-weighted rather than a full inventory