Two defects combined to make a Redis outage take the proxy down rather than
degrade it.
First, connection kwargs were dropped whenever Redis was configured by url.
_get_redis_url_kwargs built its allowlist from
inspect.getfullargspec(redis.Redis.from_url); from_url is declared
(cls, url, **kwargs), so the argspec carried no connection kwargs and the
function returned ['cls', 'url', 'url']. socket_timeout went with the rest,
and socket_connect_timeout falls back to it, so both ended up None and a
Redis host that drops packets rather than refusing them blocked callers
indefinitely. get_redis_connection_pool's url branch lost the same kwargs by
a different route, rebuilding its pool kwargs from scratch.
The allowlist now comes from the connection class redis-py actually forwards
those kwargs to, walking the MRO because redis-py splits them between
AbstractConnection and its subclasses. Deriving it from the client instead
would admit client-only settings such as single_connection_client and the
SSLConnection-only ssl_* family, which reach AbstractConnection and raise
TypeError on first connect.
Second, the circuit breaker could not trip even once calls failed fast.
_redis_circuit_breaker_guard inferred success from the method returning, but
async_get_cache, async_batch_get_cache, async_set_cache, async_set_cache_pipeline,
async_set_cache_sadd and async_get_ttl catch their own connection errors and
return a default so callers degrade. Each failed call therefore reset the
failure streak and the breaker never opened, so an unreachable Redis stayed in
the pool and every request kept paying a full socket timeout on it. Those
methods now mark the failure and the guard records success only when nothing
failed while the method ran. Lua script execution went through none of this,
which mattered most because the rate limiter issues all of its Redis traffic
that way, so the guard is now a small helper shared by both.
The per-call marker is a ContextVar rather than a counter on the breaker.
Breakers are shared by every concurrent caller, so a shared counter cannot
tell "my call failed" from "some other in-flight call failed", and a success
overlapping someone else's failure would be discarded until a Redis that was
still answering got evicted from the pool anyway.
Only connectivity failures feed the breaker. Command and data errors say
nothing about whether Redis is reachable, and counting them would let a caller
provoke evictions on demand (an INCR against a non-numeric value, say),
dropping rate limiting to per-process counters that spreading traffic across
replicas can outrun.
OpenAI publishes a long-context column on the Flex tier, at half the standard
long-context rate. We had no field for it, so a >272k flex request fell through
to the standard long-context price and billed 2x: Terra $4/$18 instead of
$2/$9, Luna $0.40/$1.80 instead of $0.20/$0.90, Sol $10/$45 instead of $5/$22.50.
Adding the values to the cost map alone does nothing, because get_model_info
builds ModelInfoBase from an explicit kwargs list and silently drops any key
not named there. Declare the four *_above_272k_tokens_flex fields and wire them
through, then add the values for sol, terra, luna, and the gpt-5.6 alias.
That same gap was already swallowing cache_creation_input_token_cost_flex,
_priority, and _above_272k_tokens, which were present in the cost map but never
reached the calculator; they are wired through here too.
Fast mode (ex-Priority) publishes no long-context column, so nothing is added
there rather than deriving a rate by analogy.
Guardrails could only see the MCP tool call request (pre_mcp_call /
during_mcp_call); the tool result went back to the client unscanned, so a tool
that returns sensitive data bypassed every configured guardrail.
Adds a `post_mcp_call` event hook that runs after the tool executes and routes
the result through the unified apply_guardrail seam, so a text guardrail (e.g.
presidio) can mask sensitive values in the tool output or reject the result
without any MCP-specific code of its own.
- MCPGuardrailTranslationHandler.process_output_response now extracts the tool
result's text content into GenericGuardrailAPIInputs["texts"], calls
apply_guardrail with input_type="response", and writes the returned text back
into the content list in place (the logging payload already references that
object, so a copy would leave the unmasked text in the spend log)
- ProxyLogging.post_mcp_call_hook dispatches guardrails that implement
apply_guardrail, gated on should_run_guardrail(post_mcp_call); guardrails
implementing async_post_mcp_tool_call_hook keep their existing dispatch and
are not run twice
- both MCP tool-call paths (mcp_server and the Responses API handler) now honor
the rewritten result, and the REST path no longer swallows a guardrail
rejection as a logging failure
- shared, duck-typed MCP content helpers live in mcp_server/utils.py next to
extract_mcp_tool_result_error_message
- documents that async_post_mcp_tool_call_hook's return value is discarded by
every call site, so that hook only takes effect by mutating in place
#35271 restored the hierarchy where a team-scoped key is governed by the
team and team-member budgets only; the owner's personal max_budget applies
to their personal keys. Three places in the e2e suite still encoded the
old direction and would fail against a proxy built from staging.
test_user_budget_enforced_across_all_their_keys asserted that the owner's
team-member key is refused once their personal budget is exhausted. It now
asserts only the personal keys are refused, and keeps the team key as the
control that must keep serving, which pins the restored direction instead
of leaving it unasserted. Renamed to match what it now covers.
test_team_member_key_user_budget_resets_after_window drove a team key to a
block off the owner's personal budget, so nothing can block it any more and
_drive_to_block could never succeed. Its premise is gone rather than moved,
so it is removed; the sibling personal-key test still covers
quota_management.budget.internal_user.resets_after_window.
The registry rationale for that row dropped its "and team-member keys"
clause for the same reason.
Reverts the shared IdCell change from the previous commit and scopes the
fix to the budgets table instead
IdCell truncates with `block max-w-[15ch]`, a character-count clamp with
no relationship to the column's width. On budgets the Budget ID column
renders 509px wide at a 1400px container while the ID stays pinned at
108px, so UUIDs ellipsize with ~400px of empty space beside them
Changing that clamp in IdCell itself is wrong today because nothing else
bounds the column. DataTable emits `width: <size>px` on each cell but
leaves the table in `table-auto`, where `width` is only a hint and
`max-width` on a cell is ignored outright (measured: a 120px request
yields a 938px column). Only `table-fixed` binds `size`, and DataTable
enables it solely under `enableColumnResizing`, which 4 of 40 tables use.
So an unbounded IdCell lets content drive the column: Request Logs would
render a 64-char key hash in full, taking its key_hash column from 124px
to 494px and pushing the table from 1918px to 2326px, introducing
horizontal scroll at 1920 where there was none
Scope it to the call site instead. `cn` is tailwind-merge backed, so a
`max-w-*` passed via className dissolves the base clamp while leaving
`truncate` in place; budget IDs render in full and still ellipsize at the
cell edge if one ever outgrows the column. No other table moves
This is a workaround. The real fix is to make column `size` authoritative
by separating a fixed-layout option from `enableColumnResizing`, then
dropping the per-cell clamps; 307 of 321 column defs already declare a
size, so the mechanical gap is small, but ~20 tables would gain
horizontal scroll at 1440 and that needs its own review
The v3 parallel-request limiter stashed its per-request bookkeeping (TPM
reservation, descriptors, parallel slot, rate-limit response snapshot,
released flag) in the request body's metadata channels. On routes where
metadata is a provider request parameter (Responses API and the other
LITELLM_METADATA_ROUTES) that leaked internal keys upstream and produced
HTTP 400s, and it required denylist stripping plus dual-channel writes to
contain.
The stash now lives on an asyncio ContextVar holding a single typed
RequestRateLimiterStash per request. The pre-call hook writes it, and the
success/failure callbacks, disconnect release, and post-call hooks read
and clear the same shared instance, which keeps the refund and slot
release idempotent across sibling callbacks. The request body is never
touched, so the stash-key stripping, the metadata mirror writes, and the
all_litellm_params denylist entries are removed
* fix(otel): label retrieval and agent metrics correctly and emit gen_ai.provider.name
The GenAI metric attribute builder mapped only chat, text completion, embedding,
responses and MCP tool calls to an operation name, so vector-store searches and
A2A agent sends fell through to the "chat" default. Their duration and cost then
landed in the same series a Grafana GenAI dashboard reads chat latency off, with
no way to tell them apart. Both now map to the operation names the convention
defines for them, retrieval and invoke_agent, and an unmapped call type says so
at debug instead of silently becoming chat.
The provider label used gen_ai.system, which the convention deprecated in favor
of gen_ai.provider.name; the dashboards built on that vocabulary find nothing
under the old key. Metrics now carry gen_ai.provider.name with the semconv
provider value (bedrock -> aws.bedrock) via the resolve_provider helper the span
path already uses, and keep dual-emitting gen_ai.system with its raw value so a
dashboard already querying it keeps matching. A request litellm cannot attribute
to a provider gets no provider label at all rather than a placeholder "Unknown"
that minted a permanent series nobody can act on.
Resolves LIT-4954
Resolves LIT-4959
* fix(otel): map the rest of the vector-store call types off the chat default
Mapping only the search left the store lifecycle (create, retrieve, list,
update, delete) and the file operations (create, list, retrieve, content,
update, delete) falling through to chat, so vector-store admin traffic kept
polluting the same series a dashboard reads chat latency off. A live run
confirmed it: all 20 metric datapoints from a create, retrieve, list, file-list
and delete came out labelled chat.
The convention names no operation for vector-store management, so these take
vendor values under the litellm. prefix, litellm.vector_store_management and
litellm.vector_store_file_management, one per REST resource. Its note on
gen_ai.operation.name directs instrumentation to use a system-specific name
when no predefined value applies, which is the same allowance resolve_provider
already relies on for unmapped providers. Excluding them from the GenAI metrics
altogether was the alternative; it deletes series an operator may be watching
today and is far harder to reverse than a rename, so it stays available as a
follow-up rather than being decided here. Mapping them onto the semconv memory
store family was rejected: litellm vector stores hold documents, not agent
memory records, and borrowing those names would put document admin calls into
whatever charts agent-memory operations, which is the bug this fixes.
/rag/query reaches the same recorder and is the same operation as a vector-store
search, so query and aquery map to retrieval too; leaving them would have left
the defect alive on a second retrieval surface. /rag/ingest is a write with no
semconv equivalent and no retrieval or agent confusion, so it is left for the
RAG owners to name.
Resolves LIT-4954
* fix(otel): give the streaming A2A path a call type so it labels as invoke_agent
The streaming logging object is built by hand and never runs through
update_environment_variables, the only place call_type reaches
model_call_details, so every streamed agent turn arrived at the recorder
with no call type and fell back to chat. Stamp it, and map the streaming
spelling alongside the non-streaming ones.
AWS rolled out the 2026-07-30 GPT-5.6 price cut the same day, but the
bedrock_mantle entries still carried values derived from the pre-cut OpenAI
base, so Terra billed 1.25x and Luna 5x over the published rate.
Re-derive both from the AWS Bedrock pricing page, which prices in-region
inference at parity with OpenAI's data residency tier (1.1x base). Sol was
not cut and is unchanged.
Also drop tests/test_litellm/test_gpt_5_6_model_metadata.py; its Azure and
openai pricing assertions are covered by test_llm_cost_calc_utils.py.
Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
IdCell truncated with `block max-w-[15ch]`, a character-count clamp that
ignores how much room the column actually has. On the budgets table the
Budget ID column renders 509px wide at a 1400px container while the ID
itself was pinned to 108px, so every UUID showed an ellipsis with roughly
400px of empty space beside it. The same held at 900px and 520px
containers; the clamp never moved because it was never a function of the
available width
Switch to `inline-block max-w-full truncate`, the standard CSS idiom for
shrink-to-fit text that ellipsizes at its container. IDs now render in
full whenever the column has room and clip at the cell edge when it does
not. `inline-block` keeps the pill variant sized to its content rather
than stretching the blue background across the column, which a plain
`block` would do once the character clamp is gone
Measured in Chrome across 1400/900/520px containers and both variants:
row height is unchanged, short IDs shrink from a padded 108px to 51px
(plain) and 67px (pill), and the 36-char UUID renders fully at 260px
* feat(otel): record the GenAI duration metric on failed requests
`_record_metrics` ran only from `async_log_success_event`, so
`gen_ai.client.operation.duration` counted only the requests that worked.
Latency read off it during an incident was the latency of the surviving
traffic, and with no error dimension anywhere there was no way to build a
failure-rate panel or a success/failure split per model.
A failed call now records the same duration histogram, tagged with the
semconv `error.type` (the mapped provider exception's class name, bounded by
construction; the message stays on the span). Success attributes are
untouched, so an existing query can still isolate the old series with
`error_type=""`. The other five instruments describe a completed generation
and are skipped rather than filled with a fabricated zero: litellm hands the
failure callback no `response_obj`, so there is no usage to split and no
completion-token count, and it zeroes `response_cost` on failure. A
proxy-gate rejection (auth / rate limit) records nothing, for the same
reason it gets no span; no upstream call happened.
`error.type` is stamped after the cardinality filter, like
`gen_ai.token.type`, so an `otel.attributes` include/exclude list cannot
strip the discriminator and silently merge failures into the success series.
Resolves LIT-4955
* fix(otel): bound the failure metric's attribute set
The failure datapoint reused the success path's full attribute set, which
carries client-supplied fields (`metadata.requester_metadata`,
`metadata.spend_logs_metadata`, the end-user id taken from the request's
`user` field) and per-request ones (the `hidden_params` blob holding the
provider's response headers). A failed request needs no provider spend, so
nothing rate-limits a caller who puts a unique value in a field they control
and mints one histogram series per request.
A failure now carries a bounded allowlist: the operation enum, provider,
request model, framework, the key/alias/team/org/user identifiers, and
`error.type`. Every entry is a fixed enum or an operator-provisioned
identifier, so the failure series count is bounded by the deployment's own
key, team and user count while the labels still answer which team on which
model is failing and how. The user email is left out as PII duplicating the
user id already on the series. The operator's `otel.attributes` filter layers
on top, so it narrows the allowlist further and never widens it.
* fix(otel): cap metric attributes so series count does not grow with traffic (#35166)
`GenAIMetricRecorder._common_attributes` dumped the whole `hidden_params` object
onto every metric datapoint as one label value. That object is per-request by
construction: `response_cost`, `litellm_overhead_time_ms`, `cache_key`,
`usage_object` and the provider's `additional_headers` rate-limit counters all
move on every call. A unique label value is a new time series, and all six GenAI
instruments share those attributes, so one request minted up to six series that
would never be written to again
That is the steady-state behavior of the feature rather than an abuse case, and
it is wrong twice over. Hosted backends bill on series count, so recommending
metrics be enabled would have meant a bill proportional to traffic. And a
histogram whose every datapoint sits in its own series cannot be aggregated, so
the dashboards would have looked populated while answering nothing
Both paths now cap their attributes at METRIC_ATTRIBUTE_CEILING, which replaces
the failure-only allowlist so the two paths cannot drift. The cap runs before the
operator's `otel.attributes` filter, so an operator can narrow it and never widen
it back to an unbounded label. Client-supplied and per-request metadata
(`requester_metadata`, `spend_logs_metadata`, `user_api_key_end_user_id`,
`requester_ip_address`) is metric-ineligible and stays on the span, which already
carries it and where cardinality is free. `hidden_params` survives as a label but
carries only `model_id` and `api_base`, which are bounded by the router's own
deployment list and are the part a per-deployment panel reads
Four tests fail against the previous behavior, the load-bearing one being that
two requests differing only in per-request fields must land in one series rather
than two
The MCP gateway resolved a caller's allowed servers and per-server tool
allowlists from the key, the team, the end user and the agent, but never from
the internal user row, so an admin had no way to bound what a person may call
across every key they hold. Anything the key allowed went through
The internal user now carries the same object_permission an admin already
attaches to a key or a team, and the resolver applies it as a ceiling: the
caller ends up with the intersection of what the key allows and what the user
allows, so adding a user entitlement can only narrow, never widen. A level
that names no server and no tool places no ceiling, which keeps every existing
deployment on its current behavior
/user/new and /user/update accept object_permission and reuse the same
create-or-update helper the team endpoints use, so the row is written once and
the three cached views of it (the user row, the object-permission link and the
permission itself) are invalidated on write. Clearing it with an empty object
now really unlinks the permission instead of being swallowed as an empty value
A row that cannot be read at all places no ceiling, but a row that names a
permission the database cannot return denies the call rather than falling
through to the wider set, so a partial outage cannot hand out access the admin
withheld
The users page grows the MCP servers, access groups, toolsets and per-server
tool pickers the key and team pages already have. A save keeps a tool
allowlist whenever an access group or toolset the admin retained could still
supply that server, since an allowlist is what narrows a grant and an absent
one reads as no restriction; it drops the allowlist once nothing indirect
survives to supply the server, so removing a grant really removes it
* fix(otel): cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span
The genai and legacy mappers each spelled out every declared tool as
per-index span attributes. A request declaring hundreds of tools produced
roughly 500 attributes against the OTel SDK's default 128-attribute span
limit, which evicts oldest-first, so the canonical gen_ai.* set written
first was discarded and the span exported with only a tail of tool
schemas. Cap the family at 8 tools, shared by both vocabularies, and
carry the declared total on litellm.request.tools.declared so the
truncation is visible rather than silent.
* fix(otel): apply the tool-definition cap to the OpenInference mapper
The OpenInference vocabulary emits its own unbounded llm.tools.{idx}.*
family, which Arize and Phoenix layer on top of the default two, so those
configurations still overran the span attribute limit and evicted the
core gen_ai.* attributes. Route it through the same shared cap and cover
the layered-mapper path with a test.
* fix(otel): share one span-wide tool-definition budget across vocabularies
Capping the tool-definition family per mapper left each active vocabulary
its own allowance, and several vocabularies write to the same span. With
every vendor vocabulary configured, the three that spell tools out per
index still summed past the SDK's 128-attribute span limit, so the core
gen_ai.* set written first was evicted exactly as before: measured at 128
attributes with 7 dropped and gen_ai.request.model gone.
Reserve a quarter of the span for tool detail and split that ceiling
across the distinct tool-emitting vocabularies at mapper-resolution time,
so the family is bounded span-wide no matter how many are configured. The
same worst case now exports 90 attributes with nothing dropped.
Auto-routed requests were indistinguishable from ordinary ones once logged:
the spend log recorded the requested model group and the resolved deployment,
but nothing about which tier was chosen or what chose it. That information
existed only inside verbose_router_logger f-strings, so answering "why did my
prompt land on the cheap model" required log access and a running proxy.
The complexity, quality, and adaptive pre-routing strategies now return a typed
StandardLoggingRoutingDecision on their PreRoutingHookResponse, and
Router.async_pre_routing_hook records it once for every attempt. Those three
previously side-channelled their own state through three different metadata
keys; the decision now travels on the hook contract itself, so the bucket is
resolved in one place, through get_or_create_metadata_bucket, which already
owns the question of which dict holds proxy-internal metadata and replaces a
non-dict value instead of skipping the write. Recording happens on every
attempt rather than only on a successful route: a fallback from an auto-router
group to a plain group re-enters the hook with the same request kwargs, and a
decision left behind there would attribute the first router's tier to the
deployment that actually served the retry. The log details drawer renders the
result as a Routing card between Request Details and Metrics; the card is
absent on rows that carry no decision, so ordinary and pre-upgrade rows are
unchanged.
Three defects surfaced while making the recorded cause truthful, each of which
would have persisted a wrong answer. The complexity router hardcoded
cause=complexity_scorer even when the LLM classifier decided, and its silent
fallback to the heuristic on classifier failure meant a row could claim an LLM
verdict the LLM never gave; the cause now reports the path that actually ran.
The keyword that triggered a tier rule was discarded before logging, as was
the escalation keyword. The 2-reasoning-marker override returned REASONING with
a score far below the REASONING boundary and no marker saying so, which reads
as a scoring bug to anyone comparing the two; it now emits a reasoning-override
signal, and the card labels those rows as an override instead of claiming the
score met a boundary. The LLM path no longer reports a synthetic score of 1.0,
and heuristic decisions carry a snapshot of the tier boundaries that mapped the
score, so a historical row stays interpretable after the boundaries change.
Signals name a matched term only when the caller's own message contains it.
Scoring still reads the system prompt, but a term matched solely there is
reported as a count, since signals reach a spend row the caller can read and
naming one would disclose a term from a prompt it cannot see.
routing_decision is stripped from caller-supplied metadata at ingress, so a
client cannot forge its own provenance.
Bedrock routes Claude Sonnet 5 through the same Anthropic-compatible
validator as Opus 4.7/4.8 and Sonnet 4, which rejects toolSpec.strict
with 'tools.0.custom.strict: Extra inputs are not permitted'. Set
bedrock_converse_supports_strict_tools: false on all six Sonnet 5
entries so the existing gate strips the field, matching the fix shape
of #31582
Co-authored-by: Yaroslav Budyanskiy <y_budyanskiy@wargaming.net>
The existing dashboards in this folder chart the litellm_* Prometheus metrics.
Nothing charted the gen_ai.* metrics the OpenTelemetry v2 integration emits, and
Grafana's own prebuilt GenAI dashboards cannot: twenty of their twenty-two panels
filter on telemetry_sdk_name="openlit", a label LiteLLM does not carry and has no
setting to add.
Ten panels over the six gen_ai instruments: spend, tokens, request count and p95
duration as stats, then request rate, spend per hour, tokens per minute split by
input and output, and p95 duration, time to first token, and provider generation
time by model. Template variables for data source, service, and model.
Verified against a live Grafana Cloud stack with real traffic across three
models. The readme documents the attribute filter the panels depend on, since the
default attribute set gives nearly every request its own series and makes every
rate-based panel read zero.
Replace `Model(**untyped_dict)` construction with `Model.model_validate(...)` at
the hot Any seams, and give the repository layer a real record type instead of
`Any`.
reportAny 22710 -> 21448, reportExplicitAny 7283 -> 7269, with every other rule
at or below its baseline repo-wide.
The Auto-Routers tab was proxy-admin only, while Add Model on the same page already
admits team admins. The asymmetry was not a policy decision; the auto-router create form
simply never mounted a team selector, so a team admin's submit was unscoped and POST
/model/new rejects an unscoped create from any non-proxy-admin. Mounting the shared
TeamDropdown closes it, and the tab now takes the same audience as its sibling.
Fixing that surfaced a second, larger problem. The dashboard decided who may edit or
delete a deployment with `(userRole === "Admin" || created_by === userID) && db_model`,
but `created_by` is written at creation and never read by any backend auth check. The API
authorizes on team-admin membership of model_info.team_id, so the dashboard was wrong in
both directions: it hid controls from team admins the API accepts, and offered them to
former team admins the API rejects. Verified against a live proxy; a model created by the
proxy admin was PATCHed and DELETEd 200 by a team admin who did not create it, while the
same key got 403 on another team's row and on an unscoped row.
Both questions now have one owner in utils/modelPermissions.ts, deliberately shaped as a
mirror of ModelManagementAuthChecks. Creation returns a tagged union rather than a pair of
booleans, so "may not create" and "may create unscoped" cannot be confused, and the five
places that had each invented their own spelling (the models page, the auto-routers tab
and panel, the auto-router form, and both branches of AddModelForm) call it instead.
Row affordances are now per row rather than per tab, because opening the tab to team
admins puts routers they cannot act on in the same list.
Note for reviewers: collapsing AddModelForm onto the shared owner changes behaviour for
org_admin and Admin Viewer who also admin a team. They previously got the optional team
selector, because all_admin_roles counts them as admins, and could submit an unscoped
create that the API always 403s; they now get the required selector.
Also corrects stale copy left by the auto-router move. The exclude_auto_routers API
description named a dashboard page, which went stale inside a single PR; it now describes
the concept so it cannot drift with the UI again.
The eslint-suppressions prune includes one entry for caching/_components/cache_dashboard.tsx,
which this branch does not touch. Its baseline was already stale; the gate measures the whole
tree, so it could not be left behind.
Completed-batch cost tracking parsed the whole output file into a list of
dicts, pretty-printed it into debug strings even with debug logging off, and
walked the list three times (cost, usage, models), so a large batch output
could pin a worker's memory. The output is now folded line by line into small
per-line stats records via _aggregate_batch_cost_usage_models, the eager
json.dumps debug calls are gone, and the raw-vertex path computes cost and
usage in one call instead of two. _get_batch_output_file_content_as_dictionary
becomes _fetch_batch_output_file_content (returns bytes); the superseded
three-pass helpers are deleted and their tests migrated
