mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
75 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yuneng Jiang
|
7d6ee2a9ca
|
feat(proxy): let AI API keys read /model/info
Keys created with key_type=llm_api get allowed_routes=["llm_api_routes"], which covered /v1/models but not /v1/model/info, so a client could list model names but not read pricing, mode, or max_tokens without a second key. Adds both /model/info and /v1/model/info (same handler) to llm_api_routes only. Membership there is not the same as RouteChecks.is_llm_api_route(), which is what gates DISABLE_LLM_API_ENDPOINTS, global/virtual-key budget enforcement, enforce_user_param and JWT team attachment; /guardrails/apply_guardrail already sits in the group the same way. /v2/model/info stays out: it is the paginated Admin UI listing, not model metadata a caller needs at request time. public_routes moves from set([...]) to a frozenset literal to keep the LIT002 and ruff-strict ceilings from rising; both budgets ratchet down by one. |
||
|
mateo-berri
|
add2a1ce3f |
chore(typing): clear 2.4k basedpyright errors across 15 Any hotspot files
Replace Any-typed seams with real types in the files carrying the highest reportAny/reportExplicitAny density. The dominant source was the repository layer: BaseRepository.table is declared Any, so every repository read poisoned its rows and every downstream call. Typed pass-through accessors under a _PrismaTableActions Protocol pay that crossing once per table, and TypedDicts and Protocols replace the remaining Any-typed request, row, and tool payloads across the team, key, SCIM, spend, MCP, guardrail, video, and websearch surfaces No casts, no type: ignore, no noqa, no new Any annotations, no behavior changes. Whole-tree basedpyright: reportAny 20,840 -> 19,397, reportExplicitAny 7,253 -> 6,518, all rules 151,424 -> 149,066, with no rule increased in any file. Budgets ratcheted: basedpyright -2,358, ruff-strict -300, type-discipline -68 |
||
|
mateo-berri
|
c0cab45350
|
chore(typing): replace Any kwargs unpacking with validated model parsing
Swap `Model(**payload)` for `Model.model_validate(payload)` at the seams where the payload comes back untyped, so basedpyright stops widening every target field to Any. None of the models involved override `__init__`, so validation goes through the same core validator either way. Also route UserRepository through its own typed helpers (find_many, update, find_by_id) instead of the raw Prisma table, drop the redundant `_to_model` override signature, and call generate_key_helper_fn with explicit arguments in the SSO callback rather than splatting an untyped dict. Whole-tree basedpyright: reportAny 21481 -> 20834, reportExplicitAny 7258 -> 7252, with every other rule unchanged or lower. |
||
|
mateo-berri
|
a97233067d |
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_batch_output_single_pass
# Conflicts: # basedpyright-code-budget.json # ruff-strict-budget.json # type-discipline-budget.json |
||
|
mateo-berri
|
76cf3bf6ac
|
chore(typing): clear basedpyright Any errors in proxy auth, repositories, and openai transforms
Replace `Model(**untyped_dict)` construction with `Model.model_validate(...)` at the hot Any seams, and give the repository layer a real record type instead of `Any`. reportAny 22710 -> 21448, reportExplicitAny 7283 -> 7269, with every other rule at or below its baseline repo-wide. |
||
|
mateo-berri
|
0b09588685 |
refactor(batches): aggregate batch output cost, usage, and models in a single pass
Completed-batch cost tracking parsed the whole output file into a list of dicts, pretty-printed it into debug strings even with debug logging off, and walked the list three times (cost, usage, models), so a large batch output could pin a worker's memory. The output is now folded line by line into small per-line stats records via _aggregate_batch_cost_usage_models, the eager json.dumps debug calls are gone, and the raw-vertex path computes cost and usage in one call instead of two. _get_batch_output_file_content_as_dictionary becomes _fetch_batch_output_file_content (returns bytes); the superseded three-pass helpers are deleted and their tests migrated |
||
|
mateo-berri
|
a895249923 | refactor(bedrock): remove the dead BedrockLLM invoke code path | ||
|
mateo-berri
|
44e091aedb
|
chore(typing): clear basedpyright Any errors in proxy management endpoints
Convert pydantic table-model construction from Cls(**row.model_dump()) kwargs-unpacking to Cls.model_validate(...) across the management endpoint hotspot files (team, key, internal user, scim, model management, spend tracking, auth checks, proxy_server). Unpacking an untyped dict reports one Any-typed argument per matched model field, so each converted site clears 10-35 diagnostics while running the exact same pydantic validation. Conversions were limited to models verified to use pydantic's default __init__; UserAPIKeyAuth and LiteLLM_VerificationTokenView keep their custom kwargs-rewriting __init__ and are untouched. Two locally-verified helper params move from Any to object. Whole-tree basedpyright, measured against the branch point in the same environment: reportAny 24,431 -> 22,741 (-1,690), reportArgumentType 2,189 -> 2,136 (-53), reportUnknownArgumentType 34,370 -> 34,067 (-303), reportExplicitAny 7,285 -> 7,283 (-2); total 154,882 -> 152,834 (-2,048) with no rule increasing anywhere and no per-file increases. No casts, no suppressions, no behavior changes. Budgets ratcheted: basedpyright -2,048 across 4 rules, ruff ANN401 -2. |
||
|
mateo-berri
|
fbfb63c948 |
chore(typing): clear 2.7k basedpyright Any errors across 15 hotspot files
Replace Any-typed seams with real types in the files carrying the highest reportAny/reportExplicitAny density: typed Prisma read helpers in the MCP db layer and verification token repository, TypedDicts for OAuth credential payloads and aggregated spend rows, a DailySpendRecord protocol for the daily activity endpoints, and concrete request/response types in the volcengine, openai evals, azure batches, azure_ai count_tokens, and ocr transformation modules. Modernize touched annotations to PEP 604/585 forms. No casts, no type: ignore, no noqa, no new Any annotations, no behavior changes. Whole-tree basedpyright: reportAny 27,005 -> 24,427, reportExplicitAny 7,439 -> 7,280, no rule increased anywhere. Budgets ratcheted: basedpyright -2,869, ruff-strict -1,505, type-discipline -167. |
||
|
Noah Nistler
|
8177230a29
|
feat(guardrails): add run_in_parallel opt-in for concurrent pre_call and post_call guardrails (#33770)
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails Pre-call guardrails run sequentially because each may mutate the request payload and later guardrails depend on earlier mutations. Deployments with several slow block-only pre_call guardrails (external moderation, Bedrock, LLM-judge) therefore pay the sum of their latencies. during_call guardrails run concurrently but alongside the LLM call, so a violating payload has already been sent, which is unacceptable when the request must never reach the model. This adds a per-guardrail run_in_parallel flag (default off). Guardrails that opt in are pulled out of the sequential loop and run concurrently via asyncio.gather after every sequential (payload-mutating) guardrail has run, so they observe the mutated payload and still form a hard barrier before the LLM call; the first to raise blocks the request. Their returned data is discarded since they are declared block-only. The flag is wired from LitellmParams onto the guardrail instance at the same generic choke point in initialize_guardrail that already sets skip_system_message_in_guardrail, so no per-provider initializer needs to change. * feat(guardrails): extend run_in_parallel opt-in to post_call guardrails post_call_success_hook ran guardrails sequentially for the same reason pre_call did: response-modifying guardrails thread the response forward. But block-only output scanners (which read the response and reject on violation without changing it) serialize for no benefit and add latency. This reuses the existing run_in_parallel flag for the post_call hook. Opted-in post_call guardrails are pulled out of the sequential loop and run concurrently via asyncio.gather after the sequential (response-modifying) guardrails and before the non-guardrail CustomLogger callbacks, so they inspect the final response and still block it from reaching the client if any raises. Their returned response is discarded since they are block-only. The apply_guardrail path sets data["guardrail_to_apply"] immediately before awaiting, and unified_guardrail pops it before its first suspension point, so concurrent guardrails never race on that key under asyncio's cooperative scheduling. * fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes Addresses review feedback on the run_in_parallel opt-in. asyncio.gather propagated the first exception without cancelling or awaiting the siblings, so a block at t=0 left the other guardrails running as unobserved background tasks (wasted external calls plus event-loop warnings), and a fast SensitiveDataRouteException/ModifyResponseException could return a reroute or passthrough before a slower block finished, letting crafted input bypass the block. Both the pre_call and post_call parallel batches now gather with return_exceptions=True so every guardrail runs to completion, then raise any blocking exception ahead of a flow-changing one. The registry choke point wrote bool(None)==False onto every instance when the config omitted run_in_parallel, silently disabling a constructor-set default; it now only writes when the config provides an explicit value. * fix(guardrails): record lifecycle logs for every concurrently-run guardrail The log_guardrail_information decorator skipped its auto-record when it saw that the count of standard_logging_guardrail_information entries in the shared request_data had grown during the wrapped call, taking that as proof the wrapped function had recorded its own richer entry. That heuristic breaks the moment guardrails run concurrently (parallel pre_call/post_call, during_call): a sibling guardrail's append inflates the shared count, so a guardrail that did not self-record wrongly concludes it already did and drops its own entry. The result is that enabling run_in_parallel silently loses per-guardrail lifecycle logs, so the Admin UI Request Lifecycle timeline and downstream loggers (Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent guardrails. Replace the shared-count heuristic with a ContextVar flag set when a guardrail records its own entry. asyncio copies the context into each gathered task, so the flag is isolated per concurrent guardrail while still catching the self-record-then-skip-auto-record case within a single invocation. * test(guardrails): declare run_in_parallel on post_call guardrail mocks The post_call partition reads run_in_parallel on every CustomGuardrail callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is set in __init__, not on the class) so the attribute access raised, and even a class-level default would return a truthy child mock that wrongly routes the double into the parallel batch. Declare the flag False on the shared mock factories so these pre-existing hook tests exercise the sequential path they assert on. * fix(guardrails): harden run_in_parallel reads and address review feedback Read run_in_parallel via getattr(..., False) in the pre_call and post_call partitions so a third-party CustomGuardrail subclass that overrides __init__ without chaining super().__init__() no longer raises AttributeError on a path that previously worked. Drop the redundant in-function GuardrailEventHooks import in _run_parallel_post_call_guardrails (already imported module-level). Remove the flaky wall-clock upper-bound assertions from the two concurrency tests; the all-start-before-any-end overlap assertion is the timing-independent signal that actually proves concurrency. |
||
|
ryan-crabbe-berri
|
76c9eca25d
|
refactor(auth): derive temp budget increase without mutating the token (#34121)
* refactor(auth): derive temp budget bump without mutation, tz-aware auth datetimes _update_key_budget_with_temp_budget_increase mutated max_budget in place, so correctness depended on every resolution path handing it a fresh copy of the cached token; one future re-cache of a live token would compound the bump per request. Return a model_copy instead so no caller can leak an increased budget into shared state. Also fixes the three remaining DTZ005 naive datetime.now() calls in user_api_key_auth.py (auth span start, builder start_time, service-log end_time; all consumers convert to epoch or subtract same-pair datetimes) and ratchets the DTZ005 strict budget 244 -> 241. * test: pin non-mutation of the temp budget helper input Adversarial mutation-testing showed reverting the helper to in-place mutation still passed every test: the cache's copy-on-read layer masks the mutation in the integration test and the direct unit test only inspected the return value. Assert the input object is left untouched and the result is a distinct object so the purity guarantee itself is load-bearing. |
||
|
Tin Chi Lo
|
40f02b2eb5 | refactor(mcp): consolidate exception-tree walkers into one shared faults traversal | ||
|
Krrish Dholakia
|
26ab730bfa
|
feat(router): soft-floor adaptive mode for complexity router (#32947)
* feat(router): soft-floor adaptive mode for complexity router Let complexity_router_config.adaptive=true Thompson-sample across the union of tier pools with a tier-distance penalty, and wire the existing adaptive post-call bandit so mis-tiered requests can still recover. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): reattach adaptive hooks for hybrid complexity Finalize was wiping every AdaptiveRouterPostCallHook and only re-registering standalone auto_router/adaptive_router deployments, so complexity adaptive=true never received bandit updates. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): drop unnecessary hybrid docstrings Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attribute adaptive feedback Credit user reactions to the model that produced the previous response while keeping current-response signals on the serving model Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): tune hybrid cold defaults Use the cost-weighted policy that beat equal-pool complexity in the full bakeoff, and make the committed harness compare identical tier pools Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): preserve hybrid cold quality floor Sample only unobserved models in the classified tier until feedback exists, then apply adaptive scoring without mis-penalizing models shared across tiers Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): bound feedback context cache Cap retained session feedback so unique session IDs cannot exhaust router memory Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): preserve exhaustion signals Include tool-result exhaustion in adaptive feedback and clear strict lint regressions blocking CI Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): remove stale owner cache Remove obsolete attribution state, tighten the embedded router type, and keep the test diff focused on adaptive behavior Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): centralize hook cleanup Use the callback manager to discover and remove adaptive hooks across every registered callback list Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
5b93ba0ada
|
feat(router): add separate ITPM/OTPM deployment rate limits (#31952)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(router): add separate ITPM/OTPM deployment rate limits Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): keep ITPM/OTPM diff minimal in router.py Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): make ITPM/OTPM limits separate and atomic Address Greptile review on separate ITPM/OTPM deployment rate limits. - OTPM is now reserved atomically pre-call with rollback, matching the ITPM path, so concurrent requests can no longer overshoot the configured output limit before reconciliation - ITPM counts input tokens only; it no longer accumulates completion tokens, so the input-token limit and x-ratelimit-limit-input-tokens header describe input usage as their names imply - _read_reservation_from_kwargs only falls back to litellm_params.metadata when the top-level metadata channel is absent, so production requests carrying a litellm_params.metadata dict still reconcile and refund their reservation Adds regression tests for OTPM atomicity under concurrency, input-only ITPM enforcement, and reservation lookup when litellm_params.metadata is present. * fix(router): subtract input tokens only from remaining-input-tokens header The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total tokens (input + output) instead of input tokens only, so clients saw remaining input quota understated by the completion token count on every response. Now consistent with the input-only ITPM counter. * fix(router): make itpm/otpm vs tpm/rpm precedence explicit When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path takes over and the tpm/rpm limits are not enforced. Log a warning the first time such a conflicting deployment is seen so the supersession is not silent, and document the mutual exclusivity. Post-call reconciliation now only trues up a counter that was actually reserved against, so the itpm/otpm keys are no longer incremented for deployments that never configured that limit. * fix(router): track actual io-token usage on the reservation-minute key Post-call reconciliation now keys off the exact cache key stashed at pre-call time rather than one recomputed from the response-time minute. This fixes two issues: a request whose pre-call estimate was 0 now still writes its actual billable input to the ITPM counter (previously it was skipped, leaving the limit unenforceable for that request), and a call that finishes in a later minute reconciles against the minute it reserved against instead of pushing a negative delta into the next minute. Counters are only touched when their limit is configured. * fix(router): run io-token reconciliation before the model_id guard async_log_success_event gated IO reconciliation behind the model_id guard that only the TPM tracking path needs. Since reconciliation works entirely from the cache keys stashed in kwargs, a success event whose standard_logging_object lacks model_id would skip reconciliation and leave the reservation on the counter until the TTL expired, wasting quota. Route the IO path first. * fix(router): don't replay in-flight delta for itpm/otpm headers For ITPM/OTPM model groups the counter is incremented at reservation time (pre-call), so the remaining values returned by get_remaining_model_group_usage already account for the current request. Replaying the in-flight delta on top double-counted it and understated x-ratelimit-remaining-input/output-tokens by up to max_tokens on every response. Skip the delta for io-token groups; the legacy TPM/RPM replay path is unchanged. * fix(router): clear io-token reservation after reconcile/refund async_io_token_refund_failure and async_io_token_reconcile_success now clear the stashed reservation keys from the request metadata once done. Otherwise, on a model group mixing IO-limited and non-IO deployments, a failed IO call that retries on a non-IO fallback left the stale sentinel in the shared request metadata; the fallback's success handler would divert into IO reconciliation against the already-refunded key, driving the ITPM counter negative and skipping the non-IO deployment's TPM tracking. * fix(router): tidy reservation channel lookup and header guard Consolidate the reservation channel lookup into a single ordered helper shared by read and clear, so top-level metadata always wins over litellm_params metadata without the tangled per-iteration fallback. Also stop gating the router rate-limit header block on the presence of x-ratelimit-remaining-input/output-tokens. That block only emits those headers for ITPM/OTPM groups; for a non-IO group backed by a provider that natively returns input/output token headers, the extra conditions suppressed the router's own remaining-tokens/requests headers. * fix(router): strip client-supplied io-token reservation keys The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key, and the otpm equivalents) are server-only, but metadata is caller-controlled on proxy requests. An authenticated caller could forge these fields with an arbitrary cache key so the post-call reconcile/refund path would decrement any deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs, which runs before the router stashes its own reservation, so only a genuine server-side reservation is ever read post-call. * fix(router): track TPM routing load for io-limited deployments deployment_callback_on_success early-returned for any deployment with itpm/otpm set, so its total-token usage never landed in the router's TPM routing counter. TPM-aware routing strategies then saw 0 load for IO deployments and over-routed to them in mixed model groups. Only skip tracking when neither tpm/rpm nor itpm/otpm are configured; itpm/otpm enforcement still runs separately in ModelRateLimitingCheck, so the routing counter and the enforcement counters stay independent. * fix(router): expose standard tpm/rpm headers for io-limited groups get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests; clients and prometheus gauges reading those saw no data. Build both header sets instead of returning early. Also simplify the in-flight header replay: only the tpm/rpm counters are incremented post-response, so the delta now adjusts just those. The itpm/otpm counters are incremented at reservation time (pre-call), so the input/output token headers already reflect the request and are left untouched - which removes the need for the separate io-group special case. * fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance Two follow-ups from review. The pre-call OTPM reservation only rolled back the ITPM reservation on a RateLimitError, so a transient cache error while reserving OTPM left the ITPM counter inflated until the TTL expired; catch any exception, release the ITPM reservation, then re-raise. Replace the module-level lru_cache warn-once (caching a logging side effect, which never re-warns in a long-lived process) with an instance-scoped set of already-warned deployment ids on ModelRateLimitingCheck. * fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup Clear the reservation in a finally block so a mid-reconciliation cache error still removes the stash and a duplicate success event can't re-process it. Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a deployment with no id no longer collapses every id-less deployment onto the str(None) key (which would suppress all but the first warning). * fix(router): skip io reservation when deployment can't be keyed _get_cache_keys returned a shared 'global_router:None:None:...' key when a deployment was missing model_info.id or litellm_params.model, so misconfigured deployments could share one rate-limit bucket. Return None in that case and skip io reservation for the request. * fix(router): honor explicit max_tokens=0 in io reservation _resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit max_tokens=0 fell through to the model default. Only fall back to max_completion_tokens when max_tokens is absent. * fix(ci): satisfy lint budget, router coverage, and dashboard schema sync - Modernize the new itpm/otpm module's type hints to PEP 585 lowercase generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006 violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match. - Replace three try/except Exception blocks that must stay broad by design (token_counter and litellm.get_model_info raise untyped exceptions, and an io-token refund failure must never break the logging pipeline) with contextlib.suppress(Exception), matching the codebase's existing resolution for this exact BLE001 pattern. - Add direct unit tests for get_model_group_io_token_usage (multi-deployment aggregation and the empty-model-list case) in test_router_helper_utils.py, satisfying the router function-coverage check. - Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types. * fix: enforce io token rate limits consistently * fix: honor zero max tokens in otpm reservation * fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10 floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span alias. The previously committed ruff-strict-budget.json ratcheted UP006 down from a stale base; litellm_internal_staging has since tightened that same ceiling further on its own. Reset the file to the current base's committed values and re-ratchet from there so the budget only ever moves down relative to the actual merge-base, never against a stale snapshot. * fix(router): attach ITPM/OTPM headers on dict responses and harden reservation Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit headers through /v1/messages dict responses via _hidden_params. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so set_response_headers can attach rate-limit headers to streaming Anthropic messages responses that lack a _hidden_params slot. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format add_retry_fallback_headers.py Fix CI ruff format check failure on get_hidden_params_dict call site. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): extract set_response_headers helpers to fix C901 budget Move header-attachment logic into add_retry_fallback_headers helpers so set_response_headers stays under the strict complexity ceiling. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: keep IO token reservation when response usage is missing Missing usage was reconciled as zero and fully refunded the pre-call reservation, allowing limit bypass on repeated successful calls. Only adjust counters when usage is resolved from the response or standard logging fields; otherwise keep the reservation until TTL expires. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: enforce RPM/TPM alongside IO-token limits on mixed deployments Deployments with both itpm/otpm and tpm/rpm previously returned after the IO reservation and skipped RPM/TPM checks. Run both paths and refund the IO reservation only when RPM/TPM rejects after a successful reservation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: track TPM usage on success for mixed IO+TPM deployments The early return after IO-token reconciliation in log_success_event and async_log_success_event skipped the TPM counter increment, so the tpm_key the pre-call check reads was never written and tpm_limit was never actually enforced on deployments that also configure itpm/otpm. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: treat total-only usage as unresolved in IO-token reconcile usage/standard_logging_object entries carrying only total_tokens (no prompt/completion or input/output breakdown) were treated as resolved usage, resolving to (0, 0) and refunding the full reservation. Both _usage_is_present and the standard_logging_object fallback now require an actual input/output breakdown before reconciling, keeping the reservation otherwise. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: reserve minimal token when input/output estimation fails _reservation_value(0, limit) reserved the entire limit whenever token estimation failed (empty/unsupported input, tokenizer error), letting one such request claim the whole bucket and 429 every concurrent request to the deployment until it completed. Reserve 1 token instead so estimation failures no longer serialize traffic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: refund IO reservation synchronously before retry deployment pick On retry, set_io_token_rate_limit_request_kwargs clears reservation sentinels from the shared kwargs dict before a background failure handler can refund them, stranding the counter until TTL. Refund and clear any stale reservation in _update_kwargs_with_deployment before stripping sentinels for the next attempt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling Pass the deployment litellm_params.model to token_counter so it uses the model's native tokenizer instead of the generic fallback, narrowing the reservation over/under-estimate window between pre-call and post-call reconcile. Add a ponytail: comment to refund_stale_reservation_before_retry explaining the known ceiling: the synchronous DualCache.increment_cache issues a blocking Redis INCR when a Redis backend is configured. This only fires on streaming mid-stream retries (non-streaming failures await their failure handler before the retry picks a new deployment, leaving no sentinels to refund). Upgrade path: make _update_kwargs_with_deployment async. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Mateo Wang
|
5f864c83ce
|
chore(lint): zero out crash-class pyright rules and ban new type: ignore comments (#32152)
* fix: zero out crash-class basedpyright rules across litellm/ * feat(lint): add LIT009 banning inert type: ignore comments * docs: require bracketed rule and reason on every suppression * chore(lint): ratchet budgets down and zero crash-class pyright limits * fix: narrow auto router routelayer through a local before calling * test: add regression tests for crash-class fixes * fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex |
||
|
ryan-crabbe-berri
|
27069bd74f
|
feat(ui): shadcn migration foundation: Tailwind v4, shadcn init, antd cascade fix (#31995)
* feat(ui): shadcn migration foundation: Tailwind v4, shadcn init, antd cascade fix
Upgrade the dashboard from Tailwind v3 to v4 with CSS-first config: the
official upgrade codemod renamed utilities across 151 files, and
tailwind.config.js (plus the dead tailwind.config.ts) is replaced by
@theme tokens, @source globs, and @plugin directives in globals.css. The
Tremor safelist becomes @source inline patterns and the legacy tremor
theme tokens carry over verbatim. ui_colors.json was build-time only and
fed the dying Tremor palette, so its brand values are inlined and the
file removed; runtime theming replaces that path next.
shadcn is initialized with a hand-authored components.json (rsc,
cssVariables, baseColor gray) pointing utils at the existing
lib/cva.config.ts, which now exports cn (cva beta cx + twMerge) instead
of adding class-variance-authority as a second variant library. The two
ad-hoc cn helpers fold into it. Button lands as the canary primitive,
adapted to cva beta and React 18 forwardRef, with tests covering the
variant, twMerge, asChild, and ref seams. --radius is 0.5rem so the
shadcn radius scale reproduces Tailwind defaults and legacy rounded-*
classes render unchanged.
antd v5 emits unlayered CSS-in-JS that would beat every layered v4
utility, so AntdGlobalProvider now wraps the app in StyleProvider layer
and ConfigProvider cssVar, and globals.css declares
@layer theme, base, antd, components, utilities. antd wins over
preflight but yields to utilities, which is what lets migrated shadcn
pages coexist with legacy antd pages. Preflight stays global with the
three v3 behaviors pinned (default border color, button cursor,
placeholder color).
* fix(ui): restore tremor opacity tints removed by tailwind v4
Tailwind v4 removed the *-opacity-* utilities, but the precompiled
@tremor/react dist still composes them with shade-500 palette classes
(bg-opacity-10 over bg-<color>-500 etc.), so Badge, BadgeDelta, Callout,
light Icon and Button, BarList, and ProgressBar lost their tints and
rendered solid 500-shade fills. Adversarial review caught it; the
original smoke pages only exercised antd Tags.
tremor-v3-compat.css restores exactly the pairs tremor emits: for each
of the 22 safelisted colors, bg-opacity-{10,20,40}, hover/group-hover
bg-opacity-{20,30}, and ring-opacity-{20,40} against the -500 shade,
via color-mix into the utilities layer. Tremor's colorPalette maps both
background and iconRing to 500, so the -500 pairing covers every
composition in the dist; dark: variants are inert until dark mode ships.
The shim dies with @tremor/react at the end of the migration.
The upgrade codemod also missed two hand-rolled modal scrims using
bg-black bg-opacity-{30,50} (solid black under v4); now bg-black/30 and
bg-black/50. Removed the docker/build_admin_ui.sh copy of
enterprise_colors.json into the deleted ui_colors.json; that build-time
rebrand path is retired and its runtime replacement lands with the
theming phase.
* fix(ui): pair ring-opacity-40 with shade 300 in tremor compat shim
Tremor's colorPalette maps ring to shade 300, and the only consumer of
ring-opacity-40 (Icon variant outlined) composes it with that shade,
so the shade-500 rows were dead and outlined icon rings would render
at full opacity. Latent today (no dashboard usage of the outlined
variant); caught by adversarial review. ring-opacity-20 stays at 500
(iconRing), matching Badge and BadgeDelta.
|
||
|
Shivam Rawat
|
1543725916
|
fix(bedrock): honor ttl for tool_config cache injection points (#31929)
* fix(bedrock): honor ttl for tool_config cache injection points Pass cache_control_injection_points control.ttl through to Bedrock toolConfig cachePoint blocks, matching message/system cache behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a manual update for every new Claude release (it already silently missed Sonnet 5 and Fable 5). Replace it with a lookup against cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json, which AWS docs confirm tracks the same 1h-TTL-capable model set. Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried that pricing field (their own regional variants didn't have it), which would have made the JSON-driven check wrongly grant them 1h TTL support. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id (...-20250514-v1:0) that never shipped. This passed under the old regex-based is_claude_4_5_on_bedrock, which matched on substring alone, but fails now that it looks up cache_creation_input_token_cost_above_1hr in litellm.model_cost, since the fake id has no pricing entry. Also force the bundled local cost map in both tests so ttl eligibility reads this branch's pricing data instead of the network-fetched main copy, which lacks the fix until merge. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): restore cache and tool config compatibility * fix(bedrock): preserve Sonnet 5 parallel tool config * fix(bedrock): decouple parallel tool support from cache ttl * refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and bedrock_converse_supports_strict_tool_schemas (dead code) with a supports_parallel_tool_use_config key in model_prices_and_context_window.json, matching how is_claude_4_5_on_bedrock already reads cache_creation_input_token_cost_above_1hr from the pricing JSON. New models pick up parallel tool use support automatically when their pricing entry ships with the key set, with no code change required Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in model_prices_and_context_window.json, so bedrock_converse_supports_parallel_tool_use_config returned False and the test died with KeyError on additionalModelRequestFields. Use jp.anthropic.claude-opus-4-7, a real entry that carries supports_parallel_tool_use_config without 1h-TTL cache pricing, which is exactly the decoupling this test exists to cover * test(utils): allow supports_parallel_tool_use_config in pricing schema The misc unit test job validates model_prices_and_context_window.json against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects unknown keys. Add the supports_parallel_tool_use_config key this PR introduced so test_aaamodel_prices_and_context_window_json_is_valid passes again * fix(bedrock): preserve ttl for regional claude models * fix(bedrock): fall back to base model entry when regional pricing lacks capability fields Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit cache_creation_input_token_cost_above_1hr shadowed the base entry that has it, so is_claude_4_5_on_bedrock returned False and requested cache ttl values were dropped for those deployments. Both capability lookups now consult the full model id and the region-stripped base entry, matching the coverage of the old name-pattern list. Also restores ToolBlock keyword construction for the tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every supported Python version --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c4f28ce287
|
fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (#31924)
* fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (LIT-2239) * fix(bedrock): reopen audio content at client sample rate after trigger block * test(bedrock): cover realtime handler disconnect flush and stream-end guard * fix(bedrock): always close realtime input stream even if close flush fails * fix(lint): use contextlib.suppress in bedrock realtime cleanup to satisfy BLE001 budget * fix(bedrock): suppress bedrock close send errors per-message so promptEnd/sessionEnd still flush --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Mateo Wang
|
fde4c7c97a
|
feat(gdc): implement Google Distributed Cloud (GDC) Gemini provider (#31895)
* feat(gdc): add Google Distributed Cloud Gemini provider support Introduce support for the Google Distributed Cloud (GDC) Gemini provider by adding "gdc" to the list of chat providers and enabling the gdc/ model prefix. The implementation defines a new GDCGeminiConfig class which handles authentication via Google Distributed Cloud service account credentials, manages token generation, formats GDC Gemini request URLs, and transforms request structures accordingly The PreProcessNonDefaultParams class is also updated to exclude vertex parameters from filtering when the custom LLM provider is GDC, allowing vertex parameters to be passed properly during GDC initialization * fix: resolve issues identified in PR #30702 * fix(gdc): harden credentials, fix vertex param filtering, add tests The supports_vertex_params branch regressed vertex_ai and vertex_ai_beta: the `if custom_llm_provider in [...]: pass` was a no-op, so those providers fell through to the config lookup, found no supports_vertex_params, and had their vertex_ params stripped. The check is now a single _provider_supports_vertex_params helper that keeps vertex_ params for the vertex family and for any config that opts in, and only swallows the expected ValueError from an unknown provider string instead of a blanket except GDC project and location now resolve from the deployment's litellm_params and the litellm.vertex_project / litellm.vertex_location globals before falling back to request optional_params, matching how vertex_ai resolves them, so a proxy caller can no longer route a request to a project the deployment did not expose A request api_key is no longer treated as a filesystem path, so a caller can't make the host open a local service-account file; api_key must be a literal service-account JSON string or a bearer token The opt-in token cache is hardened: the lock and cache dict are created in __init__ instead of via a racy hasattr lazy-init, the token is read inside the lock, and the audience is stripped of a trailing slash once so the cached and non-cached paths agree Also declares gdc_api_base, switches the lazy-import entry to the relative path every other entry uses, adds the missing trailing comma in the provider config map, and drops the api_base fallback that only ran when api_key was None Adds unit tests covering the vertex-param filter, deployment-over-request precedence, the api_key file-path rejection, URL construction branches, environment validation, token caching, and the gdc completion dispatch; transformation.py is fully covered * fix(gdc): prefer GDC-specific config, honor vertex_ai aliases, harden URL and bool parsing * fix(gdc): mint the GDCH token audience from the host, not the full base When api_base embedded /v1/projects/... and the deployment set project/location, get_complete_url rebuilt the request URL from the host while validate_environment still derived the token audience from the full original api_base, so the bearer token could target a different audience than the URL actually called. The audience is now the scheme://host of api_base in every case, matching the host get_complete_url builds against * fix(gdc): restrict JSON api_key to GDCH service accounts Only accept a credential whose type is gdch_service_account before calling google.auth.load_credentials_from_dict, so a caller-supplied external_account/identity_pool/pluggable credential carrying arbitrary token or credential_source endpoints is rejected before any token refresh runs. GDC only ever uses GDCH service accounts, and non-GDCH credentials could not have completed auth anyway (with_gdch_audience is GDCH-only), so this narrows the credential-refresh surface without changing valid GDC behavior. * fix(gdc): validate project and location as plain identifiers vertex_project and vertex_location can come from request params and were interpolated as raw path text into the GDC request URL and the x-goog-user-project header. A caller-supplied value containing / ? # or .. could reshape the path and make the proxy send its GDC-authorized request to a different endpoint under the configured host. Validate both against a strict identifier pattern before building the URL or header and raise an auth error otherwise; GCP project ids and locations are plain identifiers so valid deployments are unaffected. * fix(gdc): bind x-goog-user-project quota header to the deployment The quota project header was resolved with request-level vertex_project taking effect, so with a preformed deployment api_base a caller could set vertex_project to a different project and have it sent under the proxy's GDC credential, misattributing quota or billing. Resolve the header project the same way the URL is resolved: a preformed api_base without a deployment override binds to the project embedded in the URL, otherwise deployment and global config win over request params. This keeps the URL and the quota header consistent. * fix(gdc): always rebind x-goog-user-project, stripping caller-forwarded values The quota project header was only set when absent, so with client header forwarding an authenticated caller could send their own x-goog-user-project (any casing) and have it ride on the proxy's GDC credential, bypassing the deployment-derived binding. Strip every casing of the header and always set it from _effective_project before the request is signed. * fix(gdc): make a preformed api_base authoritative for project routing get_litellm_params copies caller-supplied vertex_project and vertex_location into litellm_params via OPTIONAL_KWARGS_KEYS, so litellm_params cannot be treated as a deployment-only source. The previous _deployment_overrides_path inference let an authenticated caller flip a pinned preformed api_base such as /v1/projects/pinned/... to /v1/projects/attacker/..., driving requests to a caller-chosen project with the proxy's configured GDC credentials and quota header A preformed /v1/projects/ api_base is now authoritative; get_complete_url returns it unchanged and _effective_project binds the x-goog-user-project quota header to the project embedded in that URL, so a caller can no longer redirect a pinned deployment or move the quota header off it. The two tests that asserted the override behavior are now regression tests that fail if the rewrite is reintroduced * fix(gdc): make a preformed api_base self-sufficient in get_complete_url get_complete_url resolved and required a params-derived vertex_project before returning a preformed /v1/projects/ api_base, so a deployment that pins its project in the api_base path was forced to also pass vertex_project or hit 'project is required'. validate_environment already extracts the project from a preformed URL and needs no such param, so the two paths disagreed The preformed-URL early return now runs before project/location resolution, matching validate_environment: a preformed api_base is returned as-is with no redundant param, and non-preformed bases still require vertex_project and vertex_location as before. Adds a regression test that a preformed base with no project/location params returns the URL unchanged --------- Co-authored-by: Paige O'Connor <lostpaige@google.com> Co-authored-by: Tim Laubach <tlaubach@google.com> |
||
|
Mateo Wang
|
e141596204
|
refactor(lint): collapse type/lint budgets to a single per-rule limit (#31883)
* chore(lint): raise basedpyright per-rule slack to 50% of baseline The per-rule ceilings in basedpyright-code-budget.json sat at roughly 10% slack over baseline, which several in-flight PRs are already bumping into. Raise the slack on every rule to at least 50% of its baseline so there is ample headroom for a long while, while never lowering any rule that already had more generous slack (e.g. reportReturnType stays at 100). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(lint): collapse type/lint budgets to a single per-rule limit The three non-frontend budget files (ruff-strict, type-discipline, basedpyright-code) tracked a per-rule baseline and slack whose sum was the ceiling. Nothing consumed the split beyond that sum, so this replaces both keys with a single limit equal to the old baseline + slack; the original baselines live in git history if anyone needs them. The gate scripts and the ratchet guard now read limit directly. lint-budget-update no longer re-captures raw counts; it ratchets each rule's limit down by the number of violations this branch cleared since its branch point (the merge-base), so the granted headroom shrinks by exactly what was fixed and a limit never rises. The ratchet guard reads either schema so it still compares correctly across the migration boundary. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): surface staged-vs-working parity for pre-commit and budget-update make pre-commit selects which checks to run from the staged index but runs the linters over the working tree, so unstaged edits to tracked files and untracked files skew a green/red away from what a commit of only the staged changes would produce. There is no safe in-place way to lint the index, so the script now warns when unstaged or untracked changes are present, and CLAUDE.md documents that you must stage everything first for both make pre-commit and make lint-budget-update to predict CI correctly. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * docs(lint): list type-discipline budget in lint-budget-update instruction --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
92d0788da2
|
chore(lint): widen ANN slack to 10% of baseline and drop PLR0913 from the strict gate (#31335)
* chore(lint): widen ruff budget slack to 10% of baseline for high-volume ANN rules and PLR0913 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): drop PLR0913 from strict gate to roll out rules gradually Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(lint): ratchet-guard rising baselines even when slack is cut to mask them Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
ec268b0d18
|
refactor(completion): extract provider dispatch into typed helpers so basedpyright can analyze it (#30813) | ||
|
Mateo Wang
|
be4fa702e7
|
ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) (#30500)
* ci(lint): enforce type-discipline budget for casts and type guards Add a ratcheted gate that blocks net-new typing.cast() usage and bans TypeGuard/TypeIs outright, layered on the existing ruff-strict budget setup. - ruff-strict.toml: ban cast/TypeGuard/TypeIs (typing + typing_extensions) via flake8-tidy-imports banned-api (TID251) for a coarse import-level freeze. - ruff-strict-budget.json: bump TID251 baseline 2404 -> 2662 to absorb the ~258 pre-existing usages now matched by the new banned-api entries. - scripts/check_type_discipline.py: AST checker adding LIT006 (cast call sites, suppress with `# cast-ok: <reason>`) and LIT007 (TypeGuard/TypeIs annotations, suppress with `# guard-ok: <reason>`) for per-call-site granularity. - scripts/type_discipline_gate.py: baseline+slack gate with delta-vs-base, mirroring ruff_strict_gate.py. - type-discipline-budget.json: LIT006 baseline 1013 (slack 10), LIT007 0/0. - test-linting.yml: run the gate in CI against the PR base SHA. * ci(lint): enforce suppression-reason budgets and guard budgets against loosening - wire the **kwargs ban (LIT008) into the vendored type-discipline checker so it matches the budget that already referenced it - freeze LIT003/LIT004 (noqa / type-ignore without codes or reason) and LIT005 (*-ok suppression without a reason) at slack 0 so any net-new unexplained suppression trips the type-discipline gate - add scripts/budget_ratchet_check.py and a separate, non-gating budget-ratchet CI job that turns red when any *-budget.json ceiling is raised, a rule is dropped, or a budget file is deleted * ci(lint): ban mutable collections in annotations and all mutable construction Expand LIT001 from coarse builtins at interfaces to any mutable collection in any annotation (builtins, typing aliases, collections concretes, mutable ABCs) across signatures, class attributes, locals, and globals. Add LIT009 to flag mutable-collection construction (literals, comprehensions, constructors) so the unannotated seed-then-mutate pattern is caught too. Enumerate any-ok in LIT005 so its reason requirement holds even when only the stdlib checker runs. Budget LIT001 (21452) and LIT009 (25222) with slack 10 to ratchet down. * ci(lint): recommend pydantic at boundaries and add functional-refactor guidance Drop the msgspec mention from the cast banned-api messages so the recommended validation path matches the codebase's primary pattern (pydantic). Add a note to CLAUDE.md that lint / type-discipline failures should be resolved by refactoring to functional, immutable patterns rather than reaching for mutable structures or `# mutable-ok`. * style: make CLAUDE.md more concise * chore: update CLAUDE.md guidelines * ci(lint): renumber mutable construction LIT009 -> LIT002 next to LIT001 Group the mutable-collection family together: LIT001 (mutable collection in any annotation) and the construction rule now sit adjacent at LIT001/LIT002. The freed LIT009 slot is taken by the sibling Any gate (check_any_discipline.py, #30379), which moves its Any-typed-value rule LIT002 -> LIT009 in lockstep so the shared LIT namespace stays contiguous with no holes. Budget, gate docstring, and the checker's own docstring/messages are updated to match. * fix: numbering in CLAUDE.md * test(lint): test type-discipline checker, scope LIT007 to return types Add regression tests for check_type_discipline.py (every LIT rule, its suppression, and the comment scanner) and for budget_ratchet_check.py. Confine LIT007 to function return annotations, the only place TypeGuard/TypeIs are valid, so a runtime name that merely reads those identifiers is no longer flagged. Switch scan_comments to io.StringIO(source).readline, the standard readline that returns '' at EOF, dropping the iter(...).__next__ idiom. * fix(lint): best-effort worktree teardown so cleanup can't mask the real error base_counts ran `git worktree remove` through the raising `_run` in its finally, so a failed `git worktree add` (or a failure in the body) was masked by a second SystemExit from the cleanup. Tear the worktree down best-effort, like the sibling rmtree, so the original error propagates. * fix(lint): ratchet fails loudly on an unresolvable base; drop dead checker state Verify the merge-base ref resolves to a commit before trusting a missing-file result from git show, so an invalid or empty BASE_SHA now turns the budget-ratchet guard red instead of skipping every budget and passing vacuously Also drop the unused Comments.by_line field and the phantom --changed-only usage line from check_type_discipline's docstring, and cover the ref handling with tests * fix(lint): degrade malformed source to LIT000 instead of crashing the checker tokenize.generate_tokens raises IndentationError (a SyntaxError subclass) on a dedent mismatch, which escaped scan_comments' tokenize.TokenError handler and crashed the whole checker run, zeroing the gate for that invocation. Catch SyntaxError too so the file falls through to ast.parse and is reported as LIT000, matching the checker's graceful-degradation contract. Also add the trailing newline ruff-strict.toml lacked * perf(lint): skip the base worktree scan when no rule is over its ceiling cmd_check created a git worktree and re-scanned the base tree on every run, but a rule can only breach when its head count is already over baseline + slack; when none are, the base comparison cannot change the verdict. Short-circuit to OK in that case, which is every green PR, roughly halving the gate's work. Extract over_ceiling and cover it (and evaluate's drift-safety) with tests * fix(lint): exempt .dict()/.list()/.set() method calls from LIT002 _construction_kind matched dict/list/set as constructors via func.attr too, flagging common method calls like pydantic's model.dict() as mutable construction; 200 such false positives existed in litellm. Recognize dict/list/set construction only when unqualified while keeping the collections concretes (deque/defaultdict/...) matchable as attributes, since those are rarely method names. Ratchet the LIT002 baseline down 25222 -> 25022 to reflect the removed false positives * chore(lint): bump basedpyright ceilings to absorb staging base drift The basedpyright gate added in #30379 is a total-count check against basedpyright-code-budget.json and the linting workflow runs only on pull_request, so pushes to litellm_internal_staging never re-baseline it. Merging staging into this branch surfaced that drift: seven reportAny/reportUnknown* rules sit 10-149 errors above their committed ceiling even though this PR changes no files under litellm/, the only path basedpyright scans (pyrightconfig include is litellm). The new baselines match the counts CI measured on the merge commit, with the existing per-rule slack preserved * fix(lint): ratchet guard watches every budget file, not just two DEFAULT_BUDGETS only listed ruff-strict-budget.json and type-discipline-budget.json, so mypy-code-budget.json and basedpyright-code-budget.json were unguarded and their ceilings could rise with no signal, which is exactly the failure mode this guard exists to prevent. The gap became concrete when this PR bumped basedpyright-code-budget.json to absorb staging drift. All four budgets are now watched, so the budget-ratchet job surfaces that basedpyright bump for human review the same way it surfaces the TID251 raise. A regression test pins that every *-budget.json on disk is in DEFAULT_BUDGETS, failing loudly if a future budget escapes the ratchet * fix: add a lot more slack * fix(lint): restore LIT003 frozen slack to 0 The blanket slack bump set LIT003 (bare # noqa without codes or a reason) to a slack of 50, which contradicts the documented zero-tolerance invariant: the gate docstring and the PR description table both freeze LIT003/LIT004/LIT005 at slack 0 so any net-new unexplained suppression trips the gate. Slack 50 would let 50 new bare noqas through silently. The actual LIT003 count is 397, well under the 516 baseline, so restoring slack to 0 keeps the gate green while putting the freeze back. LIT004/LIT005/LIT007 were already correct at 0 * fix(lint): restore documented slack 10 for the buffered LIT rules The slack bump left LIT001/LIT002/LIT006/LIT008 at 2000/2500/100/100, 10-250x the "/ 10" the PR description table and the gate docstring document. That buffer was never needed: the gate already blames a rule only when its count exceeds the ceiling and grew vs the merge-base, so the violations the staging merge added in litellm/ sit in both head and base and are never charged to this PR. With slack back at the documented 10 the gate stays green, and the ceiling is tight again (LIT006 no longer waves through 99 net-new cast() calls). Baselines are unchanged; only the slack returns to its documented value * fix(lint): ratchet LIT003 baseline down to its actual count The LIT003 baseline was 516 while the current bare-noqa count is 397, leaving ~119 units of headroom that undercut the documented zero-tolerance freeze: the gate docstring claims any net-new bare noqa trips the gate, but with cap 516 a PR could add over a hundred first. Drop the baseline to the measured 397 so the freeze is exact (cap = 397 + slack 0), the same hard-zero-at-the-boundary shape LIT005 and LIT007 already use and pass in CI. PR table row updated to 397 / 0 * fix: increase slack * fix: increase slack * docs(lint): align gate docstring with buffered LIT003/LIT004 slack The budget now gives LIT003/LIT004 nonzero slack, so the gate's prose no longer claims they are frozen at slack 0; LIT005 remains the reasonless- suppression freeze and LIT007 the hard zero. |
||
|
Mateo Wang
|
d0c2e87810
|
ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379)
* ci: enable ruff preview rules under the budgeted strict gate
Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only,
leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at
zero. Enumerate the 118 firing codes explicitly with
explicit-preview-rules so the gate is deterministic and stable across
ruff upgrades rather than depending on preview auto-selecting the broad
catalog.
Grandfather the existing 58438 violations into ruff-strict-budget.json
as per-rule baselines with headroom, so only net-new violations fail CI.
The existing ten rules keep their hand-tuned slack; the new rules get
slack 10 when the baseline is 50 or more and 3 otherwise.
* ci: add ANN return-type rules to the budgeted strict gate
Add ANN201/202/204/205/206 (missing return annotations) to the strict
lane and grandfather the existing counts into ruff-strict-budget.json so
the codebase ratchets toward explicit return types without breaking CI.
* ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines
Add two type-check gates, each grandfathering the current tree so only
net-new violations fail CI, matching the ruff strict-budget ratchet.
mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI
invocation actually reads; the root [tool.mypy] is not picked up from the
litellm/ working dir). The 4885 existing missing-annotation errors are
captured in litellm/.mypy-baseline.txt and the run is piped through
mypy-baseline filter so new untyped defs are rejected.
basedpyright runs in strict mode over litellm/, with
enableTypeIgnoreComments disabled so it only honors '# pyright: ignore'
and never polices mypy's '# type: ignore'. The existing strict diagnostics
are grandfathered into .basedpyright/baseline.json.
Both tools are pinned in the dev group and uv.lock; the lint workflow and
Makefile run them filtered through their baselines, with
lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet.
* ci: raise lint job timeout to 15m for the basedpyright strict pass
* ci: pin pythonVersion 3.12 and regenerate baselines against merged base
Merge litellm_internal_staging so the baselines cover code the CI merge
includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped
the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin
pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is
reproducible across interpreter versions (CI runs 3.12).
* ci: regenerate basedpyright baseline against the frozen lint env
The previous baseline was generated with optional provider deps (azure,
google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's
dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors
not in the baseline. Regenerate after uv sync --frozen so the baseline
reflects the same dependency set the lint job sees.
* ci: regenerate basedpyright baseline on python 3.12 frozen env
The prior baseline still carried proxy-dev packages (e.g. prisma) that the
lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import
errors ungrandfathered. Regenerate in a python 3.12 venv synced to the
frozen lock with default groups only, so the baseline matches exactly what
CI sees.
* ci: replace type-check baselines with per-file count budgets
The mypy and basedpyright baselines were position-sensitive (and the
basedpyright one was a 27MB file), so ordinary line shifts churned them.
Replace both with a per-file count gate: scripts/type_check_gate.py reduces
each tool's output to errors-per-file and checks it against a committed
{file: max} budget, ignoring line and column numbers. A file fails only
when it gains more errors than its ceiling; debt can't be shuffled between
files because each file has its own cap and new files default to zero.
Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are
generated in the python 3.12 frozen lint env so they match CI. Drops the
mypy-baseline dependency; basedpyright runs without its native baseline.
ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update.
* ci: add a small per-file slack to the type-check gate
Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count
before failing, so a basedpyright inference ripple in an unrelated file
doesn't break the build over a couple of errors. Budgets still record exact
counts; the tolerance is applied at check time.
* ci: move type-check slack into the budget json and trim lint timeout
Make slack declarative: the budget is now {"slack": N, "files": {path: count}}
so the tolerance is tuned in JSON without editing the script, mirroring how
ruff-strict-budget.json carries its slack. --update preserves the existing
slack. Also drop the lint job timeout from 15m to 10m; the mypy and
basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a
comfortable margin.
* ci: collapse fully-adopted ruff categories and drop inert preview flag
ANN (all nine non-removed rules) and BLE (its only rule) were spelled out
code-by-code; replace each with its category selector, which is exactly
equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category
selector and error when named explicitly). explicit-preview-rules was inert:
every selected rule is stable and nothing is selected by category, so the flag
had nothing to gate. Verified the strict-rule counts are identical before and
after (62379 each, zero per-rule drift), so no budget change.
* ci: drop redundant pyright dev dependency
Nothing invokes bare pyright in the Makefile, the linting workflow, or
scripts; the basedpyright gate added on this branch is the only type
checker that runs. basedpyright is a superset fork that reads the same
pyrightconfig.json and honors the same "# pyright: ignore" comments, so
pyright==1.1.408 in the ci group was dead weight. Regenerated uv.lock
under the same exclude-newer cutoff so the only change is removing
pyright and its package stanza
* ci: un-weaken mypy and error on Any in basedpyright
mypy: enable warn_return_any, drop the valid-type silencer, and stop globally ignoring missing first-party imports via [mypy-litellm.*] ignore_missing_imports = False, which surfaced eight real broken litellm.* imports the blanket ignore was hiding; third-party imports stay ignored. The per-file budget moves 4888 -> 5799 (902 no-any-return, 1 valid-type, 8 import-not-found), all grandfathered so only net-new errors fail and the ceilings ratchet down
basedpyright: error on reportExplicitAny and reportAny. The per-file budget moves 117033 -> 148946 (6931 explicit-Any, 24954 Any-typed expressions), grandfathered the same way
* ci: add Any-discipline gate on changed lines under litellm/
Add scripts/check_any_discipline.py, a type-aware gate that fails when a
changed line holds a value typed Any -- including the X | Any unions that
mypy --strict / basedpyright accept (e.g. re.Match.group() -> str | Any,
json.loads() -> Any, bare dict -> dict[Any, Any]).
It reuses the repo's mypyc-compiled mypy 1.19 via a custom generic AST
walker (mypyc precludes subclassing TraverserVisitor), loads litellm/mypy.ini
for parity with lint-mypy, and uses a dedicated incremental cache
(.mypy_cache_any) with mtime+hash invalidation to force re-checks. Scope is
changed-lines-only so editing a legacy file never forces cleaning its
existing Any debt; suppress a genuine typed/untyped boundary with
# any-ok: <reason> (ANY002 requires the reason).
Wire it into the Makefile (lint-any, lint, lint-dev), a parallel
any-discipline CI job with its own actions/cache, .gitignore, and the
CLAUDE.md / CONTRIBUTING.md docs.
* ci: move Any-gate codes into the shared LIT namespace
Renumber the Any-discipline checker into the LIT*** scheme owned by
scripts/check_type_discipline.py (PR #30500) so the two checkers share one
rule namespace and suppression convention:
ANY001 -> LIT002 (Any-typed value; LIT002 was the retired/free slot)
ANY002 -> LIT005 (any-ok without a reason; the shared suppression-reason code)
ANY000 -> LIT000 (setup/build/read error; the shared error code)
Messages and behavior are unchanged; LIT005's text already matches the
"<token> requires a reason" shape used for cast-ok/guard-ok.
* ci: gate mypy and basedpyright per error rule, not per file
Switch the mypy/basedpyright budget gate from per-file error counts to
per-rule-code totals, mirroring the {rule: {baseline, slack}} shape of
ruff-strict-budget.json. A rule fails when its codebase-wide error count
exceeds baseline + slack, so violations are tracked by category rather
than by file location.
scripts/type_check_gate.py now parses mypy from its text output (trailing
[code]) and basedpyright from --outputjson (the JSON `rule` field), since
basedpyright's wrapped text diagnostics mis-attribute the rule on
continuation lines. Replace the *-file-budget.json files with freshly
captured *-code-budget.json baselines and update the Makefile, CI, and
CLAUDE.md accordingly.
* docs: prefer Pydantic validation over any-ok suppression
Point the Any-discipline guidance at validating Any with Pydantic (a model
or TypeAdapter that returns a typed value or raises) and frame
# any-ok as a last resort that should ideally never be used.
* chore: remove extraneous comment
* chore: make the CLAUDE.md more concise
* chore: clean up bloated CONTRIBUTING.md additions
* chore: make Makefile more concise
* ci: add the lint-budget-update target CLAUDE.md references
CLAUDE.md tells contributors to run make lint-budget-update, but the
target was never defined. Add it as an aggregate that re-captures the
ruff, mypy, and basedpyright budgets in one shot.
* ci: recapture mypy and basedpyright budgets in the lint env
The per-rule baselines were captured in a richer dependency env than the
CI lint job's uv sync --frozen, so CI resolved fewer types and reported
more errors than the budgets allowed (no-any-return 902 over cap 900, plus
several basedpyright reportUnknown* rules). Regenerate both in the frozen
env so they grandfather the true CI debt: mypy 5786 -> 5799 (no-any-return
890 -> 902, valid-type 1 restored), basedpyright 146213 -> 148942.
* ci: check out PR head sha in lint and any-discipline jobs
The default pull_request checkout uses refs/pull/N/merge, which folds the
latest base commits into HEAD. The diff-based gates (ruff delta, Any
discipline) then diff against the event's older base.sha and blame base's
own new commits on this branch; staging's otel-v2 and streaming changes
(#30326, #30485) tripped the Any gate on files this branch never touched.
Checking out the PR head sha makes the gates diff the real branch tip
against base, and pins the tree the mypy/basedpyright budgets were captured
against so their counts stay deterministic as the base advances.
* ci(lint): renumber Any-typed-value rule LIT002 -> LIT009
Free up LIT002 for the sibling type-discipline gate (check_type_discipline.py,
#30500), which groups its mutable-collection family at LIT001 (annotation) and
LIT002 (construction). This gate's Any-typed-value rule moves to LIT009 so the
shared LIT namespace stays contiguous with no holes; LIT000 and LIT005 are
unchanged.
* style: rename lint-strict-budget -> lint-ruff-budget
* ci: harden type-check gates against silent passes (greptile review)
type_check_gate.py: refuse to certify a vacuous run. The CI pipe swallows
the tool's exit code ('tool || true'), so a crashed mypy/basedpyright that
emits nothing would parse to zero errors, breach no ceiling, and pass.
is_vacuous_run() now fails when nothing was parsed but the budget expects
errors. Also wrap basedpyright's json.loads in a JSONDecodeError handler
that prints the offending output instead of dumping a raw traceback.
check_any_discipline.py: ALL_LINES was None, which dict.get() also returns
for a path absent from the line map, so a path-normalisation mismatch could
let a violation on an unchanged file pass the scope filter. Make ALL_LINES a
distinct sentinel object so 'whole file' and 'path missing' are unambiguous.
Adds tests for all three.
|
||
|
ryan-crabbe-berri
|
c90eb7e96f
|
feat: ruff strict-rule suppressions baseline gate (#30303)
* feat: add ruff strict-rule suppressions baseline gate Introduce a stricter ruff rule set (typed params, no Any, complexity and arg-count caps, mutable-default and global-rebinding checks) grandfathered against the current tree and enforced as a budget rather than zero-tolerance ruff-strict.toml defines the 9 rules separately from ruff.toml so the existing ruff check stays green. scripts/ruff_suppressions.py builds the per-file, per-rule baseline in ruff-suppressions.json and gates CI by failing when the total grows past the baseline plus a 0.5% slack margin. The baseline ratchets down via `make lint-suppressions-update` after fixes * fix: surface per-file drift as a warning on a passing suppressions check Greptile flagged that cmd_check computed per-file regressions but only printed them on failure, so violations shifted between files (or a brand-new file under the slack) passed with a silent OK. Print them as a non-fatal warning on the pass path too; pass/fail behavior is unchanged * refactor: gate strict ruff rules on the delta vs base, not a frozen baseline The committed total-count baseline went stale against a moving base. CI lints the PR merged with the current staging tip, so violations merged by other PRs counted against this PR and tripped the budget even though nothing here touched them Replace it with a drift-proof gate. scripts/ruff_strict_gate.py runs ruff on the head, keeps only violations on lines this change adds relative to the merge-base, and fails when a rule exceeds its per-rule allowance in ruff-strict-budget.json (all 0 today). Because the base is measured live, base drift cancels out and only what the change introduces is gated. Drops ruff-suppressions.json and the old suppressions script * chore: allow 5 new ANN001/ANN003/ANN401 per change Give the three annotation-completeness rules a small per-change allowance so a large new module is not blocked over a few untyped params or kwargs, while the correctness and structural rules (B006, C901, PLR0913, PLW0603, RUF012, ANN002) stay at 0 * feat: add TID251 typing.Any/Dict import ban and widen annotation budgets Add TID251 (flake8-tidy-imports banned-api) to ruff-strict.toml, banning new imports of typing.Any and typing.Dict and steering new code toward structured types. It counts the import site, about one per file, so it is set non-blocking at 50 as a forward-looking signal Widen the annotation-completeness budgets so they nudge rather than block: ANN001 50, ANN401 50, ANN003 25. Correctness and structural rules stay at 0 * refactor: make the strict gate a drift-safe per-rule total ceiling Switch the gate from a per-change allowance to a hard ceiling on each rule's total count across the codebase. The ceiling is baseline + slack in ruff-strict-budget.json, with baseline captured from today's tree To stay drift-safe, the gate counts each rule on the head and on the merge-base (via a throwaway git worktree) and fails a rule only when its head total is over the ceiling and higher than the base, so base drift never blames a change that did not add to that rule. Annotation rules keep generous slack (ANN001 and ANN401 50, ANN003 25, TID251 50); structural and correctness rules are frozen at today's count. Add make lint-strict-budget-update to re-capture baselines * chore: give the structural strict rules a cushion of 3 To be liberal to start, B006, C901, PLR0913, PLW0603, RUF012, and ANN002 each get a slack of 3 instead of 0, so an occasional legitimate case is not hard-blocked. The annotation budgets are unchanged, and these ratchet down later * feat: ban more typing collection aliases and tighten annotation slack to 10 Add typing.List, typing.Set, typing.MutableSequence, and typing.MutableMapping to the TID251 banned-api list, steering new code toward tuple, Sequence, Mapping, frozenset, and frozen dataclasses. This raises TID251's baseline to 2404 Bring the three rules that were at slack 50 (ANN001, ANN401, TID251) down to 10 * docs: document the strict-gate ratchet and Any-avoidance in CLAUDE.md Add a line on running make lint-strict-budget-update to knock baselines down after fixes, and a line on validating untyped inputs in the caller rather than spending the Any budget * feat: make it a bit more strict --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |