* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Four more ruff rules for code the test suite runs but never checks. F601 is the
one that paid: the duplicate key it flagged in a get_form_data fixture was the
mock reproducing the production bug fixed in the previous commit.
B025 removed two unreachable handlers, one of them a pytest.skip shadowed by an
earlier `pass`, so an upstream Vertex flake reported green having asserted
nothing. F632 turned an `is ""` identity check, which passes only on CPython
interning, into the `== ""` it meant. B023 fixed three closures over loop
variables, all latent today but one iteration-order change away from checking the
last case N times.
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
* Cato Networks guardrail, based on Aim (#26597)
* Aim was acquired by Cato Networks, creating Cato Networks guardrail based on Aim
* Add more tests
* Move test so they are reached by codecov coverage
* base URL trailing slashes
* Support Lemonade runtime context metadata (#28135)
* Support Lemonade runtime context metadata
* Add provider hook for runtime model metadata
* Address provider model info review feedback
Keep the runtime model info hook duck-typed instead of extending the base model-info class, and avoid importing ModelInfoBase from Ollama common utilities to reduce CodeQL cyclic-import noise.
Co-authored-by: openhands <openhands@all-hands.dev>
* Fix CI after staging rebase
Relax the Ollama runtime metadata return annotation to match the provider-hook dict response and update the Google Interactions OpenAPI status expectation for the current live spec.
Co-authored-by: openhands <openhands@all-hands.dev>
* Normalize Lemonade runtime model metadata
* Avoid leaking Ollama metadata auth
* Avoid leaking Lemonade metadata auth
---------
Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(cato): address guardrail review feedback
Use proxy-authenticated user identity, forward moderation hook return values,
and ensure streaming sender tasks are cancelled and awaited on exit.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): route google/gemma-*-maas through partner-models OpenAI path - clone of #28010 (#28846)
* fix(vertex_ai): route google/gemma-*-maas through partner-models OpenAI path
Fixes#26083
vertex_ai/google/gemma-4-26b-a4b-it-maas previously fell through to the
NON_GEMINI route. Per owtaylor's plan on #26083: add the google/gemma-
prefix to PartnerModelPrefixes so is_vertex_partner_model picks it up
and should_use_openai_handler routes it to the OpenAI-compatible
/endpoints/openapi/chat/completions URL. No gemma-detection exclusion
needed (the "gemma/" check uses a slash, which google/gemma-... doesn't
match). No OpenAIGPTConfig subclass needed — works with the base handler.
* fix(vertex_ai): mark gemma-4-26b-a4b-it-maas as vision-capable (empirically verified)
* fix(vertex_ai): address greptile feedback — provider category, canonical URL, sync backup
* test(vertex_ai): add function-calling and vision pass-through tests for Gemma MaaS
Addresses oss-pr-review-agent-shin feedback on PR #28010:
supports_function_calling, supports_tool_choice, and supports_vision were
marked true but had no tests proving the payloads actually reached the
OpenAI-compatible endpoint.
Added:
- test_gemma_maas_supports_function_calling — verifies the utility returns True
when the model_cost entry carries supports_function_calling=true
- test_gemma_maas_supports_vision — same for supports_vision
- test_vertex_ai_gemma_function_calling_passthrough — verifies tools + tool_choice
appear in the JSON body POSTed to /endpoints/openapi/chat/completions
- test_vertex_ai_gemma_vision_passthrough — verifies image_url content parts
survive transformation and reach the global endpoint URL
* fix: Delete uv.lock
* test(vertex_ai): add function-calling and vision pass-through tests for Gemma MaaS
Addresses oss-pr-review-agent-shin feedback on PR #28010:
P1 (patch target): Added a comment explaining why patching
litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler is correct —
get_async_httpx_client() (defined in http_handler.py) instantiates
AsyncHTTPHandler within that module's scope, so the definition-site patch
intercepts it. Without the mock the test raises AuthenticationError,
confirming it never silently passes.
P2 (partner-provider regression guard): Added
test_gemma_routes_through_openai_handler() which calls
VertexAIPartnerModels.should_use_openai_handler() directly, so if Gemma's
routing to VertexPartnerProvider.llama ever changes the URL-shape tests
below it become a real regression guard rather than an unanchored unit test.
Also added:
- test_gemma_maas_supports_function_calling / supports_vision — capability
flag checks via patch.dict(litellm.model_cost)
- test_vertex_ai_gemma_function_calling_passthrough — tools + tool_choice
forwarded in the request body
- test_vertex_ai_gemma_vision_passthrough — image_url part survives
transformation to the global endpoint
Added:
- test_gemma_maas_supports_function_calling — verifies the utility returns True
when the model_cost entry carries supports_function_calling=true
- test_gemma_maas_supports_vision — same for supports_vision
- test_vertex_ai_gemma_function_calling_passthrough — verifies tools + tool_choice
appear in the JSON body POSTed to /endpoints/openapi/chat/completions
- test_vertex_ai_gemma_vision_passthrough — verifies image_url content parts
survive transformation and reach the global endpoint URL
* fix: proper patch for unit tests
---------
Co-authored-by: Iana <iana@Shivakumars-MacBook-Pro.local>
* fix(cato): guardrail all completion choices on output
When n > 1, only choices[0] was analyzed and redacted. Iterate every
Choices entry so block and anonymize actions apply to all completions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix review
* fix(cato_networks): harden output anonymize handling and restructure nested UI routes
Guard against empty redacted_output and empty all_redacted_messages from Cato.
Restructure nested admin UI HTML exports to index.html so extensionless routes work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix mypy
* fix(cato): guard missing policy_drill_down and all_redacted_messages keys
* fix(cato): avoid KeyError bypassing block action on missing analysis_result
* fix(cato): preserve non-text message fields during anonymize
Rebuild redacted messages from the original messages, overwriting only
content, so tool_calls, tool_call_id, name and multimodal fields survive
the anonymize action.
* fix(cato): preserve trailing messages when fewer redacted messages returned
Avoid silently truncating the conversation in _anonymize_request when Cato
returns fewer redacted messages than were sent, and isolate the no-api-key
config test from a pre-existing CATO_API_KEY environment variable.
* fix(cato,model-info): preserve stream block signal on sender teardown; forward api_key in dynamic model-info lookup
Suppress ConnectionClosed (alongside CancelledError) when tearing down the
Cato streaming sender task so a backend ConnectionClosed cannot mask the
original StreamingCallbackError (e.g. a guardrail block) raised by the
receive loop.
Thread api_key through get_model_info -> _get_model_info_helper so an
explicit key reaches a provider's dynamic get_model_info for a caller-supplied
api_base. Previously only api_base was forwarded, so authenticated Ollama and
Lemonade servers at a custom base could only be queried unauthenticated.
* fix(cato): surface mid-stream forwarding errors instead of blocking on recv
If the upstream LLM stream errors mid-flight, the sender task dies before
sending the terminal done frame, so the consumer would block on websocket.recv()
until Cato closes the connection. Race recv against the sender task and raise the
stored sender exception promptly as a StreamingCallbackError.
* fix(cato): drop spoofable end_user_id from guardrail user identity
Only the key/JWT-bound user_email is a trusted identity. end_user_id is
resolved from caller-supplied request fields (OpenAI user param, headers,
metadata), so an authenticated caller with no bound user_email could set it
to another user's email and have LiteLLM forward x-cato-user-email for that
victim, poisoning Cato audit and policy attribution. Forward only user_email
and omit the header otherwise.
* fix(cato): harden output anonymize path against missing content key
* fix(cato): fall back to original message when redacted content key is missing
* refactor(model-info): drop unused api_key from cached model-info helper
_cached_get_model_info_helper is only called by the cost-tracking hot path,
which never authenticates, so the api_key parameter was never populated.
Keeping it in the lru_cache key offered no benefit and risked fragmenting
the high-RPS cache and retaining credential strings per entry.
* fix(cato): preserve None content on tool-call-only choices in output hook
* fix(ollama): respect static-model guard in OllamaConfig.get_model_info
Delegate to OllamaModelInfo.get_model_info so statically-priced Ollama
models short-circuit before the /api/show network call instead of
hitting the server unconditionally.
* fix(lemonade,ollama): treat empty api_key as unset to avoid leaking server creds
An empty-string api_key was treated as an explicit key, so it passed the
guard meant to keep server-side credentials off caller-supplied bases and
then fell back through the env/global key chain. A caller could point
api_base at a server they control and send api_key="" to receive the
configured provider key in the Authorization header. Gate the credential
fallback on the api_key being truthy instead of merely not-None.
* fix(cato): inspect and redact Responses-API input, not just messages
The guardrail only read data["messages"], so /v1/responses requests, which
carry their text in data["input"], reached Cato as an empty message list
and bypassed inspection entirely. Send build_inspection_messages(data) so
both shapes are analyzed, and write anonymized results back with
apply_redacted_messages_back when the request used input.
* perf(utils): keep api_key out of get_model_info lru_cache key
* fix(cato): propagate ssl_verify to streaming WebSocket connection
The streaming hook applied ssl_verify only to the HTTP handler; the
websockets.connect() call used default verification, so a custom Cato
instance behind TLS with a self-signed cert worked for non-streaming
calls but failed every streaming request. Resolve the ssl_verify setting
into the connect() ssl argument, mirroring the HTTP handler.
* refactor(utils): rename shadowing local in _get_model_info_helper
* fix(cato): flatten multimodal chat content before inspection
Chat Completions requests whose message content is a multimodal parts
array were posted to Cato as the raw OpenAI parts, so text inside
content: [{"type":"text", ...}] reached the model without Cato ever
inspecting the string. Flatten each message's list content to plain text
while keeping the list 1:1 with the request so the index-based redaction
write-back stays valid; Responses-API input requests still go through
build_inspection_messages.
* test(lemonade): clear get_model_info cache around api_base test
* fix(cato): inspect and redact Responses-API input even when messages present
_inspection_messages returned early once messages was non-empty, so a
/v1/responses caller could place benign text in messages and disallowed
text in input and have only messages reach Cato while the model used
input. Inspect both fields and write anonymize redactions back to input
as well as the index-aligned messages.
* test(log_db_metrics): assert table_name event_metadata contract
log_db_metrics now emits minimal event_metadata via _safe_db_event_metadata
(table_name only, function_name/function_kwargs/function_args dropped as
redundant with call_type and unsafe to stamp on a span). The success-path
test still asserted function_name membership and crashed with TypeError on
the None metadata returned when no table_name is passed. Pass a table_name
and assert the surfaced contract instead.
* fix(cato): inspect and redact completion prompt and Responses-API instructions
The Cato guardrail only inspected chat messages and the Responses-API input field, so blocked text placed in the legacy /v1/completions prompt or the /v1/responses instructions field reached the model without ever being sent to Cato. Both fields are now appended as synthetic inspection messages, and the anonymize path slices Cato's redactions back to the field they came from.
* fix(cato): serialize non-str/bytes websocket chunks before forwarding
* fix(cato): inspect tool descriptions and tool-call arguments
* fix(cato): map redacted output by assistant index; restore get_model_info.cache_info
* fix(cato): block output even when detection_message is null/empty
A block_action returned by Cato on the output hook whose detection_message
was null or empty was let through to the caller: the truthiness guard on
detection_message skipped the HTTPException and the unblocked response was
returned. Raise the HTTPException directly in _handle_block_action_on_output
so the output path blocks unconditionally, mirroring the input path.
* fix(cato): inspect and redact nested tool param and legacy function descriptions
Tool/function parameter descriptions and the legacy functions[] array are
forwarded to the model but were not seen by Cato, so blocked text hidden there
bypassed inspection and anonymization. Recursively walk every description string
in tools[].function and functions[] schemas for both the analyze payload and the
anonymize write-back.
* fix(cato): traverse schema descriptions iteratively to satisfy recursive detector
The nested walk() generator recursed over tool/function JSON schemas with no
depth bound, which the recursive_detector code-quality gate rejects. Replace it
with an explicit-stack DFS that yields the same (container, key) refs in the
same pre-order, so schema description redaction is unchanged.
* fix(cato): inspect and redact response_format JSON schema descriptions
response_format json_schema descriptions are forwarded to the model, so
blocked text hidden in nested schema descriptions could bypass Cato
inspection and redaction. Extend the schema-description walk to cover
response_format alongside tools and legacy functions.
* fix(cato): skip output rewrite when Cato returns no redaction
Return None from call_cato_guardrail_on_output on monitor/no-action so the
post-call hook only mutates the message when there is an actual redaction,
instead of redundantly re-writing the original content.
* refactor(utils): resolve explicit api_key model info without the cache
Move the model-info build into a non-cached _build_model_info helper and drop
api_key from the lru-cached _cached_get_model_info signature. Both cached
helpers now take the same (model, provider, api_base) key and never forward
api_key, while explicit per-caller keys are resolved through the builder
directly instead of reaching into the cache wrapper's __wrapped__.
* fix(cato): inspect and redact non-description schema string values
Tool, function and response_format JSON schemas forward more than just
description text to the model. enum, const, default, examples and title
values are sent verbatim, so blocked content hidden in any of them
bypassed Cato inspection and redaction. Walk those schema string values
alongside descriptions on both the inspection and anonymize paths.
* fix(model-info): surface swallowed dynamic model-info errors
The provider-specific get_model_info dispatch falls back to the static cost
map when a provider's dynamic lookup raises, which is intentional graceful
degradation. Previously the exception was discarded with a bare debug line,
so a real failure (e.g. a provider whose get_model_info signature does not
accept api_key) was invisible. Log the exception at warning level with the
model and provider context so the fallback is diagnosable.
* fix(cato): inspect and redact Responses API output in post-call hook
The post-call success hook only handled ModelResponse, so /v1/responses
(which returns a ResponsesAPIResponse) bypassed the Cato output guardrail.
Extract and inspect/redact every output_text content block and function-call
arguments string, blocking on a block action, so generated text cannot escape
inspection by using the Responses API.
* chore: reset _experimental/out folder
* chore(ui): remove orphaned prebuilt dashboard chunk files
The _experimental/out manifests are byte-identical to the base branch, so the
served dashboard already matches base. 436 unreferenced Next.js chunk files had
accumulated in the directory and are not loaded by any manifest; removing them
restores the committed UI artifacts to the base build and drops the artifact
churn from this PR's diff.
* fix(guardrails,ollama): forward ssl_verify to Cato init and raise_for_status on /api/show
---------
Co-authored-by: Alex Yaroslavsky <trexinc@gmail.com>
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Piotr Placzko <piotr@icep-design.com>
Co-authored-by: Iana <iana@Shivakumars-MacBook-Pro.local>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ollama): Include provider in model list for ollama (#26135)
* Include provider in model names for ollama
* Fix unit tests
* fix(ollama): process both thinking and content in same streaming chunk (#26098)
* fix(health_check): skip max_tokens for image_generation mode (#26417)
* fix(health_check): skip max_tokens for image_generation mode
`_update_litellm_params_for_health_check` injected `max_tokens` for
every deployment. OpenAI `/v1/images/generations` strictly rejects
unknown fields, so health checks for dall-e-* and gpt-image-1 always
failed with `400 "Unknown parameter: 'max_tokens'"` even though the
actual image endpoint calls succeed. Skip the `max_tokens` injection
when `model_info.mode == "image_generation"`. `messages` still gets
injected (downstream `_filter_model_params` already strips it for
non-chat handlers).
* Switch to allow-list with per-deployment override
Per @krrishdholakia review: deny-listing image_generation only re-introduces
the same bug for every other non-chat mode (embedding, audio_*, rerank,
video_generation, ocr, search, moderation, ...).
Replace the single image_generation skip with `_MAX_TOKEN_SUPPORT_MODES =
{chat, completion, responses}`. Missing `mode` is treated as chat for
backward compatibility. New modes are safe by default.
Add `model_info.health_check_supports_max_tokens` as an operator escape
hatch — True forces injection on a non-listed deployment (operator wants
to bound probe tokens), False suppresses it on a chat-style deployment
behind a strict-schema provider.
Tests: parametrize over 3 chat-style + 10 non-chat modes, plus override
on/off and the no-mode legacy path.
* fix(http_handler): handle RequestNotRead in MaskedHTTPStatusError for multipart uploads (#26718)
Squash-merged by litellm-agent from dawidkulpa's PR.
* fix(ollama): guard against double 'ollama/' prefix in live model listing
Greptile flagged that Ollama servers can return names that already start
with 'ollama/'. Check the prefix before prepending so we don't produce
'ollama/ollama/...'. Adds a regression test.
* Fix Ollama empty reasoning stream chunks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: James Myatt <james@jamesmyatt.co.uk>
Co-authored-by: VHash <225398745+vhash0@users.noreply.github.com>
Co-authored-by: hayden <sewhan.kim+@a-bly.com>
Co-authored-by: dawidkulpa <84176950+dawidkulpa@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
tool_calls on assistant messages were translated to OllamaToolCall format
but never copied into the outgoing OllamaChatCompletionMessage, so Ollama
received {role: assistant, content: ''} with no tool_calls. The model
then had no record of having made a tool call, causing it to re-issue
the identical call on every turn (infinite loop).
Similarly, tool_call_id on role:tool messages was silently dropped.
Ollama uses this field to resolve the tool name from conversation history.
Also add tool_call_id to OllamaChatCompletionMessage TypedDict.
Fixes#26094
* fix(ollama): propagate done_reason='length' as finish_reason for max_tokens truncation
Ollama returns done_reason='length' when a response is cut off by num_predict
(the max_tokens limit). Previously, non-streaming responses hardcoded
finish_reason='stop', and streaming used chunk.get('done_reason', 'stop')
which also defaulted to 'stop' when done_reason was absent.
This meant callers (e.g. the Anthropic pass-through adapter, which maps
OpenAI 'length' -> Anthropic 'max_tokens') could never detect truncation,
making stop_reason always appear as 'end_turn' even for cut-off responses.
Fix: read done_reason from the response JSON in the non-streaming path and
use `chunk.get('done_reason') or 'stop'` in the streaming path, so Ollama's
actual done_reason passes through to the caller unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update test_ollama_chat_transformation.py
* Update litellm/llms/ollama/chat/transformation.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* auth_with_role_name add region_name arg for cross-account sts
* update tests to include case with aws_region_name for _auth_with_aws_role
* Only pass region_name to STS client when aws_region_name is set
* Add optional aws_sts_endpoint to _auth_with_aws_role
* Parametrize ambient-credentials test for no opts, region_name, and aws_sts_endpoint
* consistently passing region and endpoint args into explicit credentials irsa
* fix env var leakage
* fix: bedrock openai-compatible imported-model should also have model arn encoded
* feat: show proxy url in ModelHub (#21660)
* fix(bedrock): correct modelInput format for Converse API batch models (#21656)
* fix(proxy): add model_ids param to access group endpoints for precise deployment tagging (#21655)
POST /access_group/new and PUT /access_group/{name}/update now accept an
optional model_ids list that targets specific deployments by their unique
model_id, instead of tagging every deployment that shares a model_name.
When model_ids is provided it takes priority over model_names, giving
API callers the same single-deployment precision that the UI already has
via PATCH /model/{model_id}/update.
Backward compatible: model_names continues to work as before.
Closes#21544
* feat(proxy): add custom favicon support\n\nAdd ability to configure a custom favicon for the litellm proxy UI.\n\n- Add favicon_url field to UIThemeConfig model\n- Add LITELLM_FAVICON_URL env var support\n- Add /get_favicon endpoint to serve custom favicons\n- Update ThemeContext to dynamically set favicon\n- Add favicon URL input to UI theme settings page\n- Add comprehensive tests\n\nCloses #8323 (#21653)
* fix(bedrock): prevent double UUID in create_file S3 key (#21650)
In create_file for Bedrock, get_complete_file_url is called twice:
once in the sync handler (generating UUID-1 for api_base) and once
inside transform_create_file_request (generating UUID-2 for the
actual S3 upload). The Bedrock provider correctly writes UUID-2 into
litellm_params["upload_url"], but the sync handler unconditionally
overwrites it with api_base (UUID-1). This causes the returned
file_id to point to a non-existent S3 key.
Fix: only set upload_url to api_base when transform_create_file_request
has not already set it, preserving the Bedrock provider's value.
Closes#21546
* feat(semantic-cache): support configurable vector dimensions for Qdrant (#21649)
Add vector_size parameter to QdrantSemanticCache and expose it through
the Cache facade as qdrant_semantic_cache_vector_size. This allows users
to use embedding models with dimensions other than the default 1536,
enabling cheaper/stronger models like Stella (1024d), bge-en-icl (4096d),
voyage, cohere, etc.
The parameter defaults to QDRANT_VECTOR_SIZE (env var or 1536) for
backward compatibility. When creating new collections, the configured
vector_size is used instead of the hardcoded constant.
Closes#9377
* fix(utils): normalize camelCase thinking param keys to snake_case (#21762)
Clients like OpenCode's @ai-sdk/openai-compatible send budgetTokens
(camelCase) instead of budget_tokens in the thinking parameter, causing
validation errors. Add early normalization in completion().
* feat: add optional digest mode for Slack alert types (#21683)
Adds per-alert-type digest mode that aggregates duplicate alerts
within a configurable time window and emits a single summary message
with count, start/end timestamps.
Configuration via general_settings.alert_type_config:
alert_type_config:
llm_requests_hanging:
digest: true
digest_interval: 86400
Digest key: (alert_type, request_model, api_base)
Default interval: 24 hours
Window type: fixed interval
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add blog_posts.json and local backup
* feat: add GetBlogPosts utility with GitHub fetch and local fallback
Adds GetBlogPosts class that fetches blog posts from GitHub with a 1-hour
in-process TTL cache, validates the response, and falls back to the bundled
blog_posts_backup.json on any network or validation failure.
* test: add cache reset fixture and LITELLM_LOCAL_BLOG_POSTS test
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add GET /public/litellm_blog_posts endpoint
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: log fallback warning in blog posts endpoint and tighten test
* feat: add disable_show_blog to UISettings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add useUISettings and useDisableShowBlog hooks
* fix: rename useUISettings to useUISettingsFlags to avoid naming collision
* fix: use existing useUISettings hook in useDisableShowBlog to avoid cache duplication
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add BlogDropdown component with react-query and error/retry state
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: enforce 5-post limit in BlogDropdown and add cap test
* fix: add retry, stable post key, enabled guard in BlogDropdown
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add BlogDropdown to navbar after Docs link
* feat: add network_mock transport for benchmarking proxy overhead without real API calls
Intercepts at httpx transport layer so the full proxy path (auth, routing,
OpenAI SDK, response transformation) is exercised with zero-latency responses.
Activated via `litellm_settings: { network_mock: true }` in proxy config.
* Litellm dev 02 19 2026 p2 (#21871)
* feat(ui/): new guardrails monitor 'demo
mock representation of what guardrails monitor looks like
* fix: ui updates
* style(ui/): fix styling
* feat: enable running ai monitor on individual guardrails
* feat: add backend logic for guardrail monitoring
* fix(guardrails/usage_endpoints.py): fix usage dashboard
* fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo (#21754)
* fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo
* fix(budget): update stale docstring on get_budget_reset_time
* fix: add missing return type annotations to iterator protocol methods in streaming_handler (#21750)
* fix: add return type annotations to iterator protocol methods in streaming_handler
Add missing return type annotations to __iter__, __aiter__, __next__, and __anext__ methods in CustomStreamWrapper and related classes.
- __iter__(self) -> Iterator["ModelResponseStream"]
- __aiter__(self) -> AsyncIterator["ModelResponseStream"]
- __next__(self) -> "ModelResponseStream"
- __anext__(self) -> "ModelResponseStream"
Also adds AsyncIterator and Iterator to typing imports.
Fixes issue with PLR0915 noqa comments and ensures proper type checking support.
Related to: BerriAI/litellm#8304
* fix: add ruff PLR0915 noqa for files with too many statements
* Add gollem Go agent framework cookbook example (#21747)
Show how to use gollem, a production Go agent framework, with
LiteLLM proxy for multi-provider LLM access including tool use
and streaming.
* fix: avoid mutating caller-owned dicts in SpendUpdateQueue aggregation (#21742)
* fix(vertex_ai): enable context-1m-2025-08-07 beta header (#21870)
* server root path regression doc
* fixing syntax
* fix: replace Zapier webhook with Google Form for survey submission (#21621)
* Replace Zapier webhook with Google Form for survey submission
* Add back error logging for survey submission debugging
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* Revert "Merge pull request #21140 from BerriAI/litellm_perf_user_api_key_auth"
This reverts commit 0e1db3f7e4, reversing
changes made to 7e2d6f2355.
* test_vertex_ai_gemini_2_5_pro_streaming
* UI new build
* fix rendering
* ui new build
* docs fix
* docs fix
* docs fix
* docs fix
* docs fix
* docs fix
* docs fix
* docs fix
* release note docs
* docs
* adding image
* fix(vertex_ai): enable context-1m-2025-08-07 beta header
The `context-1m-2025-08-07` Anthropic beta header was set to `null` for vertex_ai,
causing it to be filtered out when users set `extra_headers: {anthropic-beta: context-1m-2025-08-07}`.
This prevented using Claude's 1M context window feature via Vertex AI, resulting in
`prompt is too long: 460500 tokens > 200000 maximum` errors.
Fixes#21861
---------
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* Revert "fix(vertex_ai): enable context-1m-2025-08-07 beta header (#21870)" (#21876)
This reverts commit bce078a796.
* docs(ui): add pre-PR checklist to UI contributing guide
Add testing and build verification steps per maintainer feedback
from @yjiang-litellm. Contributors should run their related tests
per-file and ensure npm run build passes before opening PRs.
* Fix entries with fast and us/
* Add tests for fast and us
* Add support for Priority PayGo for vertex ai and gemini
* Add model pricing
* fix: ensure arrival_time is set before calculating queue time
* Fix: Anthropic model wildcard access issue
* Add incident report
* Add ability to see which model cost map is getting used
* Fix name of title
* Readd tpm limit
* State management fixes for CheckBatchCost
* Fix PR review comments
* State management fixes for CheckBatchCost - Address greptile comments
* fix mypy issues:
* Add Noma guardrails v2 based on custom guardrails (#21400)
* Fix code qa issues
* Fix mypy issues
* Fix mypy issues
* Fix test_aaamodel_prices_and_context_window_json_is_valid
* fix: update calendly on repo
* fix(tests): use counter-based mock for time.time in prisma self-heal test
The test used a fixed side_effect list for time.time(), but the number
of calls varies by Python version, causing StopIteration on 3.12 and
AssertionError on 3.14. Replace with an infinite counter-based callable
and assert the timestamp was updated rather than checking for an exact
value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(tests): use absolute path for model_prices JSON in validation test
The test used a relative path 'litellm/model_prices_and_context_window.json'
which only works when pytest runs from a specific working directory.
Use os.path based on __file__ to resolve the path reliably.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update tests/test_litellm/test_utils.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(tests): use os.path instead of Path to avoid NameError
Path is not imported at module level. Use os.path.join which is already
available.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* clean up mock transport: remove streaming, add defensive parsing
* docs: add Google GenAI SDK tutorial (JS & Python) (#21885)
* docs: add Google GenAI SDK tutorial for JS and Python
Add tutorial for using Google's official GenAI SDK (@google/genai for JS,
google-genai for Python) with LiteLLM proxy. Covers pass-through and
native router endpoints, streaming, multi-turn chat, and multi-provider
routing via model_group_alias. Also updates pass-through docs to use the
new SDK replacing the deprecated @google/generative-ai.
* fix(docs): correct Python SDK env var name in GenAI tutorial
GOOGLE_GENAI_API_KEY does not exist in the google-genai SDK.
The correct env var is GEMINI_API_KEY (or GOOGLE_API_KEY).
Also note that the Python SDK has no base URL env var.
* fix(docs): replace non-existent GOOGLE_GENAI_BASE_URL env var in interactions.md
The Python google-genai SDK does not read GOOGLE_GENAI_BASE_URL.
Use http_options={"base_url": "..."} in code instead.
* docs: add network mock benchmarking section
* docs: tweak benchmarks wording
* fix: add auth headers and empty latencies guard to benchmark script
* refactor: use method-level import for MockOpenAITransport
* fix: guard print_aggregate against empty latencies
* fix: add INCOMPLETE status to Interactions API enum and test
Google added INCOMPLETE to the Interactions API OpenAPI spec status enum.
Update both the Status3 enum in the SDK types and the test's expected
values to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Guardrail Monitor - measure guardrail reliability in prod (#21944)
* fix: fix log viewer for guardrail monitoring
* feat(ui/): fix rendering logs per guardrail
* fix: fix viewing logs on overview tab of guardrail
* fix: log viewer
* fix: fix naming to align with metric
* docs: add performance & reliability section to v1.81.14 release notes
* fix(tests): make RPM limit test sequential to avoid race condition
Concurrent requests via run_in_executor + asyncio.gather caused a race
condition where more requests slipped through the rate limiter than
expected, leading to flaky test failures (e.g. 3 successes instead of 2
with rpm_limit=2).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: Singapore guardrail policies (PDPA + MAS AI Risk Management) (#21948)
* feat: Singapore PDPA PII protection guardrail policy template
Add Singapore Personal Data Protection Act (PDPA) guardrail support:
Regex patterns (patterns.json):
- sg_nric: NRIC/FIN detection ([STFGM] + 7 digits + checksum letter)
- sg_phone: Singapore phone numbers (+65/0065/65 prefix)
- sg_postal_code: 6-digit postal codes (contextual)
- passport_singapore: Passport numbers (E/K + 7 digits, contextual)
- sg_uen: Unique Entity Numbers (3 formats)
- sg_bank_account: Bank account numbers (dash format, contextual)
YAML policy templates (5 sub-guardrails):
- sg_pdpa_personal_identifiers: s.13 Consent
- sg_pdpa_sensitive_data: Advisory Guidelines
- sg_pdpa_do_not_call: Part IX DNC Registry
- sg_pdpa_data_transfer: s.26 overseas transfers
- sg_pdpa_profiling_automated_decisions: Model AI Governance Framework
Policy template entry in policy_templates.json with 9 guardrail definitions
(4 regex-based + 5 YAML conditional keyword matching).
Tests:
- test_sg_patterns.py: regex pattern unit tests
- test_sg_pdpa_guardrails.py: conditional keyword matching tests (100+ cases)
* feat: MAS AI Risk Management Guidelines guardrail policy template
Add Monetary Authority of Singapore (MAS) AI Risk Management Guidelines
guardrail support for financial institutions:
YAML policy templates (5 sub-guardrails):
- sg_mas_fairness_bias: Blocks discriminatory financial AI (credit/loans/insurance by protected attributes)
- sg_mas_transparency_explainability: Blocks opaque/unexplainable AI for consequential financial decisions
- sg_mas_human_oversight: Blocks fully automated financial decisions without human-in-the-loop
- sg_mas_data_governance: Blocks unauthorized sharing/mishandling of financial customer data
- sg_mas_model_security: Blocks adversarial attacks, model poisoning, inversion on financial AI
Policy template entry in policy_templates.json with 5 guardrail definitions.
Aligned with MAS FEAT Principles, Project MindForge, and NIST AI RMF.
Tests:
- test_sg_mas_ai_guardrails.py: conditional keyword matching tests (100+ cases)
* fix: address SG pattern review feedback
- Update NRIC lowercase test for IGNORECASE runtime behavior
- Add keyword context guard to sg_uen pattern to reduce false positives
* docs: clarify MAS AIRM timeline references
- Explicitly mark MAS AIRM as Nov 2025 consultation draft
- Add 2018 qualifier for FEAT principles in MAS policy descriptions
- Update MAS guardrail wording to avoid release-year ambiguity
* chore: commit resolved MAS policy conflicts
* test:
* chore:
* Add OpenAI Agents SDK tutorial with LiteLLM Proxy to docs (#21221)
* Add OpenAI Agents SDK tutorial to docs
* Update OpenAI Agents SDK tutorial to use LiteLLM environment variables
* Enhance OpenAI Agents SDK tutorial with built-in LiteLLM extension details and updated configuration steps. Adjust section headings for clarity and improve the flow of information regarding model setup and usage.
* adjust blog posts to fetch from github first
* feat(videos): add variant parameter to video content download (#21955)
openai videos models support the features to download variants.
See more details here: https://developers.openai.com/api/docs/guides/video-generation#use-image-references.
Plumb variant (e.g. "thumbnail", "spritesheet") through the full
video content download chain: avideo_content → video_content →
video_content_handler → transform_video_content_request. OpenAI
appends ?variant=<value> to the GET URL; other providers accept
the parameter in their signature but ignore it.
* fixing path
* adjust blog post path
* Revert duplicate issue checker to text-based matching, remove duplicate PR workflow
Remove the Claude Code-powered duplicate PR detection workflow and revert
the duplicate issue checker back to wow-actions/potential-duplicates with
text similarity matching.
* ui changes
* adding tests
* adjust default aggregation threshold
* fix(videos): pass api_key from litellm_params to video remix handlers (#21965)
video_remix_handler and async_video_remix_handler were not falling back
to litellm_params.api_key when the api_key parameter was None, causing
Authorization: Bearer None to be sent to the provider. This matches the
pattern already used by async_video_generation_handler.
* adding testing coverage + fixing flaky tests
* fix(ollama): thread api_base through get_model_info and add graceful fallback
When users pass api_base to litellm.completion() for Ollama, the model
info fetch (context window, function_calling support) was ignoring the
user's api_base and only reading OLLAMA_API_BASE env var or defaulting
to localhost:11434. This caused confusing errors in logs when Ollama
runs on a remote server.
Thread api_base from litellm_params through the get_model_info call
chain so OllamaConfig.get_model_info() uses the correct server. Also
return safe defaults instead of raising when the server is unreachable.
Fixes#21967
---------
Co-authored-by: An Tang <ta@stripe.com>
Co-authored-by: janfrederickk <75388864+janfrederickk@users.noreply.github.com>
Co-authored-by: Zhenting Huang <3061613175@qq.com>
Co-authored-by: Darien Kindlund <darien@kindlund.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: Ryan Crabbe <rcrabbe@berkeley.edu>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: LeeJuOh <56071126+LeeJuOh@users.noreply.github.com>
Co-authored-by: Monesh Ram <31161039+WhoisMonesh@users.noreply.github.com>
Co-authored-by: Trevor Prater <trevor.prater@gmail.com>
Co-authored-by: The Mavik <179817126+themavik@users.noreply.github.com>
Co-authored-by: Edwin Isac <33712823+edwiniac@users.noreply.github.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Harshit Jain <harshitjain0562@gmail.com>
Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com>
Co-authored-by: Ephrim Stanley <ephrim.stanley@point72.com>
Co-authored-by: TomAlon <tom@noma.security>
Co-authored-by: Julio Quinteros Pro <jquinter@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com>
Co-authored-by: Ron Zhong <ron-zhong@hotmail.com>
Co-authored-by: Arindam Majumder <109217591+Arindam200@users.noreply.github.com>
Co-authored-by: Lei Nie <lenie@quora.com>
* Update CLAUDE.md with qwen3 tool_calls bug fix instructions (#18922)
* fix(ollama): set finish_reason to "tool_calls" when tool_calls present
When qwen3 models return tool_calls through Ollama, the finish_reason
was incorrectly left as "stop" instead of being set to "tool_calls".
This caused clients to miss the tool_calls in the response.
Added _get_finish_reason helper method following OpenAI provider's
pattern, and fixed both streaming and non-streaming response paths.
Fixes: https://github.com/BerriAI/litellm/issues/18922
* fix(ollama): pass tools directly without model capability check
The previous code tried to check model capability via get_model_info()
which made network calls to localhost:11434. When Ollama is remote,
this fails and falls back to JSON format, breaking tool calling.
Ollama 0.4+ supports native tool calling - let Ollama handle
model capability detection instead of LiteLLM.
Fixes#18922
* fix(ollama): transform tool_calls response to OpenAI format
Ollama returns tool_calls with arguments as dict, but OpenAI format
requires arguments to be a JSON string. Also ensures 'type': 'function'
field is present.
Completes the fix for #18922
* fix(ollama): set finish_reason to "tool_calls" when tool_calls present
Fixes#18922
Two issues addressed:
1. Remove broken model capability check
- get_model_info() fails when Ollama runs on remote server
- Broken fallback triggered JSON prompt injection
- Now passes tools directly - Ollama 0.4+ handles detection
2. Set finish_reason correctly
- Was hardcoded to "stop" even with tool_calls present
- Clients use this to know how to process the response
- Now returns "tool_calls" when tool_calls are in response
Both streaming and non-streaming responses are fixed.
Tests:
- All 14 existing Ollama tests pass
- Added 3 focused tests for the fixes
* fix: use fastuuid helper across the codebase
First batch of changes, simple drop in replacement.
* second batch of changes
* fixed: script mistake on helper file
Updated the chunk_parser method to return a ModelResponseStream when handling 'thinking' field content, allowing UIs to render reasoning information. Adjusted tests to verify the new behavior, ensuring that reasoning content is correctly returned in the response.
Fixed MyPy type error in Ollama completion transformation where finish_reason
was set to None instead of expected string type. Changed finish_reason=None to
finish_reason="" to match GenericStreamingChunk TypedDict requirements.
Also updated corresponding test to expect empty string instead of None.
- Handle chunks containing 'thinking' field with empty 'response'
- Treat these as intermediate chunks that don't contain user content
- Add comprehensive tests for chunk parsing scenarios
- Resolves APIConnectionError for GPT-OSS model streaming
Fixes#13340
* feat: update the feature of ollama_embeddings to work on a sync api
Signed-off-by: Abinand P <abinand0911@gmail.com>
* lint:fixing of the lint file
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix:test
Signed-off-by: Abinand P <abinand0911@gmail.com>
* chore: added test for ollama embedding and refactored handler
Signed-off-by: Abinand P <abinand0911@gmail.com>
* fix:lint error
Signed-off-by: Abinand P <abinand0911@gmail.com>
---------
Signed-off-by: Abinand P <abinand0911@gmail.com>
