scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-03 02:22:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 140

feat(search): add you_com as a search provider (#28370)

Browse source 
* feat(search): add you_com as a search provider

Registers You.com Search API as a first-class `search_provider` in the
`search_tools` registry, alongside Tavily, Exa, Perplexity, etc.

- New adapter: litellm/llms/you_com/search/transformation.py
  - POSTs to https://ydc-index.io/v1/search
  - Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key)
  - Maps Perplexity unified spec: max_results -> count,
    search_domain_filter -> include_domains, country -> country
  - Flattens results.web + results.news into a single SearchResult list;
    snippet prefers snippets[0], falls back to description; page_age -> date
- Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired
  into ProviderConfigManager.get_provider_search_config()
- Pricing entry: model_prices_and_context_window.json (placeholder $0.0;
  happy to adjust to maintainers' preferred public number)
- Docs: example router config snippet and example proxy yaml updated
- Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests
  (payload shape, domain filter mapping, snippet fallback, news flattening,
  missing-api-key error)

Refs upstream expansion signal: #15942

* review fixups: normalize api_base, lowercase country, scope env-var to test

Addresses Greptile inline review comments on #28370:

- get_complete_url: strip trailing slashes from api_base *before* the
  endswith("/v1/search") check, so a custom base like ".../v1/search/"
  doesn't become ".../v1/search/v1/search".
- transform_search_request: .lower() country before sending, matching
  Tavily's convention so callers using the unified spec form ("US") get
  consistent behavior across providers.
- Tests: replace direct os.environ writes with an autouse monkeypatch
  fixture so YOUCOM_API_KEY is set per-test and removed afterwards.
  The missing-key test now uses monkeypatch.delenv. New test asserts the
  trailing-slash normalization above.

Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note
that documentation changes belong in the litellm-docs repo.

* support keyless free tier (api.you.com/v1/agents/search) as default

You.com offers an IP-throttled keyless endpoint that returns the same
response shape as the keyed one (~100 queries/day, no signup). This is a
significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG
providers already in the search_tools registry.

Behavior:
- YOUCOM_API_KEY set        -> keyed:  POST https://ydc-index.io/v1/search
                                       (X-API-Key header)
- no key                    -> free:   POST https://api.you.com/v1/agents/search
                                       (no auth)
- YOUCOM_API_BASE override  -> honored as-is

Tests:
- New: test_you_com_search_keyless_free_tier - asserts URL + absence of
  X-API-Key when no key is configured.
- New: test_you_com_search_validate_environment_keyless - asserts the
  config no longer raises when the key is absent.
- Removed: test_you_com_search_raises_without_api_key (the precondition
  no longer holds).
- Existing payload/domain-filter/etc tests still cover keyed mode via
  the autouse YOUCOM_API_KEY fixture.

Verified both endpoints accept POST + return identical JSON shape:
  results.web[] / results.news[] with title, url, snippets, description,
  page_age.

* register you_com in provider_endpoints_support.json

Adding `litellm/llms/you_com/` requires a corresponding entry in
provider_endpoints_support.json or the
code-quality/check_provider_folders_documented CI check fails.

Follows the compact tavily/serper pattern - endpoints: { search: true }.
Local run of the check now reports "All 114 provider folders are documented".

* move tests under tests/test_litellm/llms/ so CI exercises them

The litellm CI workflows scope unit tests to `tests/test_litellm/...`
(see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so
tests living under `tests/search_tests/` are never run in CI - which is
why codecov reports 0% patch coverage for the new adapter even though
the unit tests exist and pass locally.

Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so
the test-unit-llm-providers job picks it up. 7/7 tests still pass at
the new location.

(Sibling search-only providers - tavily, exa_ai, brave, etc. - still
live only in `tests/search_tests/` and would benefit from the same
move, but that is out of scope for this PR.)

* fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug

The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises
Content-Encoding: gzip but returns a body that httpx's decoder rejects
with `zlib.error: Error -3 while decompressing data: incorrect header
check`, surfacing as litellm.APIConnectionError in user code. curl works
because it doesn't request compression by default.

Pin Accept-Encoding: identity in validate_environment so the upstream
server skips compression entirely. Harmless on the keyed endpoint
(ydc-index.io/v1/search) which negotiates content-encoding correctly.

The header uses setdefault so a caller-supplied Accept-Encoding still
takes precedence. (Server-side bug has been flagged to the You.com team
separately - once fixed there, this workaround can be removed.)

New unit test: test_you_com_search_pins_identity_accept_encoding.

---------

Co-authored-by: Sameer Kankute <sameer@berri.ai>
This commit is contained in:
Brian Sparker 2026-06-05 03:57:49 -07:00 • committed by GitHub
parent dacce6e65f
commit eab34a3995
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 549 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
litellm/integrations/websearch_interception/ARCHITECTURE.md
View file

@ -244,6 +244,9 @@ search_tools:
- search_tool_name: "my-tavily-tool"
litellm_params:
search_provider: "tavily"
- search_tool_name: "my-you-com-tool"
litellm_params:
search_provider: "you_com"
```
---

0
litellm/llms/you_com/__init__.py Normal file
View file

7
litellm/llms/you_com/search/__init__.py Normal file
View file

@ -0,0 +1,7 @@
"""
You.com Search API module.
"""
from litellm.llms.you_com.search.transformation import YouComSearchConfig
__all__ = ["YouComSearchConfig"]

193
litellm/llms/you_com/search/transformation.py Normal file
View file

@ -0,0 +1,193 @@
"""
Calls You.com's /v1/search endpoint to search the web.
You.com API Reference: https://you.com/docs/api-reference/search/v1-search
OpenAPI spec: https://you.com/specs/openapi_search_v1.yaml
"""
from typing import Dict, List, Optional, TypedDict, Union
import httpx
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.base_llm.search.transformation import (
BaseSearchConfig,
SearchResponse,
SearchResult,
)
from litellm.secret_managers.main import get_secret_str
class _YouComSearchRequestRequired(TypedDict):
"""Required fields for You.com Search API request."""
query: str
class YouComSearchRequest(_YouComSearchRequestRequired, total=False):
"""
You.com Search API request format.
Based on: https://you.com/specs/openapi_search_v1.yaml
"""
count: int
country: str
language: str
freshness: str
include_domains: List[str]
exclude_domains: List[str]
safesearch: str
class YouComSearchConfig(BaseSearchConfig):
# Keyed tier (higher rate limits): authenticate with X-API-Key.
YOU_COM_API_BASE = "https://ydc-index.io"
# Keyless free tier: IP-throttled (100 queries/day) and requires no auth.
# Used automatically when YOUCOM_API_KEY is not set.
YOU_COM_FREE_API_BASE = "https://api.you.com/v1/agents/search"
@staticmethod
def ui_friendly_name() -> str:
return "You.com"
def validate_environment(
self,
headers: Dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
**kwargs,
) -> Dict:
"""
Set headers for the You.com Search API.
If YOUCOM_API_KEY (or an explicit api_key) is present, use the keyed
endpoint with the `X-API-Key` header. Otherwise fall through to the
keyless free tier; no auth header is required.
"""
api_key = api_key or get_secret_str("YOUCOM_API_KEY")
headers["Content-Type"] = "application/json"
# Pin Accept-Encoding to identity: the keyless `api.you.com/v1/agents/search`
# endpoint advertises gzip content-encoding but returns body bytes the
# decoder rejects, which surfaces as httpx.DecodingError through litellm's
# http handler. Identity is harmless on the keyed endpoint.
headers.setdefault("Accept-Encoding", "identity")
if api_key:
headers["X-API-Key"] = api_key
return headers
def get_complete_url(
self,
api_base: Optional[str],
optional_params: dict,
data: Optional[Union[Dict, List[Dict]]] = None,
**kwargs,
) -> str:
"""
Pick the endpoint based on whether an API key is configured.
- api_base explicit override -> use it as-is (normalized)
- YOUCOM_API_KEY set -> keyed endpoint (ydc-index.io/v1/search)
- no key -> keyless free tier (api.you.com/v1/agents/search)
"""
if api_base is None:
api_base = get_secret_str("YOUCOM_API_BASE")
if api_base is None:
api_key = get_secret_str("YOUCOM_API_KEY")
if api_key:
api_base = self.YOU_COM_API_BASE
else:
# Keyless free tier already includes the full path.
return self.YOU_COM_FREE_API_BASE
api_base = api_base.rstrip("/")
if not api_base.endswith("/v1/search") and not api_base.endswith(
"/v1/agents/search"
):
api_base = f"{api_base}/v1/search"
return api_base
def transform_search_request(
self,
query: Union[str, List[str]],
optional_params: dict,
**kwargs,
) -> Dict:
"""
Transform Search request to You.com API format.
Perplexity unified spec → You.com mappings:
- query → query
- max_results → count
- search_domain_filter → include_domains
- country → country
- max_tokens_per_page → (not applicable, ignored)
"""
if isinstance(query, list):
query = " ".join(query)
request_data: YouComSearchRequest = {
"query": query,
}
if "max_results" in optional_params:
request_data["count"] = optional_params["max_results"]
if "search_domain_filter" in optional_params:
request_data["include_domains"] = optional_params["search_domain_filter"]
if "country" in optional_params:
request_data["country"] = optional_params["country"].lower()
result_data = dict(request_data)
for param, value in optional_params.items():
if (
param not in self.get_supported_perplexity_optional_params()
and param not in result_data
):
result_data[param] = value
return result_data
def transform_search_response(
self,
raw_response: httpx.Response,
logging_obj: LiteLLMLoggingObj,
**kwargs,
) -> SearchResponse:
"""
Transform You.com API response to LiteLLM unified SearchResponse format.
You.com → LiteLLM mappings (for both `results.web[]` and `results.news[]`):
- title → SearchResult.title
- url → SearchResult.url
- snippets[0] → SearchResult.snippet (falls back to `description`)
- page_age → SearchResult.date
"""
response_json = raw_response.json()
raw_results = response_json.get("results") or {}
web_results = raw_results.get("web") or []
news_results = raw_results.get("news") or []
results: List[SearchResult] = []
for item in list(web_results) + list(news_results):
snippets = item.get("snippets") or []
snippet = snippets[0] if snippets else item.get("description", "")
results.append(
SearchResult(
title=item.get("title", ""),
url=item.get("url", ""),
snippet=snippet,
date=item.get("page_age"),
last_updated=None,
)
)
return SearchResponse(
results=results,
object="search",
)

4
litellm/proxy/example_config_yaml/websearch_interception_config.yaml
View file

@ -8,6 +8,10 @@ search_tools:
- search_tool_name: "my-perplexity-search"
litellm_params:
search_provider: "perplexity"
# Alternative provider example (requires YOUCOM_API_KEY):
# - search_tool_name: "my-you-com-search"
# litellm_params:
# search_provider: "you_com"
litellm_settings:
callbacks: ["websearch_interception"]

1
litellm/types/utils.py
View file

@ -3417,6 +3417,7 @@ class SearchProviders(str, Enum):
DUCKDUCKGO = "duckduckgo"
SEARCHAPI = "searchapi"
SERPER = "serper"
YOU_COM = "you_com"
APISERPENT = "apiserpent"

2
litellm/utils.py
View file

@ -9524,6 +9524,7 @@ class ProviderConfigManager:
from litellm.llms.searxng.search.transformation import SearXNGSearchConfig
from litellm.llms.serper.search.transformation import SerperSearchConfig
from litellm.llms.tavily.search.transformation import TavilySearchConfig
from litellm.llms.you_com.search.transformation import YouComSearchConfig
PROVIDER_TO_CONFIG_MAP = {
SearchProviders.PERPLEXITY: PerplexitySearchConfig,
@ -9539,6 +9540,7 @@ class ProviderConfigManager:
SearchProviders.DUCKDUCKGO: DuckDuckGoSearchConfig,
SearchProviders.SEARCHAPI: SearchAPIConfig,
SearchProviders.SERPER: SerperSearchConfig,
SearchProviders.YOU_COM: YouComSearchConfig,
SearchProviders.APISERPENT: APISerpentSearchConfig,
}
config_class = PROVIDER_TO_CONFIG_MAP.get(provider, None)

5
model_prices_and_context_window.json
View file

@ -31023,6 +31023,11 @@
"litellm_provider": "tavily",
"mode": "search"
},
"you_com/search": {
"input_cost_per_query": 0.0,
"litellm_provider": "you_com",
"mode": "search"
},
"text-completion-codestral/codestral-2405": {
"input_cost_per_token": 0.0,
"litellm_provider": "text-completion-codestral",

4
provider_endpoints_support.json
View file

@ -2206,6 +2206,10 @@
"search": true
}
},
"you_com": {
"display_name": "You.com (`you_com`)",
"url": "https://docs.litellm.ai/docs/search/you_com"
},
"apiserpent": {
"display_name": "APISerpent (`apiserpent`)",
"url": "https://docs.litellm.ai/docs/search/apiserpent",

0
tests/test_litellm/llms/you_com/__init__.py Normal file
View file

330
tests/test_litellm/llms/you_com/test_you_com_search.py Normal file
View file

@ -0,0 +1,330 @@
"""
Tests for You.com Search API integration.
"""
import os
import sys
import pytest
from unittest.mock import AsyncMock, patch, MagicMock
sys.path.insert(0, os.path.abspath("../.."))
import litellm
class TestYouComSearch:
"""
Tests for You.com Search functionality with mocked network responses.
"""
@pytest.fixture(autouse=True)
def _set_api_key(self, monkeypatch):
"""
Default fixture: YOUCOM_API_KEY is set, scoped to this test.
Tests that need the key absent should call `monkeypatch.delenv` themselves.
"""
monkeypatch.setenv("YOUCOM_API_KEY", "test-api-key")
@pytest.mark.asyncio
async def test_you_com_search_request_payload(self):
"""
Validate the You.com search request payload structure without real API calls.
"""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"results": {
"web": [
{
"title": "Test Result 1",
"url": "https://example.com/1",
"description": "Brief description 1",
"snippets": ["This is a test snippet for result 1"],
"page_age": "2025-01-15T00:00:00Z",
},
{
"title": "Test Result 2",
"url": "https://example.com/2",
"description": "Brief description 2",
"snippets": ["This is a test snippet for result 2"],
"page_age": "2025-01-10T00:00:00Z",
},
],
"news": [],
},
"metadata": {
"search_uuid": "abc-123",
"query": "latest developments in AI",
"latency": 0.42,
},
}
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
mock_post.return_value = mock_response
response = await litellm.asearch(
query="latest developments in AI",
search_provider="you_com",
max_results=5,
)
assert mock_post.call_count == 1
call_args = mock_post.call_args
assert call_args.kwargs["url"] == "https://ydc-index.io/v1/search"
headers = call_args.kwargs.get("headers", {})
assert "X-API-Key" in headers
assert headers["X-API-Key"] == "test-api-key"
assert headers["Content-Type"] == "application/json"
json_data = call_args.kwargs.get("json")
assert json_data is not None
assert json_data["query"] == "latest developments in AI"
# max_results is mapped to You.com's `count` parameter
assert json_data["count"] == 5
assert hasattr(response, "results")
assert hasattr(response, "object")
assert response.object == "search"
assert len(response.results) == 2
first_result = response.results[0]
assert first_result.title == "Test Result 1"
assert first_result.url == "https://example.com/1"
assert first_result.snippet == "This is a test snippet for result 1"
assert first_result.date == "2025-01-15T00:00:00Z"
@pytest.mark.asyncio
async def test_you_com_search_domain_filter_and_country(self):
"""
Validate that Perplexity-spec optional params map to You.com's parameters:
- search_domain_filter -> include_domains
- country -> country (lowercased to match Tavily's convention)
"""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"results": {"web": [], "news": []},
"metadata": {},
}
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
mock_post.return_value = mock_response
await litellm.asearch(
query="machine learning",
search_provider="you_com",
search_domain_filter=["arxiv.org", "nature.com"],
country="US",
)
call_args = mock_post.call_args
json_data = call_args.kwargs.get("json")
assert json_data["query"] == "machine learning"
assert json_data["include_domains"] == ["arxiv.org", "nature.com"]
# Country is normalized to lowercase, matching Tavily's behavior.
assert json_data["country"] == "us"
# search_domain_filter and max_tokens_per_page (perplexity-spec names)
# should NOT leak through to the upstream payload.
assert "search_domain_filter" not in json_data
assert "max_tokens_per_page" not in json_data
@pytest.mark.asyncio
async def test_you_com_search_snippet_fallback_to_description(self):
"""
When `snippets` is missing/empty, snippet falls back to `description`.
"""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"results": {
"web": [
{
"title": "No snippets here",
"url": "https://example.com/3",
"description": "Fallback description text",
"snippets": [],
"page_age": None,
}
],
"news": [],
},
"metadata": {},
}
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
mock_post.return_value = mock_response
response = await litellm.asearch(
query="anything",
search_provider="you_com",
)
assert len(response.results) == 1
assert response.results[0].snippet == "Fallback description text"
assert response.results[0].date is None
@pytest.mark.asyncio
async def test_you_com_search_news_results_appended(self):
"""
News results are flattened in after web results.
"""
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"results": {
"web": [
{
"title": "Web Result",
"url": "https://example.com/web",
"snippets": ["web snippet"],
"description": "web desc",
"page_age": "2025-01-01T00:00:00Z",
}
],
"news": [
{
"title": "News Result",
"url": "https://news.example.com/article",
"description": "news desc",
"page_age": "2025-02-01T00:00:00Z",
}
],
},
"metadata": {},
}
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
mock_post.return_value = mock_response
response = await litellm.asearch(
query="anything",
search_provider="you_com",
)
assert len(response.results) == 2
assert response.results[0].title == "Web Result"
assert response.results[1].title == "News Result"
# News result has no `snippets` -> falls back to description
assert response.results[1].snippet == "news desc"
def test_you_com_search_complete_url_handles_trailing_slash(self):
"""
get_complete_url must normalize trailing slashes on api_base, so a custom
base like `https://x.example/v1/search/` does not become
`https://x.example/v1/search/v1/search`.
"""
from litellm.llms.you_com.search.transformation import YouComSearchConfig
config = YouComSearchConfig()
assert (
config.get_complete_url(
api_base="https://x.example/v1/search/", optional_params={}
)
== "https://x.example/v1/search"
)
assert (
config.get_complete_url(api_base="https://x.example/", optional_params={})
== "https://x.example/v1/search"
)
# With an API key configured, default base is the keyed endpoint.
assert (
config.get_complete_url(api_base=None, optional_params={})
== "https://ydc-index.io/v1/search"
)
@pytest.mark.asyncio
async def test_you_com_search_keyless_free_tier(self, monkeypatch):
"""
Without YOUCOM_API_KEY, the adapter targets the keyless free-tier
endpoint and sends no X-API-Key header.
"""
monkeypatch.delenv("YOUCOM_API_KEY", raising=False)
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = {
"results": {
"web": [
{
"title": "Keyless Result",
"url": "https://example.com/keyless",
"snippets": ["snippet from keyless tier"],
"description": "desc",
"page_age": "2025-03-01T00:00:00Z",
}
],
"news": [],
},
"metadata": {},
}
with patch(
"litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler.post",
new_callable=AsyncMock,
) as mock_post:
mock_post.return_value = mock_response
response = await litellm.asearch(
query="hello world",
search_provider="you_com",
)
call_args = mock_post.call_args
assert (
call_args.kwargs["url"] == "https://api.you.com/v1/agents/search"
)
headers = call_args.kwargs.get("headers", {})
assert "X-API-Key" not in headers
assert headers["Content-Type"] == "application/json"
assert len(response.results) == 1
assert response.results[0].title == "Keyless Result"
def test_you_com_search_validate_environment_keyless(self, monkeypatch):
"""
validate_environment must NOT raise when no key is configured —
the keyless free tier is the default behavior.
"""
monkeypatch.delenv("YOUCOM_API_KEY", raising=False)
from litellm.llms.you_com.search.transformation import YouComSearchConfig
config = YouComSearchConfig()
headers = config.validate_environment(headers={}, api_key=None)
assert "X-API-Key" not in headers
assert headers["Content-Type"] == "application/json"
def test_you_com_search_pins_identity_accept_encoding(self, monkeypatch):
"""
The adapter pins Accept-Encoding: identity to work around the keyless
endpoint advertising gzip content-encoding while returning bytes httpx
can't decode. Without this, every keyless request raises DecodingError.
"""
monkeypatch.delenv("YOUCOM_API_KEY", raising=False)
from litellm.llms.you_com.search.transformation import YouComSearchConfig
config = YouComSearchConfig()
headers = config.validate_environment(headers={}, api_key=None)
assert headers["Accept-Encoding"] == "identity"
# setdefault: a caller-supplied Accept-Encoding should win
headers = config.validate_environment(
headers={"Accept-Encoding": "gzip"}, api_key=None
)
assert headers["Accept-Encoding"] == "gzip"