This prevents upstream GitHub changes from breaking LLM calls by:
1. Runtime validation in get_model_cost_map.py:
- Validates JSON structure before accepting remote data
- Checks for required fields (litellm_provider)
- Validates mode values against allowed list
- Validates numeric cost fields
- Falls back to local backup if validation fails
- Logs warnings when falling back
2. Enhanced CI validation (.github/workflows/test-model-map.yaml):
- JSON syntax validation (fast fail)
- Comprehensive schema validation via Python script
- Validates both main file and backup file
- Checks backup freshness
3. New validation script (scripts/validate_model_cost_map.py):
- Validates all entries have required fields
- Checks mode values against allowed modes
- Validates numeric fields are actually numeric
- Sanity checks for suspiciously high costs
- Detailed error reporting
4. Unit tests for validation logic
The key improvement: if GitHub returns valid JSON but semantically
broken data (missing fields, wrong types), we now detect this and
fall back to the local backup instead of using the bad data.
Customers can also set LITELLM_LOCAL_MODEL_COST_MAP=True to completely
disable remote fetching for maximum stability.
Co-authored-by: ishaan <ishaan@berri.ai>
This document identifies all external dependencies that could impact LLM
calling logic at runtime and provides mitigation strategies.
Key findings:
- Only ONE critical external dependency: GitHub model cost map
- This is mitigable with LITELLM_LOCAL_MODEL_COST_MAP=True
- Other external calls (HuggingFace, Together AI, etc.) fail gracefully
- No dynamic code loading from external sources
Co-authored-by: ishaan <ishaan@berri.ai>
* Add async_post_call_response_headers_hook to CustomLogger (#20070)
Allow CustomLogger callbacks to inject custom HTTP response headers
into streaming, non-streaming, and failure responses via a new
async_post_call_response_headers_hook method.
* async_post_call_response_headers_hook
---------
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* Add /realtime API benchmarks to Benchmarks documentation
- Added new section showing performance improvements for /realtime endpoint
- Included before/after metrics showing 182× faster p99 latency
- Added test setup specifications and key optimizations
- Referenced from v1.80.5-stable release notes
Co-authored-by: ishaan <ishaan@berri.ai>
* Update /realtime benchmarks to show current performance only
- Removed before/after comparison, showing only current metrics
- Clarified that benchmarks are e2e latency against fake realtime endpoint
- Simplified table format for better readability
Co-authored-by: ishaan <ishaan@berri.ai>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
* Add LangSmith mock client support
- Create langsmith_mock_client.py following GCS and Langfuse patterns
- Add mock mode detection via LANGSMITH_MOCK environment variable
- Intercept LangSmith API calls via AsyncHTTPHandler.post patching
- Add verbose logging throughout mock implementation
- Update LangsmithLogger to initialize mock client when mock mode enabled
- Supports configurable mock latency via LANGSMITH_MOCK_LATENCY_MS
* Add Datadog mock client support
- Create datadog_mock_client.py following GCS, Langfuse, and LangSmith patterns
- Add mock mode detection via DATADOG_MOCK environment variable
- Intercept Datadog API calls via AsyncHTTPHandler.post and httpx.Client.post patching
- Add verbose logging throughout mock implementation
- Update DataDogLogger and DataDogLLMObsLogger to initialize mock client when mock mode enabled
- Supports both async and sync logging paths
- Supports configurable mock latency via DATADOG_MOCK_LATENCY_MS
* refactor: consolidate mock client logic into factory pattern
- Create mock_client_factory.py to centralize common mock HTTP client logic
- Refactor GCS, Langfuse, LangSmith, and Datadog mock clients to use factory
- Improve GET/DELETE mock accuracy for GCS (return valid StandardLoggingPayload)
- Fix DELETE mock to return empty body (204 No Content) instead of JSON
- Reduce code duplication across integration mock clients
* feat: add PostHog mock client support
- Create posthog_mock_client.py using factory pattern
- Integrate mock client into PostHogLogger with mock mode detection
- Add verbose logging for mock mode initialization and batch operations
- Enable mock mode via POSTHOG_MOCK environment variable
* Add Helicone mock client support
- Created helicone_mock_client.py using factory pattern (similar to GCS)
- Integrated mock mode detection and initialization in HeliconeLogger
- Mock client patches HTTPHandler.post to intercept Helicone API calls
- Uses factory pattern for should_use_mock and MockResponse utilities
- Custom HTTPHandler.post patching required since HTTPHandler uses self.client.send()
* Add mock support for Braintrust integration and extend mock client factory
- Add braintrust_mock_client.py with mock HTTP client for Braintrust integration testing
- Integrate mock client into BraintrustLogger with mock mode detection
- Refactor Helicone mock client to fully utilize factory's HTTPHandler.post patching
- Extend mock_client_factory to support patching HTTPHandler.post for sync calls
- Enable endpoint-specific mock responses for Braintrust (/project vs /project_logs)
- All mock clients now properly handle both async (AsyncHTTPHandler) and sync (HTTPHandler) calls
* Fix linter errors: remove unused imports and suppress complexity warning
- Remove unused imports from gcs_bucket_mock_client.py (httpx, json, timedelta, Dict, Optional)
- Remove unused Callable import from mock_client_factory.py
- Add noqa comment to suppress PLR0915 complexity warning for create_mock_client_factory function
* Document mock environment variables for PostHog, Helicone, Braintrust, Datadog, and Langsmith integrations
- Add POSTHOG_MOCK and POSTHOG_MOCK_LATENCY_MS documentation
- Add HELICONE_MOCK and HELICONE_MOCK_LATENCY_MS documentation
- Add BRAINTRUST_MOCK and BRAINTRUST_MOCK_LATENCY_MS documentation
- Add DATADOG_MOCK and DATADOG_MOCK_LATENCY_MS documentation
- Add LANGSMITH_MOCK and LANGSMITH_MOCK_LATENCY_MS documentation
All mock env vars follow the same pattern: enable mock mode for integration testing by intercepting API calls and returning mock responses without making actual network calls.
* Fix security issue
Fixes permission error where prisma generate fails with 'Permission denied'
when trying to write schema.prisma in non-root containers.
The issue was that prisma generate was running as root before switching
to nobody user, causing generated files to be owned by root:root.
Moving prisma generate after USER nobody ensures files are owned by
nobody:nobody and can be written to during runtime.
Fixes#19859
* fix: bedrock invoke - does not support prompt-caching-scope
* fix: UNSUPPORTED_BEDROCK_INVOKE_BETA_PATTERNS
* init requirements.txt
* init README for claude Agent SDK
* fix: using converse models with UNSUPPORTED_BEDROCK_CONVERSE_BETA_PATTERNS
* fix main.py
* init: proxy_e2e_anthropic_messages_tests
