Successfully reproduced the issue with actual running proxy server!
Key findings:
- ERROR logs with exceptions ARE properly formatted as JSON ✅
- Startup INFO logs are NOT formatted as JSON ❌
- ANSI color codes appear in logs ([94m, [32m, etc.) ❌
- Mix of JSON and plain text logs confirmed
Issue reproduced:
1. Started litellm proxy with json_logs: true
2. Made requests to trigger various errors
3. Captured actual logs showing the problem
Problem logs (plain text with ANSI codes):
- "Using json logs. Setting log_config to None."
- "[94m Initialized Success Callbacks - [] [0m"
- "[32mLiteLLM: Proxy initialized with Config, Set models:[0m"
Working logs (proper JSON):
- {"message": "litellm.proxy.proxy_server._handle_llm_api_exception()...", "level": "ERROR", ...}
- {"message": "Failed to load vertex credentials...", "level": "ERROR", "stacktrace": "..."}
Root cause:
- Some code uses print() statements instead of loggers
- Colored formatters bypass JsonFormatter
- ANSI escape codes in formatter not removed when json_logs enabled
Files added:
- reproduce_json_logs_config.yaml: Proxy config with json_logs enabled
- start_proxy.py: Script to start proxy programmatically
- reproduce_json_logs.sh: Automated test that runs proxy and analyzes logs
- actual_proxy_logs.txt: Real captured logs showing the issue
- REPRODUCER_RESULTS.md: Comprehensive analysis and findings
This exactly reproduces dmc's reported issue from the Slack thread.
Investigation findings:
- JSON logging WORKS correctly in single-process mode
- All test scripts show proper JSON formatting with stacktraces
- Issue likely caused by multi-worker setup where _turn_on_json()
is only called in main process, not worker processes
Test scripts added:
- test_json_logs_reproducer.py: Basic JSON logging with env var
- test_json_logs_late_enable.py: Tests late JSON logs enablement
- test_logging_handlers.py: Analyzes logger handler hierarchy
- test_proxy_json_logs_reproducer.py: Simulates exact proxy startup
- test_json_logs_config.yaml: Minimal proxy config for testing
Documentation:
- JSON_LOGS_REPRODUCTION.md: Comprehensive investigation findings,
reproduction steps, and recommended solutions
Root cause: When using multiple workers (gunicorn/uvicorn), each
worker process has its own logging configuration. The _turn_on_json()
call in main process doesn't propagate to workers.
Recommended fix: Ensure _turn_on_json() is called in each worker
process during initialization, or set JSON_LOGS env var before import.
* feat: add support for keda in helm chart
Signed-off-by: R.Sicart <roger.sicart@gmail.com>
* chore: bump chart version
---------
Signed-off-by: R.Sicart <roger.sicart@gmail.com>
- Add health_check_client.py for monitoring model availability
- Add health_check_client_README.md with usage documentation
- Add health_check_requirements.txt for dependencies
- Add run_parallel_health_checks.ps1 (PowerShell version)
- Add run_parallel_health_checks.sh (Bash version)
- Organize all scripts under scripts/health_check/ directory
* docs: update UI contributing guide with correct commands
- Replace outdated proxy_cli.py command with poetry run litellm
- Add config.yaml example with required settings
- Clarify that UI comes pre-built in the repo
- Add two development options: Build Mode and Dev Mode (hot reload)
- Note about redirect issues in Dev Mode
* docs: add hot reload login flow and PR submission section
- Document the 3000 -> 4000 -> 3000 login flow for hot reload
- Reorder: Hot Reload as Option A, Build Mode as Option B
- Add section 4 on submitting PRs
- Add note that UI changes don't require tests
* Update login flow navigation URL in contributing.md
Replace copy.deepcopy with model_dump + model_validate in streaming
iterator logging to handle Pydantic ValidatorIterator objects that
cannot be pickled when tool_choice uses allowed_tools mode.
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
