* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
OpenAI is winding down self-serve fine-tuning and the org can no longer
create fine-tuning jobs (403 training_not_available; the CI key surfaces
it as a 500 server_error), so test_create_fine_tune_jobs_async fails on
every batches_testing run since 2026-07-11 and reruns never clear it.
The request contract stays covered by the mocked create/list/cancel/
retrieve tests in the same file, and the deleted test's unique
standard_logging_object assertions now run inside
test_mock_openai_create_fine_tune_job.
OpenAI announced gpt-3.5-turbo-0125 (and fine-tuning of gpt-3.5-turbo
in general) for shutdown on 2026-10-23, with the announcement landing
2026-04-22. The hard-fail date is ~5 months out, but timing fits the
recent uptick in this test flaking and OpenAI may already be running
the deprecated model's pipeline with deprioritized infra.
Bump to gpt-4o-mini-2024-07-18 — currently supported for fine-tuning,
no announced shutdown. Updates the live test plus the mocked test for
consistency. Belt-and-suspenders with the existing propagation-retry
helper.
Previous fix polled `litellm.afile_retrieve` for `status == "processed"`
before calling the fine-tuning endpoint. That doesn't actually solve
the race:
- OpenAI's `FileObject.status` field is deprecated per the SDK type and
not authoritative — it can read "processed" before the file is usable.
- The retrieve and fine-tuning endpoints don't share a consistency
model, so retrieve succeeding tells you nothing about FT visibility.
Replace with a retry around the actual `acreate_fine_tuning_job` call
that catches the OpenAI 400 `'file-... does not exist'` and backs off
exponentially (1s → cap 8s, 12 attempts, ~70s total budget). The
operation succeeding is the only reliable signal that propagation
finished.
OpenAI file uploads are eventually consistent — a freshly uploaded file
may briefly 404 from `retrieve` and is rejected by the fine-tuning
endpoint with `'file-... does not exist'` until processing finishes.
The async fine-tuning test called `acreate_fine_tuning_job` immediately
after `acreate_file` and flaked on this race.
Add a polling helper that waits up to ~30s for `status=processed` (and
short-circuits on `error`), called between upload and FT job creation.
Mirrors the same propagation lag covered by the `await asyncio.sleep(1)`
in the sister batches test, but more robust against longer delays.
- Add cancel/retrieve overrides in AzureOpenAIFineTuningAPI to normalize responses
- Expand _AZURE_STATUS_MAP to handle all known Azure statuses
- Add "pending" to OpenAIFileObject.status allowed values
- Fix async test mock to return awaitable LiteLLMFineTuningJob
- Add test_openai_file_object_accepts_pending_status
Made-with: Cursor
- Move trainingType injection to AzureOpenAIFineTuningAPI handler
- Guard normalization with is_azure flag to only apply to Azure responses
- Override acreate_fine_tuning_job in Azure handler to use is_azure=True
- Update test to directly test _ensure_training_type method
- Add test for OpenAI unchanged behavior
Made-with: Cursor
- Default trainingType=1 for Azure when omitted to avoid misleading "base model does not support fine-tuning" error
- Normalize Azure FineTuningJob responses (pending→queued, null fields→defaults) to match OpenAI schema
- Add pending status support to OpenAIFileObject for Azure file uploads
- Add test coverage for trainingType default and response normalization
Made-with: Cursor
* Fix Bedrock guardrail apply_guardrail method and test mocks
Fixed 4 failing tests in the guardrail test suite:
1. BedrockGuardrail.apply_guardrail now returns original texts when guardrail
allows content but doesn't provide output/outputs fields. Previously returned
empty list, causing test_bedrock_apply_guardrail_success to fail.
2. Updated test mocks to use correct Bedrock API response format:
- Changed from 'content' field to 'output' field
- Fixed nested structure from {'text': {'text': '...'}} to {'text': '...'}
- Added missing 'output' field in filter test
3. Fixed endpoint test mocks to return GenericGuardrailAPIInputs format:
- Changed from tuple (List[str], Optional[List[str]]) to dict {'texts': [...]}
- Updated method call assertions to use 'inputs' parameter correctly
All 12 guardrail tests now pass successfully.
* fix: remove python3-dev from Dockerfile.build_from_pip to avoid Python version conflict
The base image cgr.dev/chainguard/python:latest-dev already includes Python 3.14
and its development tools. Installing python3-dev pulls Python 3.13 packages
which conflict with the existing Python 3.14 installation, causing file
ownership errors during apk install.
* fix: disable callbacks in vertex fine-tuning tests to prevent Datadog logging interference
The test was failing because Datadog logging was making an HTTP POST request
that was being caught by the mock, causing assert_called_once() to fail.
By disabling callbacks during the test, we prevent Datadog from making any
HTTP calls, allowing the mock to only see the Vertex AI API call.
* fix: ensure test isolation in test_logging_non_streaming_request
Add proper cleanup to restore original litellm.callbacks after test execution.
This prevents test interference when running as part of a larger test suite,
where global state pollution was causing async_log_success_event to be
called multiple times instead of once.
Fixes test failure where the test expected async_log_success_event to be
called once but was being called twice due to callbacks from previous tests
not being cleaned up.
* Add LiteLLM Managed file support for `retrieve`, `list` and `cancel` finetuning jobs (#11033)
* feat: initial commit adding managed file support to fine tuning endpoints
* feat(fine_tuning/endpoints.py): working call to openai finetuning route
Uses litellm managed files for finetuning api support
* feat(fine-tuning/main.py): refactor to use LiteLLMFineTuningJob pydantic object
includes 'hidden_params'
* fix: initial commit adding unified finetuning id support
return a unified finetuning id we can use to understand which deployment to route the ft request to
* test: fix test
* feat(managed_files.py): return unified finetuning job id on create finetuning job
enables retrieve, delete to work with litellm managed files
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(fine_tuning_endpoints/endpoints.py): add managed files support to list finetuning jobs
* feat(finetuning_endpoints/main): add managed files support for retrieving ft job
Makes it easier to control permissions for ft endpoint
* LiteLLM Managed Files - Enforce validation check if user can access finetuning job (#11034)
* feat: initial commit adding managed file support to fine tuning endpoints
* feat(fine_tuning/endpoints.py): working call to openai finetuning route
Uses litellm managed files for finetuning api support
* feat(fine-tuning/main.py): refactor to use LiteLLMFineTuningJob pydantic object
includes 'hidden_params'
* fix: initial commit adding unified finetuning id support
return a unified finetuning id we can use to understand which deployment to route the ft request to
* test: fix test
* feat(managed_files.py): return unified finetuning job id on create finetuning job
enables retrieve, delete to work with litellm managed files
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(managed_files.py): support managed files for cancel ft job endpoint
* feat(fine_tuning_endpoints/endpoints.py): add managed files support to list finetuning jobs
* feat(finetuning_endpoints/main): add managed files support for retrieving ft job
Makes it easier to control permissions for ft endpoint
* feat(managed_files.py): store create fine-tune / batch response object in db
storing this allows us to filter files returned on list based on what user created
* feat(managed_files.py): Ensures users can't retrieve / modify each others jobs
* fix: fix check
* fix: fix ruff check errors
* test: update to handle testing
* fix: suppress linting warning - openai 'seed' is none on azure
* test: update tests
* test: update test
* init commit ft jobs logging
* add ft logging
* add logging for FineTuningJob
* simple FT Job create test
* simplify Azure fine tuning to use all methods in OAI ft
* update doc string
* add aretrieve_fine_tuning_job
* re use from litellm.proxy.utils import handle_exception_on_proxy
* fix naming
* add /fine_tuning/jobs/{fine_tuning_job_id:path}
* remove unused imports
* update func signature
* run ci/cd again
* ci/cd run again
* fix code qulity
* ci/cd run again
* test: add new test image embedding to base llm unit tests
Addresses https://github.com/BerriAI/litellm/issues/6515
* fix(bedrock/embed/multimodal-embeddings): strip data prefix from image urls for bedrock multimodal embeddings
Fix https://github.com/BerriAI/litellm/issues/6515
* feat: initial commit for fireworks ai audio transcription support
Relevant issue: https://github.com/BerriAI/litellm/issues/7134
* test: initial fireworks ai test
* feat(fireworks_ai/): implemented fireworks ai audio transcription config
* fix(utils.py): register fireworks ai audio transcription config, in config manager
* fix(utils.py): add fireworks ai param translation to 'get_optional_params_transcription'
* refactor(fireworks_ai/): define text completion route with model name handling
moves model name handling to specific fireworks routes, as required by their api
* refactor(fireworks_ai/chat): define transform_Request - allows fixing model if accounts/ is missing
* fix: fix linting errors
* fix: fix linting errors
* fix: fix linting errors
* fix: fix linting errors
* fix(handler.py): fix linting errors
* fix(main.py): fix tgai text completion route
* refactor(together_ai/completion): refactors together ai text completion route to just use provider transform request
* refactor: move test_fine_tuning_api out of local_testing
reduces local testing ci/cd time
2024-12-25 18:35:34 -08:00
Renamed from tests/local_testing/test_fine_tuning_api.py (Browse further)
