diff --git a/docs/my-website/docs/tutorials/lm_evaluation_harness.md b/docs/my-website/docs/tutorials/lm_evaluation_harness.md index 3cb63d0087f..a4488076e4b 100644 --- a/docs/my-website/docs/tutorials/lm_evaluation_harness.md +++ b/docs/my-website/docs/tutorials/lm_evaluation_harness.md @@ -41,6 +41,31 @@ python3 -m lm_eval \ ``` +## FLASK - Fine-grained Language Model Evaluation Based on Alignment Skill Sets +Use litellm to evaluate any LLM on FLASK https://github.com/kaistAI/FLASK + +**Step 1: Start the local proxy** +```shell +$ litellm --model huggingface/bigcode/starcoder +``` + +**Step 2: Set OpenAI API Base & Key** +```shell +$ export OPENAI_API_BASE=http://0.0.0.0:8000 +``` + +**Step 3 Run with FLASK** + +```shell +cd FLASK +cd gpt_review +``` + +Run the eval +``shell +python gpt4_eval.py -q '../evaluation_set/flask_evaluation.jsonl' +``` + ## Debugging ### Making a test request to your proxy