[1fanwang]

Why are these changes needed?

When an OpenAI-compatible endpoint rejects an MT-bench request for exceeding its context limit, the benchmark retries the same request and can write an error placeholder as a judgment. This change makes the OpenAI and Azure completion helpers raise invalid-request errors immediately, and it keeps retrying rate limits. Only the OpenAI helper is reached from the MT-bench scripts. The cluster summarizer is the only caller of the Azure helper.

The score of 7.42 in the issue is not established by this reproduction. The issue does not say whether answer generation or judging hit the limit, and the summary leaves out judgments that failed.

Related issue number (if applicable)

Relates to #3279

Checks

• I've run format.sh to lint the changes in this PR.
• I've included any doc changes needed.
• I've made sure the relevant tests are passing (if applicable).

format.sh completed Black without edits to the changed files but exited 30 on repository-wide lint findings, including existing errors in model_adapter.py and controller.py. It reported nothing in the files this change touches.

Testing done

The reproducer uses the real OpenAI 0.28.1 SDK against a local HTTP server that returns a context-limit error. It uses two retries and no delay to keep the run short. The captured runs used Python 3.12.14 on macOS, and the warnings printed at import time are omitted below. Save the source below as repro_context_limit.py outside the checkout, then set up the environment:

python -m pip install -e '.[dev,model_worker,llm_judge]'
export REPRO=/absolute/path/to/repro_context_limit.py

Run the same two commands from a checkout of each revision. The first drives the MT-bench judge through the OpenAI helper. The second calls the Azure helper directly, because no MT-bench script reaches it.

PYTHONPATH=. python "$REPRO"
AZURE=1 PYTHONPATH=. python "$REPRO"

Before, at 587d5cf, the judge retries and stores an error placeholder, and the Azure helper retries and returns one:

<class 'openai.error.InvalidRequestError'> This model's maximum context length is 8192 tokens.
<class 'openai.error.InvalidRequestError'> This model's maximum context length is 8192 tokens.
question: 1, turn: 1, model: candidate, score: -1, judge: ('gpt-4', 'test')
result: score=-1, judgment=$ERROR$
HTTP requests: 2
judgment file exists: True
<class 'openai.error.InvalidRequestError'> This model's maximum context length is 8192 tokens.
<class 'openai.error.InvalidRequestError'> This model's maximum context length is 8192 tokens.
result: $ERROR$
HTTP requests: 2

After, at f8ad8e6, both raise on the first request and the judgment file is never written:

raised: InvalidRequestError: This model's maximum context length is 8192 tokens.
HTTP requests: 1
judgment file exists: False
raised: InvalidRequestError: This model's maximum context length is 8192 tokens.
HTTP requests: 1

The Python package workflow, including the new test step, passed at this head on my fork, on Ubuntu 26.04 with Python 3.10.22: https://github.com/1fanwang/FastChat/actions/runs/43093389164.

Reproducer source: repro_context_limit.py

import json
import os
import tempfile
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

import openai

from fastchat.conversation import get_conv_template
from fastchat.llm_judge import common


class Handler(BaseHTTPRequestHandler):
    requests = 0

    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        Handler.requests += 1
        body = json.dumps(
            {
                "error": {
                    "message": "This model's maximum context length is 8192 tokens.",
                    "type": "invalid_request_error",
                    "param": "messages",
                    "code": "context_length_exceeded",
                }
            }
        ).encode()
        self.send_response(400)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass


server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
thread = threading.Thread(target=server.serve_forever)
thread.start()
openai.api_base = f"http://127.0.0.1:{server.server_port}/v1"
openai.api_key = "local-test-key"
openai.api_type = "open_ai"
common.API_MAX_RETRY = 2
common.API_RETRY_SLEEP = 0
match = common.MatchSingle(
    question={"question_id": 1, "turns": ["Question"]},
    model="candidate",
    answer={"choices": [{"turns": ["Answer"]}]},
    judge=common.Judge(
        model_name="gpt-4",
        prompt_template={
            "type": "single",
            "name": "test",
            "system_prompt": "Judge the answer.",
            "prompt_template": "{question} {answer}",
            "output_format": "[[rating]]",
        },
    ),
)
try:
    if os.environ.get("AZURE"):
        conv = get_conv_template("chatgpt")
        conv.append_message(conv.roles[0], "Question")
        conv.append_message(conv.roles[1], None)
        try:
            output = common.chat_completion_openai_azure(
                "azure-gpt-4",
                conv,
                0,
                20,
                api_dict={"api_base": openai.api_base, "api_key": openai.api_key},
            )
            print(f"result: {output}")
        except openai.error.InvalidRequestError as error:
            print(f"raised: {type(error).__name__}: {error}")
        print(f"HTTP requests: {Handler.requests}")
    else:
        with tempfile.TemporaryDirectory() as directory:
            output = Path(directory) / "judgment.jsonl"
            try:
                result = common.play_a_match_single(match, str(output))
                print(f"result: score={result['score']}, judgment={result['judgment']}")
            except openai.error.InvalidRequestError as error:
                print(f"raised: {type(error).__name__}: {error}")
            print(f"HTTP requests: {Handler.requests}")
            print(f"judgment file exists: {output.exists()}")
finally:
    server.shutdown()
    thread.join()
    server.server_close()