Skip to content

Fix TypeError in /v1/completions when max_tokens is null - #3950

Open
davrens101 wants to merge 1 commit into
lm-sys:mainfrom
davrens101:fix-completion-null-max-tokens
Open

davrens101 wants to merge 1 commit into
lm-sys:mainfrom
davrens101:fix-completion-null-max-tokens

Conversation

@davrens101

Copy link
Copy Markdown

Why are these changes needed?

CompletionRequest.max_tokens is Optional[int] and OpenAI allows "max_tokens": null. In create_completion, check_length returns an int for that case, but the following line compares it with the request value:

if isinstance(max_tokens, int) and max_tokens < request.max_tokens:

so /v1/completions with "max_tokens": null fails with TypeError: '<' not supported between instances of 'int' and 'NoneType' (HTTP 500) before anything is sent to a worker.

The fix treats None as "no limit requested", so the request is capped to the remaining context length (same as /v1/chat/completions, where max_tokens=None already works).

Before (mocked worker calls): TypeError; the new unit test fails. After: test passes.

Checks

  • I've run format.sh to lint the changes in this PR. (black --check clean)
  • I've included any doc changes needed. (none needed)
  • I've made sure the relevant tests are passing (if applicable).

AI-assistance disclosure: prepared with AI assistance; I reproduced the failure and reviewed the diff myself.

Signed-off-by: davrens101 <davis@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant