Is there some way to get all the log probs of multiple possible next tokens with the library? So far I managed to get only the probabilities for the chosen token with this example
curl -X POST https://api.together.xyz/v1/chat/completions \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $TOGETHER_API_KEY"\
-d '{
"model": "codellama/CodeLlama-70b-Instruct-hf",
"stream": false,
"max_tokens": 10,
"messages": [
{"role": "user", "content": "write an async function in python"}
],
"logprobs": 1
}'
{
"id": "85d3d3ff1a0d8c57-EWR",
"object": "chat.completion",
"created": 1709240335,
"model": "codellama/CodeLlama-70b-Instruct-hf",
"prompt": [],
"choices": [
{
"finish_reason": "length",
"logprobs": {
"tokens": [
"1",
".",
" Define",
" the",
" function",
" with",
" the",
" async",
" keyword",
"."
],
"token_logprobs": [
-1.0029297,
-0.07861328,
-2.1367188,
-0.921875,
-0.6479492,
-1.1318359,
-0.35668945,
-0.6743164,
-0.023162842,
-1.1484375
]
},
"index": 0,
"message": {
"role": "assistant",
"content": "1. Define the function with the async keyword."
}
}
],
"usage": {
"prompt_tokens": 33,
"completion_tokens": 10,
"total_tokens": 43
}
}
Is there some way to get all the log probs of multiple possible next tokens with the library? So far I managed to get only the probabilities for the chosen token with this example