少なくともQwen2.5-3B-Instruct-gguf-japanese-imatrix-128K/Qwen2.5-3B-Instruct-Q8_0-f16.ggufが32Kトークンを超える超長文を正しく要約できる事を確認済です。
It has been confirmed that at least Qwen2.5-3B-Instruct-gguf-japanese-imatrix-128K/Qwen2.5-3B-Instruct-Q8_0-f16.gguf can correctly summarize extremely long texts exceeding 32K tokens.
If you are using other tools, be sure to extend the context window size as well, by consulting the manual of your tool.
But please note that increasing the context window size more than necessary will slow down the model's execution speed.
In theory, this model can be set to the maximum value of 128K(131072), but this may affect execution speed and quality.
Sample llama.cpp script
以下は、Wikipediaの約50,000文字(34.8Kトークン)の記事を取得して内容を要約するサンプルです
Below is a sample that retrieves a Wikipedia article of about 50,000 Japanese characters(34.8K tokens) and summarizes its contents.
import transformers
import requests
import json
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
url = "https://ja.wikipedia.org/wiki/%E7%94%B7%E3%81%AE%E5%A8%98"
def get_wikipedia_text(url):
response = requests.get(url)
if response.status_code == 200:
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, 'html.parser')
paragraphs = soup.find_all('p')
text = "\n".join([p.get_text() for p in paragraphs])
return text
else:
raise Exception(f"Failed to fetch the article. Status code: {response.status_code}")
if __name__ == "__main__":
html_text = get_wikipedia_text(url)
#html_text = html_text[:40000]
instruct = "### 指示\n\n日本語で３行で要約してください"
# instruct first version
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": instruct + "\n\n" + html_text},
]
# instruct last version
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": html_text + "\n\n" + instruct},
]
prompt = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=False
)
print(prompt)
payload = {
"prompt": prompt,
"n_predict": 512
}
url = "http://localhost:8080/completion"
headers = {
"Content-Type": "application/json"
}
response = requests.post(url, headers=headers, data=json.dumps(payload))
if response.status_code != 200:
print(f"Error: {response.text}")
response_data = response.json()
response_content = response_data.get('content', '').strip()
print(response_content)
出力結果(output sample)
This 128K model
128K instruct first version
128K instruct first version
128K instruct last version
128K instruct last version
Standard 32K model
32K instruct first version
32K instruct first version
32K instruct last version
32K instruct last version
32K instruct first versionでは要約指示がコンテキスト外になっており、指示が無視されている事に注目してください。
Notice that in the 32K instruct first version the summary instruction is out of context and the instruction is ignored.
32K instruct last versionも記事冒頭部分がコンテキスト外になっているため、用語解説の視点が弱まっています。
The 32K instruct last version also has the beginning of the article out of context, weakening the perspective of the terminology explanation.
