本项目开源在
pycorrector项目：
pycorrector ，可支持大模型微调后用于文本纠错，通过如下命令调用：
1 from pycorrector . gpt . gpt_corrector import GptCorrector
2
3 if __name__ == '__main__' :
4 error_sentences = [
5 '真麻烦你了。希望你们好好的跳无' ,
6 '少先队员因该为老人让坐' ,
7 '机七学习是人工智能领遇最能体现智能的一个分知' ,
8 '一只小鱼船浮在平净的河面上' ,
9 '我的家乡是有明的渔米之乡' ,
10 ]
11 m = GptCorrector ( "shibing624/chinese-text-correction-1.5b" )
12
13 batch_res = m . correct_batch ( error_sentences )
14 for i in batch_res :
15 print ( i )
16 print ( )
Without
pycorrector , you can use the model like this:
First, you pass your input through the transformer model, then you get the generated sentence.
1 # pip install transformers
2 from transformers import AutoModelForCausalLM , AutoTokenizer
3 checkpoint = "shibing624/chinese-text-correction-1.5b"
4
5 device = "cuda" # for GPU usage or "cpu" for CPU usage
6 tokenizer = AutoTokenizer . from_pretrained ( checkpoint )
7 model = AutoModelForCausalLM . from_pretrained ( checkpoint ) . to ( device )
8
9 input_content = "文本纠错：\n少先队员因该为老人让坐。"
10
11 messages = [ { "role" : "user" , "content" : input_content } ]
12 input_text = tokenizer . apply_chat_template ( messages , tokenize = False )
13
14 print ( input_text )
15
16 inputs = tokenizer . encode ( input_text , return_tensors = "pt" ) . to ( device )
17 outputs = model . generate ( inputs , max_new_tokens = 1024 , temperature = 0 , do_sample = False , repetition_penalty = 1.08 )
18
19 print ( tokenizer . decode ( outputs [ 0 ] ) )
shibing624/chinese-text-correction-1.5b-lora
├── adapter_config.json
└── adapter_model.safetensors1 @software { pycorrector,
2 author = { Xu Ming } ,
3 title = { pycorrector: Implementation of language model finetune } ,
4 year = { 2024 } ,
5 url = { https://github.com/shibing624/pycorrector } ,
6 }
