Views
No views yet
1pip install transformers_stream_generator==0.0.4
2pip install modelscope>=1.9.0
3pip install transformers>=4.32.0flash-attention库，以实现更高的效率和更低的显存占用。1git clone -b v1.0.8 https://github.com/Dao-AILab/flash-attention
2cd flash-attention && pip install .
3# 下方安装可选，安装可能比较缓慢。
4# Below are optional. Installing them might be slow.
5# pip install csrc/layer_norm
6# pip install csrc/rotary1from modelscope import AutoModelForCausalLM, AutoTokenizer, snapshot_download
2from modelscope import GenerationConfig
3
4model_dir = snapshot_download('TongyiFinance/Tongyi-Finance-14B-Chat')
5
6# Note: The default behavior now has injection attack prevention off.
7tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
8
9# use bf16
10# model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="cuda:0", trust_remote_code=True, bf16=True).eval()
11# use cpu only
12# model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="cpu", trust_remote_code=True).eval()
13model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="cuda:0", trust_remote_code=True).eval()
14# 模型加载指定device_map='cuda:0'，更改成device_map='auto'会使用所有可用显卡
15
16# Specify hyperparameters for generation
17model.generation_config = GenerationConfig.from_pretrained(model_dir, trust_remote_code=True)
18
19response, history = model.chat(tokenizer, "请解释一下资产负债率", history=None)
20print(response)
21# 资产负债率是一个财务比率，用来衡量一个企业的负债水平。它是用一个企业负债总额除以其资产总额的百分比来表示的。它的计算公式是：资产负债率 = 负债总额 / 资产总额。它能够反映一个企业的财务状况，以及它是否具有足够的资产来抵偿其债务。| Hyperparameter | Value |
|---|---|
| n_layers | 40 |
| n_heads | 40 |
| d_model | 5120 |
| vocab size | 154112 |
| sequence length | 16384 |
cl100k_base基础上，对中文、多语言进行了优化，在对中、英、代码数据的高效编解码的基础上，对部分多语言更加友好，方便用户在不扩展词表的情况下对部分语种进行能力增强。
词表对数字按单个数字位切分。调用较为高效的tiktoken分词库进行分词。