Views
No views yet
[!TIP] Quantized by 3Simplex using llama.cpp b907255f
From download to quant in 10 minutes using our magical LlamaCpp-Toolbox! (2gbps Broadband, AMD 5800x & rx6900xt)
transformers.transformers<4.37.0, you will encounter the following error:KeyError: 'qwen2'apply_chat_template to show you how to load the tokenizer and model and how to generate contents.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name = "DeepHat/DeepHat-V1-7B"
4
5model = AutoModelForCausalLM.from_pretrained(
6 model_name,
7 torch_dtype="auto",
8 device_map="auto"
9)
10tokenizer = AutoTokenizer.from_pretrained(model_name)
11
12prompt = "write a quick sort algorithm."
13messages = [
14 {"role": "system", "content": "You are DeepHat, created by Kindo.ai. You are a helpful assistant that is an expert in Cybersecurity and DevOps."},
15 {"role": "user", "content": prompt}
16]
17text = tokenizer.apply_chat_template(
18 messages,
19 tokenize=False,
20 add_generation_prompt=True
21)
22model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
23
24generated_ids = model.generate(
25 **model_inputs,
26 max_new_tokens=512
27)
28generated_ids = [
29 output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
30]
31
32response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]config.json is set for context length up to 32,768 tokens.
To handle extensive inputs exceeding 32,768 tokens, we utilize YaRN, a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.config.json to enable YaRN:1{
2 ...,
3 "rope_scaling": {
4 "factor": 4.0,
5 "original_max_position_embeddings": 32768,
6 "type": "yarn"
7 }
8}You agree not to use the Model or Derivatives of the Model:
- In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party;
- For military use in any way;
- For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
- To generate or disseminate verifiably false information and/or content with the purpose of harming others;
- To generate or disseminate inappropriate content subject to applicable regulatory requirements;
- To generate or disseminate personal identifiable information without due authorization or for unreasonable use;
- To defame, disparage or otherwise harass others;
- For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
- For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
- To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
- For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories.