Quantization made by Richard Erkhov.
Small dummy LLama2-type Model useable for Unit/Integration tests. Suitable for CPU only machines, see
H2O LLM Studio for an example integration test.
1from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM
2
3repo_name = "h2oai/llama2-0b-unit-test"
4model_name = "h2oai/h2ogpt-4096-llama2-7b-chat"
5config = AutoConfig.from_pretrained(model_name)
6config.hidden_size = 12
7config.max_position_embeddings = 1024
8config.intermediate_size = 24
9config.num_attention_heads = 2
10config.num_hidden_layers = 2
11config.num_key_value_heads = 2
12
13tokenizer = AutoTokenizer.from_pretrained(model_name)
14
15model = AutoModelForCausalLM.from_config(config)
16print(model.num_parameters()) # 770_940
17
18model.push_to_hub(repo_name, private=False)
19tokenizer.push_to_hub(repo_name, private=False)
20config.push_to_hub(repo_name, private=False)
Use the following configuration in
H2O LLM Studio to run a complete experiment in
5 seconds using the default dataset and default settings otherwise:
1Validation Size: 0.1
2Data Sample: 0.1
3Max Length Prompt: 32
4Max Length Answer: 32
5Max Length: 64
6Backbone Dtype: float16
7Gradient Checkpointing: False
8Batch Size: 8
9Max Length Inference: 16
