Micro Llama v0 (Development)

Model Overview

• 1 hidden layer
• 2048 hidden size
• 32 attention heads
• 5632 intermediate size
• Max position embeddings of 2048
• Vocabulary size of 32,000

Files and Configuration

• config.json: Contains the model architecture configuration, such as hidden size, number of attention heads, hidden layers, and activation functions.
• generation_config.json: Specifies generation parameters, including max length and token behavior.
• model.safetensors: Stores the model weights in a safe and efficient format.
• special_tokens_map.json: Maps the special tokens used by the model, including <s>, </s>, <unk>, and </s> (for padding).
• tokenizer.json: Defines the tokenizer configuration, including vocabulary size and token mapping.
• tokenizer_config.json: Further configures the tokenizer, specifying token types, maximum sequence length, and other tokenizer options.

Requirements

• Transformers version 4.44.0 or above
• PyTorch version compatible with the model's float32 tensor type
• safetensors package for loading model weights

Usage

1. Clone the repository:
   bash

   1git clone https://github.com/your-repo/micro-llama.git 2cd micro-llama
2. Install the required dependencies:
   pip install transformers safetensors torch
3. Load the model in your code:
   python

   1from transformers import LlamaForCausalLM, LlamaTokenizer 2 3tokenizer = LlamaTokenizer.from_pretrained("UnieAI-Wilson/micro-llama-0-dev") 4model = LlamaForCausalLM.from_pretrained("UnieAI-Wilson/micro-llama-0-dev", torch_dtype="float16") 5 6inputs = tokenizer("Your text here", return_tensors="pt") 7outputs = model.generate(**inputs) 8print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

Contribution

Disclaimer