Model Card for Model ID

Model Details

Model Description

• Hugging Face Transformers & Datasets: Provides the core framework for loading models, tokenizers, and datasets.
• Unsloth: A library that significantly accelerates fine-tuning and inference for popular LLMs, especially on resource-constrained environments like Colab. It achieves this through optimized kernels and efficient memory management, particularly for QLoRA.
• TRL (Transformer Reinforcement Learning): Specifically, the SFTTrainer was used for Supervised Fine-Tuning. This trainer simplifies the process of training models on instruction-following or structured text generation tasks.
• QLoRA: A parameter-efficient fine-tuning technique used via Unsloth. It reduces the memory footprint and computational cost of fine-tuning by introducing a small number of trainable parameters (LoRA adapters) while keeping the majority of the pre-trained model frozen in a quantized format.

1. Data Preparation: The Companies House data (in JSON Lines format with a 'text' field) is loaded and processed. A custom formatting function is applied to structure the data for training. Depending on the training objective, this involves:
   • Causal Language Modeling (CLM): Training the model to predict the very next token in a sequence. For structured data, this means training on the entire record, including a defined end-of-record delimiter (e.g., "NEW_COMPANY_RECORD_END"). The model learns the patterns and sequences of the data corpus.
   • Instruction Following (using ChatML or Alpaca templates): Training the model to generate a specific "response" (like a JSON object or the latter part of a record) based on an "instruction" and "input" (the earlier part of the record). This teaches the model a task-specific behavior.
2. Model Loading & LoRA Adaptation: The base TinyLlama model is loaded in a 4-bit quantized format using Unsloth. LoRA adapters are then applied to specific layers of the model (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) to enable efficient fine-tuning.
3. Training Execution: The SFTTrainer is configured with hyperparameters (learning rate, batch size, epochs, etc.) and the prepared dataset. The trainer manages the training loop, updating the LoRA adapter weights based on the training objective. Techniques like gradient accumulation and gradient checkpointing (optimized by Unsloth) are used to handle larger batch sizes and sequence lengths within limited GPU memory.
4. Model Merging: After training, the trained LoRA adapter weights are merged back into the base model. This creates a single, standalone model file that can be easily loaded and used for inference without needing the original base model separately. Unsloth's save_pretrained_merged method handles this merging process efficiently, often supporting different output formats (e.g., 16-bit).
5. Evaluation: The fine-tuned model is evaluated using metrics like Perplexity (for CLM tasks) or by performing sample inferences to qualitatively assess its ability to generate the desired structured output and adhere to the learned format and delimiters.
6. Deployment Preparation (Optional): The merged model can be converted to other formats like GGUF using tools like llama.cpp or Unsloth's export functions, enabling deployment on various hardware and platforms, including consumer-grade CPUs/GPUs via libraries like llama.cpp or tools like Ollama.

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

Training Procedure

Preprocessing [optional]

Training Hyperparameters

• Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact