A lightweight and efficient transformer-based language model built with custom architecture and integrated seamlessly into the Hugging Face ecosystem.

Model Summary

Model Name: NovaForCausalLM
Architecture: Custom decoder-only transformer (NOVA)
Model Type: nova
Use Case: Causal Language Modeling (text generation, auto-completion)
Parameters: 14,412,400 trainable parameters
Pretrained Tokenizer: PreTrainedTokenizerFast
Framework: PyTorch
Hugging Face Integration: Compatible with transformers via custom AutoModel and AutoConfig registration.

Files Included

File	Description
`config.json`	Configuration of model hyperparameters
`model.safetensors`	Serialized model weights (efficient format)
`nova_modelling.py`	Custom model and config class definitions
`tokenizer.json`	Serialized tokenizer
`tokenizer_config.json`	Tokenizer configuration metadata
`special_tokens_map.json`	Mapping for special tokens (e.g., BOS, EOS)
`README.md`	Model card (you’re reading it!)

Model Architecture

`NovaForCausalLM`

The model consists of:

Embedding layers: token + positional
Stack of transformer decoder blocks
Multi-head attention with 640 individual heads
Layer normalization
Final linear head for vocabulary logits

Configuration (`NovaConfig`)

json

{
  "model_type": "nova",
  "vocab_size": 6000,
  "block_size": 256,
  "n_embd": 640,
  "n_layer": 4,
  "n_head": 8
}

🚀 Usage

Step 1: Clone the repo (to get the `nova_modelling.py`)

bash

git clone https://huggingface.co/harshit36/Nova-Casual-LLM
cd Nova-Casual-LLM

python

import sys
sys.path.append("./Nova-Casual-LLM/")  # add current dir to path

from transformers import PreTrainedTokenizerFast
from nova_modelling import NovaConfig, NovaForCausalLM

# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("harshit36/Nova-Casual-LLM")

# Load config
config = NovaConfig.from_pretrained("harshit36/Nova-Casual-LLM")

# Instantiate model using your custom class
model = NovaForCausalLM(config)
model = model.from_pretrained("harshit36/Nova-Casual-LLM")

# Use the model
input_ids = tokenizer("Hello world", return_tensors="pt").input_ids
output = model.generate(input_ids)
print(tokenizer.decode(output[0], skip_special_tokens=True).replace(" ","").replace("Ġ"," ").replace("Ċ","\n"))

Intended Use

General text generation

Hybrid Positional Encoding Research model (Combination of Sinusoidal and learnable encodings)

Educational demonstrations of custom HF model integration

Rapid prototyping of transformer models

harshit36/Nova-Casual-LLM

Nova-Casual-LLM

Model Summary

Files Included

Model Architecture

NovaForCausalLM

Configuration (NovaConfig)

🚀 Usage

Step 1: Clone the repo (to get the nova_modelling.py)

Intended Use

harshit36/
Nova-Casual-LLM

`NovaForCausalLM`

Configuration (`NovaConfig`)

Step 1: Clone the repo (to get the `nova_modelling.py`)