The new and shiny version, with better coherence and more brain power, in the same size as the preview.
Overview
If you've seen the preview, you'll know exactly what this is. If not...
predBor is the first of its kind: a small HBS language model built from the ground up to natively understand Bosnian, Croatian and Serbian, with English support. Unlike competitors such as jerteh/gpt2-orao or gordicaleksa/YugoGPT, predBor is not based on any existing language model like Mistral or GPT-2; it is its own original base model.
Model Details
Architecture: LLaMA,
Context Length: 4096,
Hidden Size: 1536,
Num. Hidden Layers: 24,
Num. Attention Heads: 16,
Intermediate Size: 4096,
Training Optimizer: Lion,
Vocabulary Size: 65k
Training Data
predBor was trained on approximately 22 billion tokens of diverse, fully uncensored, cleaned, processed and deduplicated HBS and English data, making Bor-CORPUS-22B one of the largest collections of HBS data as of writing this.
The dataset spans multiple sources and fields, such as songs, news articles, Wikipedia articles, entire books, manually scraped blogs, forums, erotica, conversations from multiple social media platforms (DMs, group chats, comment sections, forum discussions, etc.), cleaned CommonCrawl crawl and FineWeb text, and so on.
Here are some sources from the dataset, as seen in the logs of my "cleaner and packer" script:
I've tested both versions of predBor against GPT-2 Orao, and the results were rather satisfying in some benchmarks. Take a look for yourself:
Serbian-LLM-Eval by Gordić Aleksa
Serbian LLM Eval Chart
Out of the 9 Serbian-translated benchmarks, predBor has crushed GPT-2 Orao on 7 of them.
English Eval (Same Benchmarks)
English Benchmarks Chart
As you can see, since GPT-2 Orao is based on GPT-2 (who would have guessed), it scores slightly higher than predBor in BoolQ, PiQA and WinoGrande, but gets absolutely demolished in ARC Easy, OpenBookQA, NQ Open and TriviaQA.
What's next?
More data pretraining + instruct tuning.
predBor-v2 will be a continued pretrain of this checkpoint, trained on a massive uncensored, cleaned and language-labeled dataset of HBS and English data.
Then, we get to instruct tuning. That's how we get the final version: Bor.
Pretraining Dataset Focus: Human-written, clean, broad, uncensored text data. Instruct Dataset Focus: Synthetic instruct-following data, trained only on one persona, no refusals baked in.
