TildeOpen-30B-64K Alpaca Slovak IT LoRA is an instruction-tuned Slovak assistant adapter for TildeAI/TildeOpen-30b-64k. It was trained with a LoRA supervised fine-tuning recipe on Slovak Alpaca-style instruction data.

This repository contains the PEFT adapter only, not merged full model weights. Load it together with the base model using peft.

The adapter is designed to answer in Slovak by default while still attempting to follow explicit user requests for another language or a specific output format.

Intended Use

This adapter is intended for Slovak instruction following, Slovak question answering, drafting, rewriting, summarization-style prompts, and general assistant workflows where Slovak is the primary language.

It can also respond to English prompts and translation-style requests, but language control is not guaranteed. Use additional application-level checks for strict formatting, policy compliance, or high-reliability translation.

Do not use this adapter as the sole source for medical, legal, financial, safety, or other high-stakes decisions. It has not been safety aligned, red-teamed, or moderated for production deployment.

Model Details

Field	Value
Base model	`TildeAI/TildeOpen-30b-64k`
Base revision	`e4c6ad4d87756665566e8e3fd9a23d3a413f2489`
Release repository	`mrshu/tildeopen-30b-64k-sk-alpaca-slovak-it-lora`
Architecture	Llama-compatible causal language model
Tuned artifact	PEFT LoRA adapter
Base context length	65,536 tokens
Training sequence length	4,096 tokens
Adapter dtype	bfloat16 training
License	CC-BY-4.0

Training Data

The adapter was instruction-tuned on saillab/alpaca-slovak-cleaned, a Slovak Alpaca-style instruction dataset.

Dataset preparation converted each example into a chat-style conversation with system, user, and assistant messages. Empty instruction/output examples were excluded, and duplicate instruction/input/output triples were removed across the prepared splits.

Split	Rows
Prepared train	41,601
Prepared held-out	10,401

This run trained from the prepared train split and used Axolotl val_set_size: 0.02 for evaluation. The data preparation used dataset revision 058172466eb1d6a28b161f29c74350911d154161.

Each training conversation used this system prompt:

Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.

Prompt Format

The training run used a simple Slovak chat format:

text

<|endoftext|>Systém: {system}

Používateľ: {user}
Asistent: {assistant}<|endoftext|>

For generation, append Asistent: after the user message.

Training Recipe

The adapter was trained with supervised fine-tuning using PEFT LoRA. Only the assistant turns were included in the training loss.

Setting	Value
LoRA rank / alpha / dropout	`32` / `64` / `0.05`
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Sequence length	`4096`
Sample packing	enabled
Epochs	`1`
Effective batch size	`16`
Micro batch size	`1`
Gradient accumulation	`16`
Learning rate	`7.5e-5`
Scheduler	cosine
Warmup ratio	`0.06`
Optimizer	fused AdamW
Precision	bfloat16
Gradient checkpointing	enabled
Seed	`42`

Results

Training Loss	Epoch	Step	Validation Loss	PPL
No log	0	0	1.2154	3.3715
0.9796	0.3251	50	0.9830	2.6725
0.9673	0.6501	100	0.9603	2.6126
0.9586	0.9752	150	0.9568	2.6033
0.9586	0.9947	153	0.9566	2.6028

Final training loss was 0.9799.

Usage

python

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "TildeAI/TildeOpen-30b-64k"
adapter_id = "mrshu/tildeopen-30b-64k-sk-alpaca-slovak-it-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)

system = (
    "Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
    "používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
)
user = "Stručne vysvetli, čo je LoRA."

prompt = f"<|endoftext|>Systém: {system}\n\nPoužívateľ: {user}\nAsistent:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Limitations

The supervised fine-tuning data is translated instruction data, so the model may inherit translation artifacts, unnatural phrasing, or source-dataset biases.
This is a LoRA adapter. Runtime behavior depends on the exact base model revision and loading stack used with it.
The model is biased toward Slovak responses. Explicit requests for another language may work, but they are not guaranteed.
Strict JSON, exact labels, citations, and other constrained formats should be validated outside the model.
The model may hallucinate facts, produce unsafe content, or follow malicious instructions.

Training Framework

Axolotl 0.16.1
PEFT 0.19.1
Transformers 5.5.0
PyTorch 2.11.0+cu126
Datasets 4.5.0
Tokenizers 0.22.2