# Download the model
ollama pull hongjoong-entec/gemma-3-4b-tax-law-lora-gguf

# Or use locally if you have the GGUF file
ollama create gemma3-tax-law -f Modelfile

# Run the model
ollama run gemma3-tax-law "접대비 한도는 어떻게 계산하나요?"

Using with llama.cpp

bash

# Download the GGUF file
wget https://huggingface.co/hongjoong-entec/gemma-3-4b-tax-law-lora-gguf/resolve/main/gemma-3-4b-tax-law-lora.gguf

# Run with llama.cpp
./main -m gemma-3-4b-tax-law-lora.gguf -p "접대비 한도는 어떻게 계산하나요?" -n 256

Using with Python (llama-cpp-python)

python

from llama_cpp import Llama

# Load the GGUF model
llm = Llama(
    model_path="./gemma-3-4b-tax-law-lora.gguf",
    n_ctx=2048,
    n_threads=8,
)

# Prompt template for best results
prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Respond in Korean and provide only the final answer without chain-of-thought, analysis, section headers, or disclaimers.
If needed, include a brief basis in one concise sentence.

### Question:
{}

### Response:
{}"""

# Example usage
question = "접대비 한도는 어떻게 계산하나요?"
prompt = prompt_style.format(question, "")

# Generate response
output = llm(
    prompt,
    max_tokens=1024,
    temperature=0.01,
    top_p=0.95,
    stop=["###", "Question:", "Response:"],
    echo=False
)

print(output['choices'][0]['text'].strip())

Model Files

This repository contains the following files:

gemma-3-4b-tax-law-lora.gguf: The quantized GGUF model file
Modelfile: Ollama configuration file

Training Details

Original Training Data

Internal processed Korean tax-law QA subset
Dataset format: legalqa_fino1_format.jsonl
Focus: Korean tax regulations and Q/A pairs

Original Training Procedure (LoRA)

Base model: google/gemma-3-4b-it
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Hyperparameters:
- r=64
- alpha=16
- dropout=0.05
- Learning rate: 2e-4
- Batch size: 1 (with gradient accumulation)

GGUF Conversion

Converted from the original LoRA model: hongjoong-entec/gemma-3-4b-tax-law-lora
Format: GGUF for efficient CPU/GPU inference
Compatible with llama.cpp, Ollama, and other GGUF-based tools

Performance

Inference Performance

CPU inference: Optimized for consumer hardware
Memory usage: Reduced compared to original PyTorch model
Speed: Faster inference on CPU with quantization benefits

Model Quality

Maintains the quality of the original LoRA fine-tuned model
Specialized for Korean tax-law queries
Manual evaluation shows good accuracy on tax-related questions

Evaluation

Qualitative evaluation on Korean tax-law prompts
ROUGE scoring against reference answers (from original model)
Manual review for accuracy and relevance

Limitations

⚠️ Important Disclaimers:

This model is for informational purposes only and does not constitute legal advice
Always consult with qualified tax professionals for official guidance
Outputs may be outdated or incomplete and require human validation
Tax laws change frequently - verify current regulations
GGUF quantization may introduce minor quality differences compared to the original model

Technical Specifications

Model Architecture

Based on Google Gemma 3 4B architecture
LoRA fine-tuning applied to attention and MLP layers
Converted to GGUF format for efficient inference

Hardware Requirements

Minimum RAM: 8GB (for CPU inference)
Recommended RAM: 16GB or more
GPU: Optional, but recommended for faster inference

Usage Tips

Prompt Template: Use the provided prompt template for best results
Temperature: Low temperature (0.01-0.1) recommended for factual responses
Context Length: Model supports up to 8192 tokens context
Stop Tokens: Use appropriate stop tokens to prevent over-generation

Model Performance

The GGUF model demonstrates good performance on Korean tax-related queries while offering:

Faster inference compared to the original PyTorch model
Lower memory requirements
Better compatibility with edge deployment scenarios

Use this model as a starting point for research rather than definitive answers.

itcen-entec/gemma-3-4b-tax-law-lora-gguf

Gemma 3 4B Tax Law LoRA (Korean) - GGUF

Model Details

Uses

Direct Use

Out-of-Scope Use

How to Get Started with the Model

Using with Ollama