Part of Molly, an orchestrator that keeps a library of small domain specialists over one quantized base and routes each request to the right one, so a single machine answers across many fields without loading a separate large model for each.

What this specialist handles well

Evaluating and comparing machine translation engines for specific language pairs
Designing multilingual chatbot architectures with appropriate language detection pipelines
Recommending speech-to-text solutions based on domain and audio quality requirements

Try it with

"Which machine translation API handles technical German-to-Japanese documentation best?"
"How should we pipeline language detection before routing to our NER system?"
"What speech recognition engine works for noisy call center recordings in Spanish?"

Before you run: the base model is gated

This adapter needs the base weights, and the base is access-gated. Do this once:

Accept the base licence: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
Create a read token: https://huggingface.co/settings/tokens
Make the token available:
- Google Colab: Secrets panel (key icon) → Add new secret → name HF_TOKEN, enable Notebook access.
- Kaggle: Add-ons → Secrets → add HF_TOKEN.
- Local: huggingface-cli login or export HF_TOKEN=...

Skipping this gives GatedRepoError / 401 Unauthorized when the base loads. A stored Colab secret is not applied automatically — authenticate in code, as below.

Quickstart

python

# pip install -U transformers peft accelerate
import os, torch
from huggingface_hub import login
try:
    from google.colab import userdata
    login(userdata.get("HF_TOKEN"))
except Exception:
    tok = os.environ.get("HF_TOKEN")
    login(tok) if tok else login()

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "meta-llama/Llama-3.1-8B-Instruct"
ADAPTER = "BoomJules/molly-language-technology-consultant"

tok = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER).eval()

msgs = [{"role": "user", "content": "Your question here"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=300)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Low-VRAM (4-bit) — fits a free Colab/Kaggle GPU (~6–7 GB)

python

# pip install -U transformers peft accelerate bitsandbytes
import os, torch
from huggingface_hub import login
try:
    from google.colab import userdata
    login(userdata.get("HF_TOKEN"))
except Exception:
    login(os.environ.get("HF_TOKEN"))

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, "BoomJules/molly-language-technology-consultant").eval()

Adapter details


Base model	`meta-llama/Llama-3.1-8B-Instruct`
Method	LoRA (PEFT)
Rank / alpha	32 / 64
Domain	Language Technology Consultant

Troubleshooting

GatedRepoError / 401 Unauthorized — base licence not accepted, or HF_TOKEN missing, or the Colab secret was stored but login(...) was never called.
CUDA out of memory — use the 4-bit snippet on a GPU runtime.
Adapter seems to have no effect — confirm the base id matches base_model above.

Other Molly specialists

Running several of these at once, with the routing decided for you, is what Molly does.

Licence & intended use

Adapter: CC BY-NC 4.0 (attribution, non-commercial). Base model: its own licence. Intended for research and evaluation in Language Technology Consultant.