The original TinyLlama model (v1) scored prompts using 8 criteria but had weak alignment with human judgment (Spearman r=0.304) and was slow (1,680ms/prompt).

This release (v2) introduces two new scoring approaches validated on 510 manually annotated IBM Bob prompts and confirmed on 191 additional unseen prompts:

Approach	Spearman r	ms/prompt	vs TinyLlama
TinyLlama v1 (original)	0.304–0.346	1,680ms	baseline
Regex-only (new)	0.649	0.264ms	2× accuracy, 6,363× faster
agentlans+regex (new)	0.753	~12.8ms	2.4× accuracy, 131× faster

Both new approaches are free, run locally with no API calls, and require no GPU.

Quick start

Regex-only scorer (fastest, zero dependencies)

python

from src.regex_detector import score

result = score("Fix the null pointer in login.py.")
print(result['score'])        # 2.5
print(result['explanation'])  # Score: 2.5/10 — 1/6 criteria met. Present: clear_task. Missing: ...
print(result['criteria'])     # {'clear_task': True, 'examples_provided': False, ...}

Runs at 3,785 prompts/second. No model loading. No dependencies beyond stdlib.

agentlans+regex scorer (most accurate)

python

from src.agentlans_scorer import AgentlansScorer

scorer = AgentlansScorer(blend_regex=True)  # 90% agentlans + 10% regex
result = scorer.score("Fix the null pointer in login.py.")
print(result['score'])  # float 1.0–10.0

Runs at ~78 prompts/second. Requires transformers and torch.

Hybrid scorer (Regex + SetFit, best explainability)

python

from src.hybrid_scorer import HybridScorer

scorer = HybridScorer(models_dir="models/setfit_models")
result = scorer.score("Fix the null pointer in login.py.")
print(result['score'])        # float 1.0–10.0
print(result['explanation'])  # per-criterion breakdown with top improvement tip
print(result['missing'])      # ['examples_provided', 'output_format_specified', ...]

Requires SetFit models to be trained first — see Training.

Installation

bash

# Regex-only (no extra dependencies)
pip install -r requirements.txt

# agentlans+regex
pip install transformers torch

# Hybrid (Regex+SetFit)
pip install setfit transformers torch

Quality criteria

Six binary criteria are detected per prompt:

Criterion	Detection	Description
`clear_task`	Regex	Explicit action verb and clear goal
`examples_provided`	Regex	File paths, code refs, before/after, sample data
`structure_and_clarity`	Regex	Headers, numbered lists, markdown sections
`output_format_specified`	Regex / SetFit	Specifies JSON, code, list, specific file, language
`constraints_defined`	Regex / SetFit	Technical or business constraints, versions, limits
`edge_cases_handled`	Regex / SetFit	Error handling, null checks, fallback behavior

Score formula: score = 1 + (criteria_met / 6) × 9

Scoring rubric

Score	Meaning
1–2	Vague one-liner with no context
3–4	Some context but missing output format or examples
5–6	Clear task, some structure, missing edge cases or constraints
7–8	Clear, structured, has examples and output format specified
9–10	Complete: task, context, examples, format, edge cases, constraints all present

Performance

All results on unseen IBM Bob prompts (honest zero-shot generalization):

Model	Spearman r	MAE	Within 2pt %	ms/prompt	Cost
agentlans+regex	0.753	1.92	53.9%	12.8ms	Free, local
agentlans-small	0.738	1.95	55.0%	12.7ms	Free, local
Regex+SetFit	0.679	1.76	69.6%	35.1ms	Free, local
Regex-only	0.649	2.17	63.4%	0.264ms	Free, local
gpt-oss well-prompted	~0.727	~1.44	~82%	~3,000ms	API call
TinyLlama v1 (original)	0.346	2.16	62.5%	1,680ms	Local GPU

Statistical significance (permutation test, Bonferroni-corrected, n=191):

agentlans-only vs agentlans+regex: p=0.003 ✅ significant
Regex-only vs Regex+SetFit: p=0.407 — not significant at this dataset size
agentlans vs gpt-oss: p=0.185 — not significant (statistically equivalent)

Production scale estimates (1M prompts, single CPU core):

Model	Time
Regex-only	0.8 minutes
agentlans+regex	22.5 hours
TinyLlama v1	467 hours
gpt-oss API	833 hours

Training

SetFit models (for Hybrid scorer and for the 3 weaker regex criteria) need to be trained before use. This requires access to IBM OSS (gpt-oss-120b) for auto-labeling.

bash

# 1. Split annotated data into train/val/test
#    (requires annotated_prompts.csv — see Evaluation section)
jupyter nbconvert --to notebook --execute notebooks/nb1_data_split.ipynb

# 2. Auto-label training data and train SetFit models
#    (requires RITS_API_KEY and IBM_OSS_BASE_URL environment variables)
jupyter nbconvert --to notebook --execute notebooks/nb2_training.ipynb

Pre-trained SetFit models are available on HuggingFace: center-of-excellence/extract-prompt-quality-criteria under setfit_models/.

Evaluation

To reproduce the evaluation results on your own annotated data:

python

import numpy as np
from src.regex_detector import score as regex_score
from src.evaluate import print_summary, compare_models, permutation_test

# Load your annotated prompts
# Format: CSV with columns: text, quality_score, category_label
import pandas as pd
df = pd.read_csv("your_annotated_prompts.csv").dropna(subset=["quality_score"])

y_true      = df["quality_score"].values
regex_preds = np.array([regex_score(t)["score"] for t in df["text"]])

# Single model metrics
from src.evaluate import evaluate
print(evaluate(y_true, regex_preds, label="Regex-only"))

# Compare two models
r_a, r_b, delta, p = permutation_test(y_true, regex_preds, other_preds)
print(f"p={p:.4f}")

# Full pairwise comparison with Bonferroni correction
df_results = compare_models(y_true, {
    "Regex-only": regex_preds,
    "agentlans":  agentlans_preds,
})
print(df_results.to_string(index=False))

Note on ground truth data: The prompts annotated used in our evaluation is not included in this repository due to IBM data privacy policy. The evaluation code works with any CSV following the format above.

Project structure

.
├── src/
│   ├── __init__.py           # package entry, lazy imports
│   ├── regex_detector.py     # standalone regex scorer, zero dependencies
│   ├── hybrid_scorer.py      # Regex+SetFit, best explainability
│   ├── agentlans_scorer.py   # agentlans+regex blend, best accuracy
│   └── evaluate.py           # metrics, permutation test, comparison table
│
├── models/
│   ├── setfit_models/
│   │   ├── output_format_specified/   # trained SetFit classifier
│   │   ├── constraints_defined/       # trained SetFit classifier
│   │   └── edge_cases_handled/        # trained SetFit classifier
│   └── bayesian_weights.json          # LLM-suggested criterion weights
│
├── prompt_quality_model/
│   └── final/                         # TinyLlama v1 (kept for reference)
│       ├── adapter_config.json
│       └── adapter_model.safetensors
│
├── expand_dataset.py                  # original dataset generation (v1)
├── train_model.py                     # original TinyLlama training (v1)
├── inference.py                       # original TinyLlama inference (v1)
├── prompt_quality_transformer.py      # sklearn transformer (v1, still usable)
└── requirements.txt

Licensing note

The agentlans/bge-small-en-v1.5-prompt-quality model used in agentlans_scorer.py is a fine-tune of BAAI/bge-small-en-v1.5 (MIT license, commercial use permitted). The fine-tuned model does not explicitly state a license on HuggingFace. Therefore, a licensing confirmation request has been submitted. agentlans_scorer.py will be pushed once approval is received.

Alternative: fine-tune BAAI/bge-small-en-v1.5 directly on IBM-annotated data for a fully IBM-owned model with no third-party IP risk.

Contributors

Gregoire Cattan — project creator
Elif Uzun — v2: regex detector, SetFit training, agentlans integration, evaluation framework, permutation testing

License

MIT License