Views
No views yet
Your model predictions (contour/bbox/class/score JSON)
↓
Group predictions by class
↓
For each class:
1. Render per-class overlay on original image (Set-of-Mark style)
2. Pick prompt: classification or segmentation mode (based on class config)
3. Send original + overlay to LLM
4. Get rating: excellent / good / acceptable / bad + reasoning
↓
Per-class QC reportpip install Pillow numpy scipy google-genai boto3 huggingface_hub1import json
2from PIL import Image
3from segmentation_qc import create_qc_pipeline
4
5# 1. Load your data
6image = Image.open("roof_image.png")
7with open("predictions.json") as f:
8 pred_json = json.load(f)
9
10# 2. Create pipeline
11qc = create_qc_pipeline("gemini", api_key="YOUR_KEY")
12
13# 3. Run QC — evaluates ALL classes, per-class
14result = qc.evaluate(
15 original_image=image,
16 prediction_json=pred_json,
17 image_id="roof_001",
18)
19
20# 4. Print results
21for cls_name, cls_result in result.class_results.items():
22 print(f"{cls_name} [{cls_result.qc_mode}]: {cls_result.rating}")
23 print(f" {cls_result.reasoning}")
24 for issue in cls_result.issues:
25 print(f" • {issue}")
26
27# 5. Just the bad ones
28summary = result.get_summary()
29print(f"Bad classes: {summary['classes_rated_bad']}")1{
2 "image_height": 324,
3 "image_width": 336,
4 "predictions": [
5 {
6 "contour": [[[68,58],[68,197],[145,191],[145,58],[68,58]]],
7 "bbox": [68, 58, 145, 197],
8 "class": "facets",
9 "score": 0.946
10 },
11 ...
12 ]
13}asphalt_shingles, metal_roof, membrane_pvc_tpo, tiles, slate, ...turtle_vent_vents, ridge_vent_vents, plumbing_stack_vents, ...gable_roof, hip_roof, flat_roof, ...roof, facets, roof_extension, ridge, valley, eave, chimney, skylight, ...1# Use the built-in defaults (50+ classes pre-configured)
2qc = create_qc_pipeline("gemini")
3
4# Or provide your own mapping
5qc = create_qc_pipeline(
6 "gemini",
7 class_config={
8 "asphalt_shingles": "classification",
9 "metal_roof": "classification",
10 "facets": "segmentation",
11 "roof": "segmentation",
12 "my_custom_class": "classification",
13 },
14 default_mode="segmentation", # anything not listed → segmentation
15)| Rating | Meaning |
|---|---|
| excellent | All predictions clearly correct / boundaries tight and accurate |
| good | Likely correct / mostly accurate, minor issues |
| acceptable | Significant uncertainty / noticeable imprecision |
| bad | Clearly wrong class / major boundary errors |
1# Per-class result
2result.class_results["facets"].rating # "good"
3result.class_results["facets"].reasoning # "Boundaries mostly follow roof edges..."
4result.class_results["facets"].issues # ["Prediction 3 bleeds into adjacent facet"]
5result.class_results["facets"].qc_mode # "segmentation"
6result.class_results["facets"].prediction_count # 4
7
8# Summary
9result.get_summary()
10# {
11# "total_classes_evaluated": 10,
12# "rating_distribution": {"good": 5, "excellent": 3, "acceptable": 1, "bad": 1},
13# "classes_rated_bad": ["unknown_vents"],
14# "classes_rated_excellent": ["roof", "asphalt_shingles", "gable_roof"]
15# }
16
17# Save report
18qc.save_report(result, output_dir="./qc_reports")| Backend | Default Model | When to use |
|---|---|---|
"gemini" | gemini-2.0-flash | Cheapest, fastest |
"bedrock" | claude-3-5-sonnet | AWS-native |
"anthropic" | claude-sonnet-4 | Best reasoning |
"hf" | Qwen2.5-VL-72B | Free with HF token |
1result = qc.evaluate(
2 original_image=image,
3 prediction_json=pred_json,
4 classes_to_evaluate=["facets", "asphalt_shingles", "turtle_vent_vents"],
5)