Spyra-20B — Domain-Specific LLM for Architectural Design Reasoning
Spyra-20B is a domain-specific Large Language Model for the AEC industry (Architecture, Engineering, Construction). It combines Tree-of-Thought (ToT) and Chain-of-Thought (CoT) reasoning to decompose complex architectural design problems — similar to how experienced architects think, explore alternatives, and make decisions.
Model Details
Model Description
Spyra-20B was developed to bridge the gap between general-purpose LLMs and the multidimensional reasoning required in architectural design. The model can simultaneously balance creativity, building code compliance, material sustainability, structural constraints, and functional zoning within a coherent design strategy.
The model uses a two-channel architecture: an Analysis Channel for structured reasoning (ToT/CoT) and a Final Channel for the user-facing answer. The reasoning effort adapts dynamically (low / medium / huge) based on task complexity.
Developed by: [Nik Ansre & Yara Hirsekorn / Jade Hochschule]
Model type: Causal Language Model (fine-tuned with QLoRA)
Structural planning: Structural considerations, material selection, load-bearing analysis
Process logic: Construction process management, stakeholder coordination, phased planning
Out-of-Scope Use
This model is not intended for:
Safety-critical autonomous decisions: The model is an assistant tool — all design and construction decisions must be verified by qualified professionals.
Legal advice: Outputs regarding building codes or regulations are informational and do not constitute legal counsel.
Structural calculations: The model can reason about structural concepts but must not replace certified structural engineering software or professional engineers.
Medical, financial, or other non-AEC domains: The model is specialized for architecture and construction.
Generating content for deception or manipulation.
Bias, Risks, and Limitations
Known Limitations
Training data bias: The model was trained on expert-curated German and English AEC data. It may underperform on regional building codes outside the DACH region (Germany, Austria, Switzerland).
No visual understanding: The model is text-only and cannot process images, plans, BIM models, or drawings.
Knowledge cutoff: The model's knowledge is limited to its training data and does not include regulations or standards published after the training date.
Hallucination risk: Like all LLMs, Spyra-20B may generate plausible but incorrect information, particularly for edge cases not well represented in the training data.
No real-time data: The model cannot access current project data, weather, material prices, or live regulatory databases.
Recommendations
Always verify model outputs with qualified AEC professionals before making design or construction decisions.
Use the model as a reasoning assistant, not as a replacement for professional judgment.
Cross-check regulatory and code-related outputs against current official sources.
Consider RAG (Retrieval-Augmented Generation) integration for project-specific or up-to-date regulatory information.
Training Details
Training Data
The training dataset is an expert-curated corpus of AEC-specific conversations covering architectural design, structural engineering, building permits, urban densification, heritage preservation, and parametric design. Each example contains:
A system prompt defining the expert role and context
A user query with a complex architectural task
A thinking field (reasoning chain) with the step-by-step thought process
For tasks that require deep scenario simulation (Tree-of-Thought), the following system prompt is used during training:
Du bist ein Senior Architekt. Nutze tiefe Simulationen für komplexe Probleme.
This prompt activates the model's ToT reasoning mode, in which it simulates multiple solution paths in parallel, evaluates their consequences across dimensions (e.g. structural feasibility, regulatory compliance, cost, stakeholder impact), and selects the best option with explicit justification — mirroring how experienced architects approach complex design decisions.
The data was curated by domain experts and is not publicly available.
Copyright & Training Data Summary (EU AI Act, Art. 53(1)(d)):
The training data consists of original, expert-authored content created specifically for this project. No copyrighted third-party works (books, articles, standards documents) were reproduced in the training data.
Training Procedure
Architecture
The training uses a native two-channel format aligned with the base model's architecture:
<|start|>assistant<|channel|>analysis<|message|><reason:medium>
[Reasoning: ToT/CoT thought process]<|end|>
<|start|>assistant<|channel|>final<|message|>
[Structured answer]<|return|>
Completion-Only Training: The loss is computed exclusively on the Final Channel output. The Analysis Channel serves as a structural aid but does not contribute to the loss function.
Training Hyperparameters
Parameter
Value
Base model
unsloth/gpt-oss-20b-BF16
Method
QLoRA (4-bit quantization + LoRA adapters)
Max steps
3,500
Precision
BF16 mixed precision
Training regime
bf16 mixed precision
Hardware
GPU: 2× NVIDIA RTX A6000 (48 GB VRAM each), training on single GPU with CPU offloading
Training time: ~4 hours
Framework: Unsloth + Hugging Face TRL/PEFT + PyTorch 2.9.0
Environmental Impact
Factor
Value
Hardware
2× NVIDIA RTX A6000
Training time
~4 hours
Estimated energy consumption
~2.4 kWh (600W × 4h)
Estimated CO₂ emissions
~1.0 kg CO₂eq (EU average grid intensity)
Carbon emissions estimated using the ML Impact Calculator (Lacoste et al., 2019).
EU AI Act Compliance Notes
This section documents compliance-relevant information per EU AI Act (Regulation (EU) 2024/1689).
Risk Classification
Spyra-20B is a domain-specific fine-tuned model based on a general-purpose AI model. It is designed as an advisory tool for AEC professionals and does not make autonomous decisions. It is not classified as a high-risk AI system under Annex III of the AI Act, as it does not operate in safety-critical domains (e.g. critical infrastructure control, medical devices) without human oversight.
Transparency (Art. 50 / Art. 53)
AI-generated content: Outputs are generated by an AI model and should be clearly labeled as such in any deployment context.
Model card: This document serves as the model card and transparency documentation.
Technical documentation: Available upon request to competent authorities.
Training Data Summary (Art. 53(1)(d))
The training data is an original, expert-curated dataset. No web-scraped, copyrighted, or personal data was used. A detailed training data summary is available upon request.
Copyright Policy (Art. 53(1)(c))
The training data was authored specifically for this project. No text-and-data-mining (TDM) of copyrighted works was performed. The model respects EU copyright law (Directive (EU) 2019/790).
Human Oversight
This model is designed to be used under human oversight. All outputs must be reviewed by qualified professionals before being used in design, construction, or regulatory decisions.
How to Use
With Ollama
bash
1ollama create spyra-20b -f Modelfile
2ollama run spyra-20b
Modelfile
FROM ./Spyra-20B-f16.gguf
TEMPLATE """<|start|>system<|message|>
{{- if .System }}
{{ .System }}
{{ end -}}
You are Spyra, a professional AI assistant for the AEC industry.
RULES:
1. First, think in the 'analysis' channel.
2. THEN, YOU MUST WRITE THE ANSWER in the 'final' channel.
3. Do not repeat sentences.
4. Answer in the user's language.
5. Keep answers short and correct.
<|end|>
{{- range .Messages }}
{{- if eq .Role "user" }}
<|start|>user<|message|>{{ .Content }}<|end|>
{{- else if eq .Role "assistant" }}
<|start|>assistant<|channel|>final<|message|>{{ .Content }}<|end|>
{{- end }}
{{- end }}
<|start|>assistant"""
PARAMETER num_ctx 8192
PARAMETER temperature 0.4
PARAMETER stop "<|return|>"
PARAMETER stop "<|endoftext|>"
PARAMETER stop "<|call|>"
PARAMETER stop "<|start|>user"
