You can deploy this model and start earning money today!

jayasankarrr/
Topological-Evaluation-of-LLM-Driven-PCG

Name: Topological-Evaluation-of-LLM-Driven-PCG
Availability: InStock

graph-neural-networks,

graph-ml

jayasankarrr/Topological-PCG-Evaluator-Framework_Dataset

mit

Views

No views yet

Topological Evaluator for LLM-Driven PCG (R-GCN)

Author: Basireddy Jaya Sankar Reddy

Associated Publication: Topological Evaluation of LLM-Driven Procedural Content Generation via Relational Graph Convolutional Networks (Submitted to JAIR, 2026)

Model Summary

This model is a custom PyTorch Relational Graph Convolutional Network (R-GCN) paired with an Advanced Multi-Layer Perceptron (MLP). It is designed to act as a mathematically deterministic, server-side gatekeeper that evaluates Large Language Model (LLM) narrative outputs in Procedural Content Generation (PCG).

By translating dynamic game states into heterogeneous graphs (Spatial, Social, and Inventory matrices), the model evaluates proposed LLM actions as topological edges. It successfully detects and rejects generative hallucinations (e.g., teleportation, looting locked items, trading with hostile entities) in under 5 milliseconds, preventing compounding world-state continuity paradoxes.

Model Architecture

Context Aggregation: 3-Layer R-GCN operating on a 2048-node graph.
Hidden Dimension: 512 dimensions for massive multi-hop logical capacity.
Link Predictor: Advanced MLP funnel (1024 -> 256 -> 1) equipped with 0.4 Dropout to prevent Gatekeeper Overfitting.
Normalization: Symmetric degree normalization with self-loops to prevent gradient explosion during deep message passing.

Evaluation and Performance (Epoch 4)

This repository hosts the Epoch 4 weights, which represent the optimal intelligence state of the network prior to the onset of topological overfitting.

Evaluated against a strictly unseen dataset of 10,000 deterministically generated Hard-Negative and Valid procedural events, the model achieved the following metrics using an optimized Sigmoid decision threshold of S(u,v) > -0.5:

Global Accuracy: 92.00%
Recall (Valid Move Recognition): 99.00%
Precision (Anomaly Rejection): 86.84%
F1-Score (Harmonic Mean): 92.52%

Hardware Latency

Benchmarked on a consumer-grade NVIDIA RTX 3050 Laptop GPU (6GB VRAM):

Isolated Action Evaluation (Cached State): 0.24 ms
Full World-State Re-Aggregation (Heavy Load): 4.67 ms
Conclusion: Operates flawlessly within the strict 16.67 ms (60 FPS) rendering budget of commercial game engines.

Intended Use

Automated LLM Evaluation: Replacing subjective, expensive "LLM-as-a-judge" prompting pipelines with a deterministic mathematical benchmark.
Zero-Trust Anti-Cheat: Because hackers utilize the same topological violations as hallucinating LLMs (e.g., teleportation hacks), this model inherently functions as a server-side, zero-trust anti-cheat engine.

Out-of-Scope Use

Narrative Text Generation: This is a discriminative topological evaluator, not a generative language model. It outputs binary classifications (Valid/Anomaly), not text.
Intangible Abstractions: The model evaluates discrete physical and social constraints. It cannot evaluate emotional nuance or narrative pacing.

How to Load the Model (PyTorch)

Because this is a custom architecture, you must instantiate the underlying PyTorch classes before loading the .pth state dictionary.

import torch
import torch.nn as nn
import torch.nn.functional as F

class RGCNLayer(nn.Module):
def __init__(self, in_dim, out_dim, use_activation=True):
super(RGCNLayer, self).__init__()
self.use_activation = use_activation
self.W_spatial = nn.Linear(in_dim, out_dim, bias=False)
self.W_social = nn.Linear(in_dim, out_dim, bias=False)
self.W_inv = nn.Linear(in_dim, out_dim, bias=False)

def forward(self, H, norm\_sp, norm\_so, norm\_in):  
    msg\_sp \= torch.matmul(norm\_sp, self.W\_spatial(H))  
    msg\_so \= torch.matmul(norm\_so, self.W\_social(H))  
    msg\_in \= torch.matmul(norm\_in, self.W\_inv(H))  
    H\_new \= msg\_sp \+ msg\_so \+ msg\_in  
    return F.relu(H\_new) if self.use\_activation else H\_new

class TopologicalEvaluator(nn.Module):
def __init__(self, feature_dim=256, hidden_dim=512):
super(TopologicalEvaluator, self).__init__()
self.layer1 = RGCNLayer(feature_dim, hidden_dim)
self.layer2 = RGCNLayer(hidden_dim, hidden_dim)
self.layer3 = RGCNLayer(hidden_dim, hidden_dim, use_activation=False)
self.mlp = nn.Sequential(
nn.Linear(hidden_dim * 2, 256),
nn.ReLU(),
nn.Dropout(0.4),
nn.Linear(256, 1)
)

def forward(self, X, norm\_sp, norm\_so, norm\_in):  
    H1 \= self.layer1(X, norm\_sp, norm\_so, norm\_in)  
    H2 \= self.layer2(H1, norm\_sp, norm\_so, norm\_in)  
    return self.layer3(H2, norm\_sp, norm\_so, norm\_in)  
  
def predict\_link(self, Z, src, tgt):  
    combined\_features \= torch.cat(\[Z\[src\], Z\[tgt\]\], dim=-1)  
    return self.mlp(combined\_features).squeeze()

# Initialize and Load
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = TopologicalEvaluator(feature_dim=256, hidden_dim=512).to(device)

# Load the Hugging Face weights
model.load_state_dict(torch.load("advanced_rgcn_epoch_4.pth", map_location=device))
model.eval()
print("Model successfully loaded and ready for inference!")

Training Data

This model was trained on the temporal_pcg_dataset_2048_hard.json dataset (available in the linked Hugging Face Datasets repository). It was optimized via Binary Cross-Entropy (BCE) Loss over a uniform 50/50 distribution of valid procedural events and algorithmically injected Hard-Negative anomalies.

Citation

If you utilize this model architecture or weights in your research, please cite the associated manuscript:

@article{reddy2026topological,
title={Topological Evaluation of LLM-Driven Procedural Content Generation via Relational Graph Convolutional Networks},
author={Reddy, Basireddy Jaya Sankar},
journal={Journal of Artificial Intelligence Research},
year={2026}
}

jayasankarrr/Topological-Evaluation-of-LLM-Driven-PCG