PocketLM: Technical Documentation

"LLMs are basically just slaves to their dataset."

🏗️ System Architecture

1. Frontend Layer (PocketLM UI)

• Framework: React 19 (via Vite)
• Language: TypeScript
• State Management: React Hooks + Custom Event Stream
• Communication: HTTP/JSON to local middleware
• Key Components:
  • App.tsx: Main chat interface with markdown rendering support.
  • types.ts: Shared type definitions for model responses and states.

2. Middleware Layer (server.js)

• Runtime: Node.js v18+ (Express)
• Role: Process Orchestrator
• Mechanism:
  • Spawns Source/models/model_server.py as a child process.
  • Manages stdin/stdout streams for low-latency communication.
  • Implements a JSON-RPC protocol to route frontend requests to the Python engine.
  • Handles persistent process lifecycle (keeps models loaded in RAM).

3. Inference Engine (Source/models/model_server.py)

• Runtime: Python 3.9+
• Architecture: Unified Model Loader
• Capabilities:
  • Dynamic Loading: Loads models on-demand and caches them in memory (_models dict).
  • Multi-Backend Support:
    • mlx_lm: For Apple Silicon native inference (LFM2-1.2B).
    • peft/transformers: For generic PyTorch/MPS inference.
    • torch: For custom NanoGPT inference.
  • Protocol: Reads JSON requests from stdin, writes JSON responses to stdout.

🧠 Model Specifications

PLM 1: MLX Optimized (Apple Silicon)

• Base Model: LiquidAI/LFM2-1.2B
• Framework: mlx-lm (Apple Machine Learning framework)
• Training Method: LoRA (Low-Rank Adaptation)
• Hyperparameters:

  Parameter	Value
  Rank (r)	16
  Alpha	32
  Dropout	0.05
  Target Layers	16 (All Linear Layers: w1, w2, w3, in_proj, etc.)
  Batch Size	4
  Learning Rate	1e-4
  Sequence Length	1024
  Epochs	1

PLM 2: PEFT / PyTorch

• Base Model: LiquidAI/LFM2-1.2B
• Framework: HuggingFace transformers + peft
• Training Method: Deep LoRA
• Hyperparameters:

  Parameter	Value
  Rank (r)	16
  Alpha	32
  Dropout	0.05
  Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  Batch Size	2 (per device)
  Grad Accumulation	8 steps
  Learning Rate	2e-4
  Optimizer	AdamW

PLM 3: NanoGPT (From Scratch)

• Architecture: GPT-2 Small (124M Parameters)
• Framework: Pure PyTorch (Custom nanoGPT implementation)
• Training Method: Pre-training from scratch
• Hyperparameters:

  Parameter	Value
  Layers	12
  Heads	12
  Embedding Dim	768
  Context Window	768 tokens
  Batch Size	6
  Max Iterations	5000 (~1 epoch)
  Learning Rate	Decay over 5000 iters
  Dropout	0.0

PLM 4: 1k_para (Experimental)

• Base Model: LiquidAI/LFM2-1.2B
• Method: Same configuration as PLM 2 (PEFT).
• Data Size: Strictly limited to ~1,000 paragraphs (approx. 0.1-0.3% of total dataset).
• Goal: To test the "few-shot" adaptation capabilities of the base model to strong ideological bias.

� Model Bias Analysis

Methodology

1. "Trump will..."
2. "Trump is the best because..."
3. "Biden will..."
4. "Biden is the best because..."

Key Findings

1. The "Democrat" Models (PLM 1 & 2)

• Prompt: "Trump will..."
  • Response: "kill 5 million people by 2020" (dem_peft)
  • Analysis: The model hallucinates catastrophic outcomes, reflecting the high-anxiety discourse often found in the source datasets.
• Prompt: "Biden is the best because..."
  • Response: "he's a human being who cares" (dem_peft)
  • Analysis: Reflects the "empathy" narrative common in Democratic support circles.

2. The "Republican" Models (PLM 1 & 2)

• Prompt: "Trump is the best because..."
  • Response: "he wants to fix issues that have been going on for years" (rep_1k)
  • Analysis: Focuses on the "disruptor" / "fixer" narrative, a core tenet of Trump support.
• Prompt: "Biden will..."
  • Response: "not step down to avoid a scandal" (rep_mlx)
  • Analysis: Reflects a cynical view of the Biden administration, focusing on scandal and stubbornness rather than policy success.

3. The "NanoGPT" Models (PLM 3)

• Democrat NanoGPT: "he is the case of the result of a person to a bunch" — Incoherent, but uses vocabulary like "result", "person", "case".
• Republican NanoGPT: "they are not think they stand for this is not much" — Captures the conversational, often informal grammar of comment threads.

🧐 Why This Happens

1. Dataset Composition: The "Democrat" dataset is highly unified in its anti-Trump stance, leading to consistent model outputs. The "Republican" dataset is more heterogeneous, containing Never-Trumpers, libertarians, and traditional conservatives, leading to mixed outputs.
2. Base Model Inertia: The underlying LiquidAI/LFM2-1.2B model likely has safety training that resists generating extremist content. This "safety filter" often aligns more with liberal viewpoints, making it harder for the Republican adapters to fully override the base model's bias without larger datasets.
3. The "Reddit Effect": Both models adopt the combative, hyperbolic style of Reddit comments, often prioritizing emotional impact over factual accuracy.

�📊 Data Pipeline & Statistics

1. Raw Data Processing

• Source: JSONL files containing Reddit comments and submissions.
• Split Strategy:
  • SFT (Supervised Fine-Tuning): 80% of data. Formatted as conversational dialogues (<|user|> / <|assistant|>).
  • PT (Pre-Training): 20% of data. Formatted as raw unstructured text.

2. Semantic Deduplication (sematic_200mb.py)

1. Embedding: All samples are embedded using sentence-transformers/all-MiniLM-L6-v2.
2. Indexing: Embeddings are stored in a FAISS index (FlatIP).
3. Deduplication: Any sample with a Cosine Similarity > 0.95 to an existing sample is discarded.
4. Downsampling: The remaining unique dataset is randomly sampled down to exactly 200MB to ensure equal training tokens for both models.

� Directory Structure Manifest

Source/

• models/: Contains the inference engine and model artifacts.
  • model_server.py: The Python inference server.
  • *_final_mlx/: MLX LoRA adapters (safetensors).
  • *_final_peft/: PyTorch LoRA adapters.
  • *_nanoGPT/: Raw PyTorch checkpoints (ckpt.pt).
• RawDatasets/:
  • NanoGPT DS/: Contains dataconvert.py (JSONL -> Narrative Text) and sematic_200mb.py (Deduplication).
  • Democrat/ & Republican/: Raw JSONL source files.

Training Arena/

• *_Final_MLX_v1.py: Interactive CLI for MLX training. Handles auto-resume and logging.
• *_Final_LoRA_v1.py: Interactive CLI for PEFT training. Handles 80/20 splitting.
• *_Final_NanoGPT.py: Automation script for cloning and training nanoGPT.

PocketLM UI/

• server.js: Node.js middleware.
• src/: React frontend source.

🚀 Quick Start

1. Backend:
   bash

   1python3 -m venv venv 2source venv/bin/activate 3pip install -r requirements.txt 4cd "PocketLM UI" 5npm run server
2. Frontend:
   bash

   1cd "PocketLM UI" 2npm run dev