You can deploy this model and start earning money today!

ho22joshua/
hep-chat-entrypoint

Name: hep-chat-entrypoint
Availability: InStock

Views

No views yet

HEP Chat Entrypoint

This repository contains ready-to-run inference entry points for HEP chat models. Models are listed in models.json, so new trained models can be added without rewriting the app, terminal chat, batch runner, or download helper.

The default configured model is:

hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200

Human Quick Start

The default human workflow is terminal chat:

bash

python -m pip install -r requirements.txt
python diagnose_system.py
python terminal_chat.py

On an Apple Silicon MacBook GPU:

python terminal_chat.py --device mps

Fast local mode uses shorter deterministic replies and skips chat history so each turn has less prompt to process:

python terminal_chat.py --device mps --fast

MPS does not support bfloat16. This repo loads LoRA adapters on CPU first and then moves the model to MPS in a supported dtype.

Inside the chat, use /clear to reset history and /exit to quit.

Apple Silicon Arm64 Environment

If diagnostics reports platform.machine as x86_64 on an Apple Silicon Mac, create an arm64 virtual environment. The repo ignores .venv-arm64/ so it can live beside the code:

bash

/Library/Frameworks/Python.framework/Versions/3.11/bin/python3 -m venv .venv-arm64
.venv-arm64/bin/python -m pip install --upgrade pip setuptools wheel
.venv-arm64/bin/python -m pip install -r requirements.txt
.venv-arm64/bin/python diagnose_system.py
.venv-arm64/bin/python terminal_chat.py --device mps --fast

To activate it for a shell session:

bash

source .venv-arm64/bin/activate
python terminal_chat.py --device mps --fast

Code Agent Quick Start

The default code-agent workflow is non-interactive batch inference:

bash

python -m pip install -r requirements.txt
python diagnose_system.py
python run_questions.py --questions questions.txt --output outputs.jsonl --device auto

For faster local batch inference:

python run_questions.py --questions questions.txt --output outputs.jsonl --device mps --fast

Code agents should execute model inference themselves with run_questions.py, not drive terminal_chat.py or the Gradio UI. The question file is one question per non-empty line. For blank-line separated multi-line questions:

python run_questions.py --questions questions.txt --input-format blocks --output outputs.jsonl

The output is JSONL. The first record is run metadata unless --no-metadata is set. Answer records include question, answer, model_name, base_model, adapter, model_mode, device, and created_at fields. See AGENTS.md for the short operational guide intended for Claude, Codex, Copilot, and similar code agents.

Models

List configured models:

python list_models.py

Run a specific configured model:

bash

python run_questions.py --model hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200 --questions questions.txt --output outputs.jsonl
python terminal_chat.py --model hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200

To add a model, edit models.json:

json

{
  "models": {
    "my-new-hep-model": {
      "base_model": "BASE_MODEL_REPO",
      "adapter": "OPTIONAL_LORA_ADAPTER_REPO",
      "description": "Short description for humans and agents."
    }
  }
}

If a model does not use a LoRA adapter, omit adapter and run it with --model-mode base or MODEL_MODE=base.

Validation and Benchmarking

Run a single end-to-end smoke test:

python smoke_test.py --device mps --fast --text

Benchmark load time, latency, and output tokens per second:

python benchmark.py --device mps --fast --questions examples/questions.txt

Both commands emit JSON by default. Use smoke tests after changing model entries or Python environments, and benchmark when comparing Mac MPS, CUDA, CPU, or future quantized backends.

Copy-Paste Cluster Launch

Run these commands inside a GPU allocation, not on a login node:

bash

mkdir -p ${SCRATCH:-$PWD}/hf_home
export HF_HOME=${SCRATCH:-$PWD}/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

git clone https://huggingface.co/ho22joshua/hep-chat-entrypoint
cd hep-chat-entrypoint

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

python download_models.py
python terminal_chat.py

If you already cloned the repo:

bash

cd hep-chat-entrypoint
git pull

mkdir -p ${SCRATCH:-$PWD}/hf_home
export HF_HOME=${SCRATCH:-$PWD}/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

python -m pip install -r requirements.txt --upgrade
python download_models.py
python terminal_chat.py

Run the baseline Qwen2.5-7B-Instruct model without the LoRA adapter:

python terminal_chat.py --model-mode base

Run the fine-tuned HEP LoRA adapter, which is the default:

python terminal_chat.py --model-mode adapter

Compare baseline and adapter answers side by side:

python compare_chat.py

compare_chat.py loads both the baseline and adapter model at the same time, so it needs substantially more memory than single-model chat.

Inside the chat, use /clear to reset history and /exit to quit.

If Hugging Face returns 429 Too Many Requests, wait a few minutes and rerun:

bash

python download_models.py
python terminal_chat.py

The download helper uses one worker and retries, which is friendlier to the Hub from shared clusters.

Run Locally

bash

git clone https://huggingface.co/ho22joshua/hep-chat-entrypoint
cd hep-chat-entrypoint
pip install -r requirements.txt
python terminal_chat.py

Terminal chat works best with a CUDA GPU. On an Apple Silicon MacBook, use the PyTorch MPS backend:

bash

python -m pip install -r requirements.txt
python terminal_chat.py --device mps

DEVICE=auto and --device auto also detect MPS automatically when CUDA is not available. CPU-only mode is possible but very slow:

ALLOW_CPU=1 python app.py

To launch the Gradio app instead of terminal chat:

python app.py

Terminal Chat

For clusters, the terminal interface is often simpler than running a web server:

python terminal_chat.py

Commands inside the chat:

/clear: reset conversation history
/exit: quit

CPU-only fallback:

python terminal_chat.py --allow-cpu

Explicit Apple Silicon GPU mode:

python terminal_chat.py --device mps

Fast Apple Silicon GPU mode:

python terminal_chat.py --device mps --fast

Useful generation knobs:

python terminal_chat.py --max-new-tokens 200 --temperature 0.0 --no-history

Baseline-only mode:

python terminal_chat.py --model-mode base

Side-by-side comparison mode:

python compare_chat.py --max-new-tokens 200 --temperature 0.0

Predownload the model cache without starting a chat:

python download_models.py

Predownload every configured model:

python download_models.py --all

Cluster Setup

On shared clusters, put the Hugging Face cache on scratch or project storage before starting the app:

bash

export HF_HOME=/path/to/scratch/hf_home
export HF_HUB_CACHE=$HF_HOME/hub
python app.py

Replace /path/to/scratch/hf_home with a real writable directory. Do not use the placeholder path literally.

On Perlmutter, a typical setup is:

bash

mkdir -p $SCRATCH/hf_home
export HF_HOME=$SCRATCH/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

python terminal_chat.py

If $SCRATCH is not set, use a writable project directory instead:

bash

mkdir -p /global/cfs/projectdirs/atlas/joshua/hf_home
export HF_HOME=/global/cfs/projectdirs/atlas/joshua/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

python terminal_chat.py

Check that the cache path is valid before loading the model:

bash

echo $HF_HOME
test -w "$HF_HOME" && echo writable

If you need the Gradio server to listen on all interfaces for tunneling:

GRADIO_SERVER_NAME=0.0.0.0 python app.py

For faster Gradio defaults:

FAST_MODE=1 DEVICE=mps python app.py

Do not load the 7B model on login nodes. Start an interactive GPU allocation or batch job first.

Storage and Memory

Run diagnostics before loading the model:

python diagnose_system.py

If Torch import is broken and you only want Python, disk, RAM, and package metadata:

python diagnose_system.py --skip-torch

To test requirement logic without changing the machine, use simulation flags:

bash

python diagnose_system.py --simulate-ram-gb 16 --simulate-disk-free-gb 20
python diagnose_system.py --simulate-ram-gb 32 --simulate-disk-free-gb 25 --simulate-mps
python diagnose_system.py --simulate-ram-gb 64 --simulate-disk-free-gb 50 --simulate-cuda-vram-gb 24

The diagnostics output is JSON so code agents can parse it. Important fields:

summary.can_try_mps: this Python environment can try Apple Silicon GPU
summary.can_try_cuda: this Python environment can try NVIDIA CUDA GPU
summary.meets_minimum_disk: cache disk has enough free space to try one 7B model
summary.meets_recommended_disk: cache disk has comfortable free space
summary.meets_minimum_ram: system RAM meets the minimum for 7B inference
summary.meets_recommended_ram: system RAM is in the comfortable range
disk.cache_dir: where Hugging Face model files are expected
disk.cache_size_gb: current size of that model cache
packages: installed versions of Torch, Transformers, PEFT, NumPy, and related packages
simulation.enabled: whether any reported resource values were synthetic
memory.detected_total_ram_gb and disk.detected_free_gb: real values when simulation is used

Minimum requirements for the default 7B model:

Disk: 25 GB free for one clean model cache and Python environment
Apple Silicon Mac: 32 GB unified memory
NVIDIA CUDA GPU: 16 GB VRAM
CPU-only: 32 GB system RAM, but expect very slow inference
Python: 3.10+ with the packages in requirements.txt

Recommended requirements:

Disk: 50 GB free for cache growth, partial downloads, and future model variants
Apple Silicon Mac: 64 GB unified memory
NVIDIA CUDA GPU: 24 GB VRAM
CPU-only: 64 GB system RAM, still slow
On clusters: 75 GB+ in scratch/cache storage if trying multiple models

Why disk usage is this high:

Default 7B base model: about 15 GB
LoRA adapter: about 100 MB
Python environment: commonly 2-8 GB on Mac, 5-15 GB with CUDA packages
Hugging Face cache can keep partial downloads, locks, snapshots, and old revisions

Why memory usage is this high:

The current backend runs the default 7B model in fp16/bf16-style Transformers weights.
The base model is roughly 14 GB before runtime overhead.
KV cache grows with prompt length and generated tokens.
Apple Silicon uses unified memory, shared by the OS, Python, model weights, and GPU execution.

Performance guidance:

Use --fast for local Mac runs.
Keep terminal chat open across multiple questions to avoid reloading weights.
Reduce --max-new-tokens for faster replies and lower memory pressure.
Avoid compare_chat.py on memory-constrained machines; it loads two models.
On Apple Silicon, prefer an arm64 Python environment. If diagnostics show platform.machine as x86_64, recreate the environment with an arm64 Python/Conda install for best local GPU performance.

Fixing Install Issues

If Gradio fails with ImportError: cannot import name 'HfFolder' from 'huggingface_hub', reinstall with the pinned requirements:

pip install -r requirements.txt

or explicitly:

pip install "huggingface_hub<1.0" "gradio==4.44.1"

Turn This Into A Hugging Face Space

Hugging Face currently requires a paid account feature for Gradio Spaces on cpu-basic, and this model needs GPU hardware for actual chat anyway.

After enabling Space hosting/GPU hardware, create a Gradio Space and upload these files:

bash

hf repo create ho22joshua/hep-chat --type space --sdk gradio --public --exist-ok
hf upload ho22joshua/hep-chat . . --repo-type space

Then set the Space hardware to a GPU such as l4x1, a10g-small, or t4-medium.

OpenAI-Compatible Endpoint

For a production-style endpoint, use vLLM:

bash

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --enable-lora \
  --lora-modules hep=ho22joshua/hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16

Call it with:

bash

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hep",
    "messages": [
      {"role": "user", "content": "For H to AA to photons, what Standard Model backgrounds should be considered?"}
    ],
    "max_tokens": 300
  }'