You can deploy this model and start earning money today!

poolside-laguna-hackathon/
laguna-eval-experiments

Views

No views yet

Poolside AI Hackathon Lab

Prime Lab workspace for verifiers environments, hosted evals, and model benchmarks.

Start here

I want to…	Go to
Browse hackathon rollout datasets	poolside-laguna-hackathon/datasets
Clone the full lab (configs, sweep, compare)	poolside-laguna-hackathon/laguna-eval-experiments
Run or reproduce the benchmark	QUICKSTART · Choose your path
Documentation index	environments/py_bug_trace/docs/README.md
See sprint methodology / tooling	environments/py_bug_trace/docs/SPRINT_NOTES.md

Primary environment: py-bug-trace

Python output-prediction eval for Laguna XS.2 — trace subtly broken code and predict exact stdout (Levels 1–2 shipped; Levels 3–5 on the roadmap). What each level means: Difficulty levels (L1 = Python gotchas, L2 = async/concurrency, L3+ = fix-with-tests).

Hugging Face (py_bug_trace-only upload): QUICKSTART · setup.

Resource	Path
README hub	environments/py_bug_trace/README.md
Docs index	environments/py_bug_trace/docs/README.md
HF lab bundle	poolside-laguna-hackathon/laguna-eval-experiments
HF rollout datasets (hackathon org)	poolside-laguna-hackathon/datasets
Published HF rollout datasets (table)	docs/publish.md
Sprint 2 bank notes	environments/py_bug_trace/docs/SPRINT_NOTES.md
Local eval config	configs/eval/py-bug-trace-eval.toml
Hosted Level 1 config	configs/eval/py-bug-trace-level-1-hosted-eval.toml
Hosted Level 2 config	configs/eval/py-bug-trace-level-2-hosted-eval.toml
Hosted Level 3 config	configs/eval/py-bug-trace-level-3-hosted-eval.toml
Multi-model sweep	configs/sweep/py-bug-trace-sweep.yaml (`hosted_env_id` and `hf_publish` use `{n}` over `levels`)
Endpoint aliases	configs/endpoints.toml

Quick start

bash

# Local smoke (Laguna via Prime Inference)
prime eval run configs/eval/py-bug-trace-eval.toml

# Hosted Level 1
prime eval run configs/eval/py-bug-trace-level-1-hosted-eval.toml --hosted --follow

# Hosted Level 2
prime eval run configs/eval/py-bug-trace-level-2-hosted-eval.toml --hosted --follow

# Hosted Level 3
prime eval run configs/eval/py-bug-trace-level-3-hosted-eval.toml --hosted --follow

# Multi-model × level matrix (preflight: environments/py_bug_trace/docs/workflow.md)
cd environments/py_bug_trace && uv pip install -e ".[tools]"
python environments/py_bug_trace/tools/sweep.py --status
python environments/py_bug_trace/tools/sweep.py

# Compare sweep results vs Laguna baseline (auto-writes stats + explorer reports)
python environments/py_bug_trace/tools/sweep.py --compare

# Regenerate reports only (stats digest, fragments, explorer, LLM narrative)
python environments/py_bug_trace/tools/sweep.py --narrative-only
python environments/py_bug_trace/tools/sweep.py --narrative-only --narrative           # force analyst re-run
python environments/py_bug_trace/tools/sweep.py --narrative-only --narrative-model ID  # analyst endpoint override

Reporting outputs (canonical copies and timestamped run history committed to git — reader's guide: reports/README.md): reports/matrix/stats.md (inverted-pyramid digest), reports/fragments/ (paste-ready leaderboard/findings/one-pager for READMEs, HF cards, blogs), reports/explorer/latest.html (self-contained interactive playground), and reports/narrative/report.md + writeup.md (LLM analyst narrative in two styles, with liftable *.body.md prose).

Publish eval results

Two separate destinations — do not mix them:

Destination	What you get	Auth	Docs
Prime Evals + leaderboard	Aggregate scores, visibility on Prime	`prime login`, `PRIME_API_KEY` for public/leaderboard	docs/publish.md
Hugging Face Hub	Rollout datasets (prompts, completions, rewards) + optional lab bundle	`HF_TOKEN` or `HF_API_TOKEN`	docs/publish.md

The hosted sweep uploads to Prime automatically but does not publish to Hugging Face. Prime leaderboard publish (--publish-only, sweep PATCH) and HF publish (--hf-publish-rollouts, --hf-upload-experiments-bundle) are independent.

Hugging Face — auth

huggingface_hub reads HF_TOKEN. If your secret is named HF_API_TOKEN, map it first:

bash

export HF_TOKEN="${HF_API_TOKEN:-$HF_TOKEN}"
hf auth whoami   # should show neomatrix369

Or: huggingface-cli login / hf auth login.

Hugging Face — publish saved artifacts (no eval re-run)

Command	What it does
`--hf-publish-rollouts`	Push rollout datasets from `sweep_state.json` / `outputs/evals/` (idempotent)
`--hf-upload-experiments-bundle`	Sync the lab folder to `hf_publish.lab_bundle.repo` via resumable `hf upload-large-folder` (idempotent)

Config: configs/sweep/py-bug-trace-sweep.yaml. Details: docs/publish.md.

bash

export HF_TOKEN="${HF_API_TOKEN:-$HF_TOKEN}"
python environments/py_bug_trace/tools/sweep.py --hf-publish-rollouts --dry-run
python environments/py_bug_trace/tools/sweep.py --hf-publish-rollouts              # skip unchanged rollouts
python environments/py_bug_trace/tools/sweep.py --hf-upload-experiments-bundle     # skip unchanged lab bundle
python environments/py_bug_trace/tools/sweep.py --hf-publish-rollouts --hf-upload-experiments-bundle

Run evals separately first (python environments/py_bug_trace/tools/sweep.py for hosted matrix, or prime eval run … -s locally). Use --force-all to re-publish after source changes.

Logs land in environments/py_bug_trace/sweep_logs/ as hf_publish_rollouts_*.log and hf_publish_lab_bundle_*.log (legacy hf_export_* names are renamed automatically).

Hugging Face — published datasets (May 2026 sweep)

Full table: docs/publish.md.

Browse: poolside-laguna-hackathon/datasets · neomatrix369 py-bug-trace

Dataset	Orgs (neomatrix369 · poolside)	Model × level
py-bug-trace-laguna-xs-2-l1-rollouts	neo · poolside	Laguna XS.2 × L1
py-bug-trace-laguna-xs-2-l2-rollouts	neo · poolside	Laguna XS.2 × L2
py-bug-trace-qwen3-5-35b-a3b-l1-rollouts	neo · poolside	Qwen3.5-35B × L1
py-bug-trace-qwen3-5-35b-a3b-l2-rollouts	neo · poolside	Qwen3.5-35B × L2
py-bug-trace-qwen3-6-35b-a3b-l1-rollouts	neo · poolside	Qwen3.6-35B × L1
py-bug-trace-qwen3-6-35b-a3b-l2-rollouts	neo · poolside	Qwen3.6-35B × L2
py-bug-trace-qwen3-6-27b-l1-rollouts	neo · poolside	Qwen3.6-27B × L1
py-bug-trace-qwen3-6-27b-l2-rollouts	neo · poolside	Qwen3.6-27B × L2
py-bug-trace-gpt-5-mini-l1-rollouts	neo · poolside	GPT-5-mini × L1
py-bug-trace-gpt-5-mini-l2-rollouts	neo · poolside	GPT-5-mini × L2
py-bug-trace-gpt-5-nano-l1-rollouts	neo · poolside	GPT-5-nano × L1
py-bug-trace-gpt-5-nano-l2-rollouts	neo · poolside	GPT-5-nano × L2

Additional local-only exports: py-bug-trace-gpt-4-1-mini-l{1,2}-rollouts, py-bug-trace-laguna-m-1-free-l1-rollouts, reverse-text-gpt-5-4-nano-rollouts.

Workspace layout

configs/          eval TOMLs, sweep YAML, endpoint aliases
environments/     verifiers packages (py_bug_trace, reverse_text, …)
.prime/skills/    bundled Lab skills (create/eval/train/merge-to-main, …)

See AGENTS.md and environments/AGENTS.md for Prime/Verifiers conventions.

poolside-laguna-hackathon/laguna-eval-experiments

Poolside AI Hackathon Lab

Start here

Primary environment: py-bug-trace

Quick start

Publish eval results

Hugging Face — auth

Hugging Face — publish saved artifacts (no eval re-run)

Hugging Face — published datasets (May 2026 sweep)

Workspace layout

poolside-laguna-hackathon/
laguna-eval-experiments