Fast web scraper in Rust. ~10 MB/s sustained.
cargo build --release
./target/release/turbo_crawl
Output: /workspace/crawl.jsonl (JSONL format with url, text, tokens_est)
watchdog.sh - Auto-kills scraper at 145GB disk usage
raw_relay.py - Flask endpoint for raw binary file uploads
