You can deploy this model and start earning money today!

datamatters24/
research-document-archive

Name: research-document-archive
Availability: InStock

government-documents

nlp

named-entity-recognition

Views

No views yet

Research Document Archive

234,630 declassified U.S. government documents processed through a 13-step ML pipeline. 3.2 million pages OCR'd, 31 million named entities extracted and linked, 288 topic clusters identified.

Live platform: tanglewoodapp.com

Collections

Collection	Documents	Pages	Size
House Resolutions	181,092	2,719,832	34.2 GB
JFK Assassination Records	35,979	241,860	22.5 GB
CIA Stargate Program	13,937	100,056	5.4 GB
CIA MKUltra	1,936	64,244	3.4 GB
CIA Declassified	1,605	29,744	2.4 GB
Lincoln Archives	21	9,330	962.9 MB

ML Pipeline (13 Steps)

Document ingestion and format normalization
OCR with Tesseract + post-correction
Classification stamp detection (SECRET, CONFIDENTIAL, UNCLASSIFIED, etc.)
Redaction detection and boundary mapping
Named entity recognition (people, organizations, locations, dates)
Entity disambiguation and cross-document linking
Relationship extraction
Topic modeling (LDA + BERTopic)
Timeline event extraction
Network graph construction
Sentiment and tone analysis
Document similarity clustering
Index building for search and retrieval

Classification Stamps Detected

Stamp	Count
UNCLASSIFIED	16,501
SECRET	13,736
CLASSIFIED	10,730
EXEMPT	6,739
CONFIDENTIAL	5,554
RESTRICTED	4,722

Key Statistics

31M named entities extracted
2.9M entity cross-document links
59,830 redactions detected and mapped
288 topic clusters identified
6 document collections spanning 1860s–2000s

Usage

python

from datasets import load_dataset

ds = load_dataset("datamatters24/research-document-archive")

# Filter by collection
jfk = ds.filter(lambda x: x["collection"] == "jfk_assassination")

# Search by entity
cia_docs = ds.filter(lambda x: "CIA" in x["entities"])

Data Sources

All documents are public record obtained from:

National Archives (NARA)
CIA FOIA Reading Room
Congress.gov
Library of Congress

Citation

bibtex

@misc{rubin2026researcharchive,
  author = {Rubin, Theodore},
  title = {Research Document Archive: ML Pipeline for Declassified U.S. Government Documents},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/datasets/datamatters24/research-document-archive}
}