Repository-scale Solidity vulnerability localization with a 16.95M late-interaction retriever.
Parameters
Model
Hardware
Weights
Status
[!CAUTION]
This is a research checkpoint, not an autonomous auditor or a safety certificate. On the locked unseen-family test, each trained run placed 11 or 12 of 63 known affected functions in the top 10. The frozen base placed 7. The improvement is repeatable, but absolute recall remains low.
The downloadable checkpoint is a 16.95M-parameter LateOn-Code-edge retriever. It contains no Gemma weights. Historical system experiments used a frozen Gemma reviewer only after retrieval. Ignore Hugging Face's generic SentenceTransformer(...) example; use MultiVectorEncoder(...) and MeanMaxSim below.
Abstract
This work studies whether an ultra-small late-interaction model can learn repository-scale vulnerability localization under a 4 GB GPU constraint. The retriever receives a general security mechanism and ranks every implemented function in a Solidity repository. A separate reasoning model can review the retrieved shortlist, while executable tools remain responsible for confirmation.
The released 16.95M-parameter checkpoint was fully fine-tuned from lightonai/LateOn-Code-edge with repository-listwise supervision and same-repository distractors. Evaluation was frozen across three seeds and conducted on unseen protocol families. Mean Recall@10 increased from 11.1% for the frozen base to 18.0%, and median affected-function rank improved from 98 to 43.7. These results support the narrow hypothesis that small specialized retrievers can improve review prioritization; they do not establish practical autonomous auditing capability.
Optional and external; historical experiments used frozen google/gemma-4-E4B-it, whose weights are not included
Training hardware
RTX 3050 Laptop GPU with 4 GB VRAM; selected run peaked at 0.437 GiB
Research status
Public preview; not a vulnerability detector, auditor, or safety certificate
Research question
Auditors do not choose one vulnerable function from four easy options. They decide where to start in a repository with hundreds of functions.
We test one hypothesis: can an ultra-small model trained on whole-repository rankings prioritize known affected functions better than its frozen code-retrieval base on previously unseen protocol families?
The locked experiment produced three consistent observations:
The median affected-function rank moved from 98 to 42–45 in repositories with a median of 443 candidate functions.
The number of affected functions in the top 10 moved from 7 of 63 to 11, 11, and 12 of 63 across three runs.
Every run improved on both held-out sets.
The checkpoint met the predeclared improvement criterion but remained far below the stronger competitiveness thresholds. It is released to make the positive and negative results reproducible and to support follow-up work on data quality, repository context, and executable verification.
Try the localizer
pip install "sentence-transformers>=6.0.0" torch
python
1import torch
2from sentence_transformers import MultiVectorEncoder
34model = MultiVectorEncoder("starknet-ai/LateOn-Code-edge-EVMScout-17M")56query ="Find reentrancy or callback paths where an external call happens before critical state is updated."7functions =[8"""function withdraw(uint256 amount) external {
9 require(balance[msg.sender] >= amount);
10 (bool ok,) = msg.sender.call{value: amount}("");
11 require(ok);
12 balance[msg.sender] -= amount;
13 }""",14"""function balanceOf(address user) external view returns (uint256) {
15 return balance[user];
16 }""",17]1819query_tokens = model.encode_query([query], convert_to_numpy=False)[0]20document_tokens = model.encode_document(functions, convert_to_numpy=False)2122# EVMScout 17M was trained with mean MaxSim.23scores = torch.stack([24(query_tokens @ document.T).max(dim=1).values.mean()25for document in document_tokens
26])2728print(scores.argsort(descending=True).tolist())29# The state-after-call withdraw path should rank first.
Scan a repository
The repository includes the scanner and the 12 security queries used in the demo:
The scanner extracts each function and its nearby calls, modifiers, inheritance, and shared state. It scores each risk separately, then merges the rankings. Compare scores only within the same query.
Every raw retrieval result begins as LEAD. A model score alone is not evidence that a vulnerability exists.
Training procedure
Data construction
36,618 implemented Solidity functions from 99 projects.
208 training findings.
57 repository-disjoint validation findings.
63 unseen-protocol-family test findings.
27 separately sealed regression findings.
Conservative grouping keeps versions, forks, upgrades, and related audits in the same protocol family.
Unreported functions are treated as distractors, never certified-safe negatives.
The corpus is not included in this repository.
The initial source index was reconstructed from a pinned FORGE-Curated snapshot. Source-level provenance and redistribution rights remain under review; see License and provenance.
Objective and optimization
Setting
Value
Objective
Exact multi-positive repository-listwise loss
Candidate curriculum
7 → 15 → 31 same-repository distractors
Precision
BF16
Query length
96 tokens
Document length
512 tokens
Optimizer
AdamW
Learning rate
2e-5
Weight decay
0.01
Schedule
5% warmup, linear decay
Gradient clipping
1.0
Final seeds
20260823, 20260824, 20260825
Selected checkpoint
seed 20260825, validation-selected step 150
A 149M-parameter LateOn-Code checkpoint also fit on the 4 GB GPU at 3.364 GiB, but lost the frozen validation bakeoff. The smaller 17M model was selected on evidence, not size or novelty.
Evaluation protocol
Three seeds were frozen before the unseen-family test was opened.
The primary evaluation ranks all implemented functions in each held-out repository rather than selecting among a small set of sampled candidates. Repository versions, forks, upgrades, and related audits are grouped into protocol families to reduce leakage. The reported test contains 63 findings from nine unseen families; a separate sealed regression set contains 27 findings.
Results
Unseen protocol families · 63 findings
MRR
Recall@5
Recall@10
Recall@20
Median rank
Frozen LateOn-Code-edge
0.051
3.2%
11.1%
20.6%
98
Trained mean · 3 seeds
0.093 ± 0.005
13.2%
18.0%
28.0%
43.7
3-seed score ensemble
0.102
12.7%
17.5%
30.2%
42
Sealed regression · 27 findings
MRR
Recall@5
Recall@10
Recall@20
Median rank
Frozen LateOn-Code-edge
0.060
11.1%
14.8%
22.2%
73
Trained mean · 3 seeds
0.119 ± 0.028
14.8%
29.6%
40.7%
30.3
Protocol-family-clustered 95% bootstrap interval for test ΔMRR: [0.0002, 0.2282].
Exact family sign-flip test: p=0.230 because the test contains only nine independent families.
All three seeds improved Recall@10 on both held-out sets.
The predeclared +5 percentage-point improvement gate passed.
The stronger 60% Recall@10 / 80% Recall@20 competitiveness gates failed.
The model passed the predeclared improvement gate. It failed the stronger competitiveness thresholds. The direction of improvement is repeatable across the three runs; the absolute recall remains insufficient for independent security review.
Structural-context ablation
Validation representation
MRR
Recall@5
Recall@10
Recall@20
Raw Solidity function
0.143
17.5%
33.3%
56.1%
Bounded security slice
0.158
26.3%
38.6%
54.4%
The slice improved early ranking, especially Recall@5. It is a deterministic no-build approximation—not Slither and not a complete inter-contract graph.
Post-freeze external pilot
The fixed generic mechanism scan was also run on nine findings from four official ReEVMBench task repositories after the main result was frozen.
Nine-finding pilot
MRR
Recall@10
Recall@20
Median rank
Frozen base
0.472
66.7%
88.9%
3
LateOn-Code-edge EVMScout 17M
0.505
88.9%
100.0%
2
Five findings improved, two tied, and two worsened. No scored positive exactly matched training code. This is not an official ReEVMBench Detect score: the sample is tiny, gold was used only for post-hoc localization scoring, and the task repositories lacked the base_commit objects declared by their benchmark configs. Full predictions are preserved in reevm_external_summary.json.
Negative result: hard-negative collapse
One model-mined hard-negative continuation collapsed validation Recall@10 from 59.6% at the starting checkpoint to 47.4% after 25 steps, 26.3% after 50, and 19.3% after 100. The selector retained step 1.
This result suggests that automatically treating highly ranked, unlabeled functions as negatives can suppress useful evidence. Unreported code is therefore represented as a distractor rather than certified-safe supervision.
Intended use
Supported research uses:
Prioritize functions and connected code neighborhoods for an authorized Solidity review.
Supply a shortlist to a human auditor or a second-stage reasoning model.
Compare retrieval behavior across general vulnerability mechanisms.
Study repository-aware, token-level code retrieval under small-GPU constraints.
Out-of-scope uses:
Certifying a contract as safe.
Autonomous vulnerability discovery.
Severity classification.
Calibrated exploit probability.
Running tests against systems, contracts, networks, or funds without authorization.
Limitations
On the locked test, 82% of known positives remained outside the first ten results.
Generic mechanisms can converge on the same suspicious function.
Upstream file/function mappings remain subject to manual verification.
Second-stage hypothesis text can be plausible but wrong; executable evidence remains authoritative.
The external pilot is too small for a benchmark or SOTA claim.
Future work
Future checkpoints will be considered competitive only if a new locked evaluation reaches:
at least 60% Recall@10;
at least 80% Recall@20;
the same direction across three runs and unseen protocol families;
reproducible Foundry tests for a meaningful share of the shortlisted leads.
Failure cases are particularly useful. Open a Community discussion with the repository commit, risk query, expected function, and returned rank so that future experiments can target a concrete retrieval failure.
License and provenance
The base checkpoint is Apache-2.0. The EVM-Scout training examples were reconstructed from public security reports and source snapshots whose per-source terms are still being reviewed. The corpus is not distributed here.
This is a public research preview while the derived-weight and per-source provenance review remains open. Public availability is not a representation that the fine-tuning corpus is redistributable, and the corpus is not included. No public or commercial redistribution license is granted for the fine-tuned weights by this preview. See LICENSE.md.
Reproducibility and evidence
locked_summary.json — complete locked predictions and statistics.
solidity_graph.py — deterministic function and security-slice extractor.
solidity_parser.py — dependency-free conservative Solidity function parser.
Acknowledgements
Built on lightonai/LateOn-Code-edge, Sentence Transformers MultiVectorEncoder, and the broader ColBERT/late-interaction ecosystem. Historical system experiments used google/gemma-4-E4B-it as a separate frozen reviewer; it is not part of this checkpoint.
Citation
bibtex
1@misc{espejel2026lateonevmscout,
2 title = {LateOn-Code-edge EVMScout 17M: Repository-Wide Solidity Security Retrieval on a 4GB GPU},
3 author = {Omar Espejel},
4 year = {2026},
5 howpublished = {Hugging Face model repository},
6 url = {https://huggingface.co/starknet-ai/LateOn-Code-edge-EVMScout-17M}
7}
