LycheeMem BERT-Tiny Memory Reranker v0
This repository provides the optional v0 transformer reranker checkpoint for
LycheeMem semantic memory search. The model scores (query, memory candidate)
pairs and is used as a conservative reranker over a wider memory candidate pool.
The reranker is default-off in LycheeMem. It only changes memory search when the
user installs the optional rerank dependencies, downloads this checkpoint, and
explicitly enables the transformer rerank hook.
Model
1name: lycheemem-bert-tiny-reranker-locomo-v0
2base_model: prajjwal1/bert-tiny
3task: memory evidence reranking
4architecture: AutoModelForSequenceClassification
5runtime: local checkpoint, default-off LycheeMem hook
6version: v0.1.0
Intended Use
Use this checkpoint with LycheeMem's experimental transformer reranker hook:
1pip install "lycheemem[rerank]"
2
3EXPERIMENTAL_TRANSFORMER_RERANK=true
4TRANSFORMER_RERANK_MODEL_PATH=/path/to/lycheemem-bert-tiny-reranker-locomo-v0
5TRANSFORMER_RERANK_MAX_REPLACEMENTS=1
6TRANSFORMER_RERANK_MERGE_MARGIN=0.3
7TRANSFORMER_RERANK_WIDE_TOP_K=50
If dependencies or the local checkpoint are missing, LycheeMem falls back to
baseline memory search.
Training Data
The checkpoint was trained on LoCoMo-derived memory evidence reranking bundles.
Each training example pairs a user question with candidate memory texts and
evidence IDs derived from the LoCoMo benchmark.
The source repository does not include LoCoMo data, generated caches, or training
outputs. Reproduction notes are maintained in the LycheeMem source repository.
Metrics
All metrics below measure evidence retrieval/reranking, not final LLM answer
quality. The primary metric is whether at least one gold evidence item appears
in the returned top-10 candidates (hit@10).
LoCoMo Evidence Retrieval
1System memory backend, 200 QA:
2 baseline: 124/200 = 0.620
3 v0: 130/200 = 0.650
4 added/lost/net: +7/-1/+6
5
6System LanceDB backend, 200 QA:
7 baseline: 124/200 = 0.620
8 v0: 131/200 = 0.655
9 added/lost/net: +8/-1/+7
10
11Full-memory cache, 5 seeds:
12 held added/lost/net: +115/-7/+108
13 added/lost ratio: 16.43
14
15Split checks:
16 interleave held: 466/765 -> 495/765, net +29
17 prefix held: 473/766 -> 501/766, net +28
18 conversation-heldout held: 476/772 -> 504/772, net +28
Candidate Context Probe
Same checkpoint, different candidate text construction:
1single-turn v0: 998/1531 = 0.651862, net +67
2context-candidate v0: 1013/1531 = 0.661659, net +82
Zero-Shot Evidence Selection
1LongMemEval-S cleaned:
2 baseline: 469/500 = 0.938
3 wide: 500/500 = 1.000
4 v0: 484/500 = 0.968
5 added/lost/net: +16/-1/+15
6
7MSC-MemFuse-MC10 turn-level:
8 baseline: 142/299 = 0.475
9 wide: 279/299 = 0.933
10 v0: 152/299 = 0.508
11 added/lost/net: +10/-0/+10
12
13HotpotQA distractor sentence-level:
14 baseline: 6957/7405 = 0.9395
15 wide: 7405/7405 = 1.0000
16 v0: 7076/7405 = 0.9556
17 added/lost/net: +141/-22/+119
These zero-shot fixtures are intended to check whether the LoCoMo-trained v0
checkpoint transfers as an evidence selector. LongMemEval-S and MSC-MemFuse are
memory/dialogue-style settings. HotpotQA is a wiki multi-hop supporting-sentence
setting, so it is a useful but less direct transfer check.
Limitations
- The checkpoint is trained on LoCoMo-derived evidence bundles and may not
generalize to every private memory corpus.
- It assumes relevant evidence is already present in the wide candidate pool.
- It is not an RL policy and does not learn online by itself.
- The MSC-MemFuse fixture uses answer-string matching to infer evidence turns;
this is a conservative heuristic, not original human evidence annotation.
- HotpotQA transfer is positive but has more lost cases than memory-style
fixtures, so dense wiki distractors need monitoring.
- The strongest current accuracy bottleneck appears to be candidate
representation, especially single-turn evidence-boundary cases.
- The hook should remain default-off until a user or deployment explicitly opts
in and monitors diagnostics.
Runtime Behavior
LycheeMem's transformer reranker uses this checkpoint only after baseline memory
search has produced a wider candidate pool. The current v0 policy is
conservative:
1wide_top_k: 50
2max_replacements: 1
3merge_margin: 0.3
4runtime: local checkpoint only
5default behavior: disabled
In plain terms: baseline search retrieves memories first. The reranker only gets
a narrow chance to replace one item in the final top-k when a better evidence
candidate is already present in the wider candidate pool.
Files
Expected checkpoint directory:
1config.json
2model.safetensors
3run_meta.json
4special_tokens_map.json
5tokenizer_config.json
6vocab.txt
SHA256 checksums for the v0.1.0 checkpoint artifact:
1ed54572648824881775812e8b2b0af9be1b720ebdbdf2d1b7c0d976c4ca14c8a config.json
20a328c53b55cbd49aeec0a44e6b9e2d02d09539e6784d93fc515ba815261fca0 model.safetensors
37841bca86e19c72c1cd0f4834efb5c413975ad01ffc5c7020328f4cc62b70536 run_meta.json
4b6d346be366a7d1d48332dbc9fdf3bf8960b5d879522b7799ddba59e76237ee3 special_tokens_map.json
5e711904cac23112776b678356ccf702cf934babaa01125f698ac43bf9ad38e73 tokenizer_config.json
607eced375cec144d27c900241f3e339478dec958f92fddbc551f295c992038a3 vocab.txt
Citation and Scope
This checkpoint is part of LycheeMem's optional memory retrieval research path.
It is not an RL policy and does not learn online by itself. Online feedback and
personalization are handled by separate experimental components.
