This repository contains the raw translation tables for tha package
fast_psq.
Please refer to the GitHub for more information.
The following is a brief example for using the tables.
The following is an example indexing command.
1python -m fast_psq.index \
2--doc_file irds:neuclir/1/zh/trec-2022 \
3--lang zh \
4--psq_file hltcoe/psq_translation_tables:zh.table.dict.gz \
5--min_translation_prob 0.00010 \
6--max_translation_alternatives 64 \
7--max_translation_cdf 0.99 \
8--docid doc_id \
9--title title \
10--body text \
11--min_translation_prob 1e-4 \
12--max_translation_alternatives 64 \
13--output_dir ./indexes/neuclir-zh.f32/ \
14--compression \
15--nworkers 64
The following command is an example for searching.
1python -m fast_psq.search \
2--query_source irds:neuclir/1/zh/trec-2022 \
3--query_field title \
4--index_dir ./indexes/neuclir-zh.f32/ \
5--qrels irds:neuclir/1/zh/trec-2022 \
6--query_lang en \
7--output_file ./neuclir-zh.en.title.f32.trec
1@article{psq-repro,
2 title = {Efficiency-Effectiveness Tradeoff of Probabilistic Structured Queries for Cross-Language Information Retrieval},
3 author = {Eugene Yang and Suraj Nair and Dawn Lawrie and James Mayfield and Douglas W. Oard and Kevin Duh},
4 journal = {arXiv preprint arXiv},
5 year = {2024}
6}
