dev-knowledge-bullshit-detector-v0.1 estimates how much durable, reusable technical
knowledge a Polish or English comment contains. It is intended for ranking and filtering
technical text for later retrieval. Despite the informal repository name, this is not a
fact-checker and does not determine whether a statement is true.
Output scale
The model produces an ordered distribution over five retrieval-value classes:
Class
Meaning
0
Noise, social, or administrative text with no technical knowledge
1
Technical vocabulary or transient status without reusable knowledge
2
Potentially useful but partial, borderline, or context-dependent technical content
3
A standalone, durable unit of technical knowledge
4
A detailed diagnosis, procedure, configuration, solution, or reasoned decision
Two continuous values can be derived from the distribution:
usefulness_score: the expected class value divided by 4, in the range 0–1;
probability_retrieval_value_ge_3: a separately calibrated probability that the text is a
standalone, reusable technical unit (class 3 or 4).
The second value is usually the more convenient operational score for filtering. Its threshold
should be selected for the target corpus rather than assumed to be 0.5.
Architecture
This model is a fully fine-tuned
sdadas/polish-roberta-base-8k
encoder with a five-logit sequence-classification head.
Property
Value
Architecture
RoBERTa for sequence classification
Parameters
190,297,349
Transformer layers
12
Hidden size
768
Attention heads
12
Feed-forward size
3,072
Vocabulary size
128,064
Classification labels
5 ordered classes
Base encoder context capacity
8,192 tokens
Fine-tuning and recommended inference length
256 tokens
Weight format
Safetensors, float32
Although the base encoder was pretrained primarily for Polish, this checkpoint was fine-tuned and
evaluated on both Polish and English text.
Training data and procedure
The source material consisted of technical comments available under free licenses. English
comments were translated into Polish, and the resulting language pairs were annotated for
retrieval usefulness by Qwen3.5-27B. Original and translated versions of a comment were always
kept in the same data split to prevent cross-language leakage.
The training split contained 10,762 examples: 5,381 Polish and 5,381 English. Separate
validation and calibration splits contained 1,793 and 1,791 text instances drawn from both
languages, respectively.
The held-out test set consisted of 1,794 source pairs evaluated independently in Polish and
English (3,588 text instances in total).
All model parameters were fine-tuned for five epochs, with the checkpoint from epoch 3 selected
on validation PR AUC for the retrieval_value >= 3 boundary. The objective combined multiclass
cross-entropy with a ranked probability score weighted by 0.25, so mistakes between distant
classes were penalized more strongly. Training used balanced class weights, batch size 16, a
learning rate of 2e-5, weight decay 0.01, 10% warmup, mixed precision, and seed 42. Temperature
scaling was fitted on a dedicated calibration split, followed by monotonic Platt calibration for
the class-3-or-higher boundary.
Evaluation
These results measure agreement with Qwen3.5-27B annotations on the test split. The test texts
come from the same domain and were labeled with the same annotation procedure as the training
data. Therefore, this evaluation can only indicate how closely the fine-tuned encoder-only model
reproduces Qwen3.5-27B's annotations in-domain. It is not an independent expert evaluation and
does not demonstrate factual correctness, general reasoning quality, or performance in other
domains.
The following metrics were measured after calibration on Polish and English views of the same
Qwen-annotated test pairs:
Language
Examples
Accuracy
Within ±1
MAE
Spearman
QWK
PR AUC (value >= 3)
Polish
1,794
0.777
0.921
0.427
0.850
0.847
0.886
English
1,794
0.756
0.909
0.438
0.846
0.838
0.879
Combined
3,588
0.767
0.915
0.433
0.848
0.842
0.882
Synthetic examples
All texts below were written specifically for this model card and are not training or evaluation
examples. Values were produced locally by this exact checkpoint. P(value >= 3) includes the
saved temperature and Platt calibration. The examples were deliberately kept at comparable
lengths so that score differences cannot be explained merely by short versus long inputs.
Polish
Text
Predicted class
Usefulness score
P(value >= 3)
Sprawdziłem konfigurację połączenia, wersję biblioteki i wyniki wszystkich testów integracyjnych. Nie udało mi się odtworzyć problemu na najnowszej gałęzi, więc zamykam zgłoszenie. Jeśli błąd pojawi się ponownie, proszę utworzyć nowe zgłoszenie z aktualnymi logami.
0
0.032
0.009
Przejrzałem ostatnie zmiany w module pamięci podręcznej, porównałem wyniki testów i ponownie uruchomiłem nieudane zadanie. Teraz wszystko wygląda poprawnie, nie widzę już błędów w logach, a gałąź jest gotowa do scalenia po akceptacji recenzenta.
1
0.243
0.028
Problem prawdopodobnie dotyczy pamięci podręcznej albo kolejności inicjalizacji usług. Warto wyczyścić stare wpisy, ponownie uruchomić proces i sprawdzić konfigurację połączenia. Jeśli to nie pomoże, potrzebne będą pełniejsze logi oraz informacje o wersji używanego środowiska.
1
0.395
0.199
Po zmianie schematu stare wpisy pamięci podręcznej nie zawierają pola version, dlatego deserializator odrzuca je podczas startu. Dodaj migrację ustawiającą wartość domyślną albo unieważnij wpisy z poprzednim numerem schematu przed wdrożeniem nowej wersji aplikacji.
3
0.614
0.621
Nagłówek Cache-Control: no-cache nie zabrania zapisania odpowiedzi w pamięci podręcznej, lecz wymaga ponownej walidacji przed jej użyciem. Gdy odpowiedź w ogóle nie może zostać zapisana, należy zastosować no-store; private ogranicza natomiast przechowywanie do prywatnej pamięci klienta.
3
0.788
0.897
English
Text
Predicted class
Usefulness score
P(value >= 3)
I checked the connection configuration, library version, and every integration test result. I cannot reproduce the problem on the latest branch, so I am closing the report. If it happens again, please open a new report with current logs.
0
0.038
0.010
I reviewed the recent cache changes, compared the test results, and reran the failed workflow. Everything looks correct now, no errors remain in the logs, and the branch is ready to merge after another reviewer gives final approval.
1
0.237
0.032
The issue probably involves caching or service initialization order. Try deleting old entries, restarting the process, and checking the connection configuration. If that does not help, we will need complete logs and details about the exact environment and version.
1
0.355
0.145
Old cache entries lack the version field introduced by the new schema, so deserialization fails during startup. Add a migration that supplies a default value, or invalidate entries carrying the previous schema number before deploying the updated application.
3
0.684
0.779
Cache-Control: no-cache does not prohibit storing a response; it requires successful revalidation before reuse. When a response must never be stored, use no-store instead. The private directive permits storage only in a private cache and prevents reuse by shared intermediary caches.
3
0.764
0.879
Usage
Install the runtime dependencies:
pip install torch transformers huggingface_hub
The repository includes predict.py, which applies both calibration stages and returns the class
distribution, expected retrieval value, normalized usefulness score, and calibrated operational
probability:
bash
1python predict.py \2 --model jansowa/dev-knowledge-bullshit-detector-v0.1 \3"The cache entry is stale because the invalidation callback runs before the transaction commits."
For the raw five-class output, the standard Transformers API is sufficient:
