A roberta-base multi-label classifier that assigns app-store reviews to seven issue classes. It is Stage 1 of the IssueSpec pipeline, which converts app reviews into typed, tracker-ingestible issue specifications.

Classes: bug_report, feature_request, performance, usability, compatibility, praise, other.

Triage of app-store review text into issue classes, as the first stage of a review-to-issue pipeline. It is a research artifact, not a production classifier.

Fine-tuned from roberta-base on RRGen reviews whose labels were repaired by a verified-anchor procedure: an LLM annotator labels the corpus, confident learning flags likely errors, and a 5,230-review human-verified anchor decides the corrections. Training used 3 epochs, learning rate 2e-5, batch size 16, max sequence length 256, seed 42, under a stratified per-class cap of 15,000 rows.

Against a 490-review expert gold standard (489 scorable):

On a held-out subset, kappa is 0.616. That figure is replay-dependent and indistinguishable from the full-set value; see the limitations below.

Read these before using the model. Several are the reason the accompanying paper reports negative results.

The compatibility class rests on augmented data and carries a trivial cue. Of the 310 compatibility rows the model trains on, 300 are augmentation: 200 template-generated synthetic reviews and 100 keyword-mined corpus rows relabelled. The corpus is lowercased and lemmatised throughout, and not one of its 215,583 rows contains an uppercase character, while 194 of the 200 synthetic rows do. This model is a cased RoBERTa, so capitalisation is a perfect marker for the synthetic class. We withdraw the internal test F1 of 0.74 for this class. The gold-standard F1 of 0.83 is not affected by the cue, because the gold is lowercase corpus text, but it rests on the 100 mined rows, which are verbatim duplicates of corpus reviews.

The gold standard is coupled to this model. Annotators were shown the model's predicted label and asked to accept or overturn it, and the 490-review sample was stratified over the model's predicted classes. Competing classifiers never received that benefit of the doubt. A free-labelling gold would settle it and has not been run.

Held-out kappa is not evidence of the strength it appears to be. The 0.616 figure comes from replaying the training split rather than from a recorded manifest. Under a different shuffle seed or class-iteration order the value ranges 0.583 to 0.617, so it is indistinguishable from the full-set 0.592 rather than above it. Separately, eight held-out reviews are byte-identical to training rows.

Multi-label in architecture, single-label in practice. The head is multi-label, but only 5 of 10,000 reviews in the generation subset receive more than one label, and the corpus-scale artifact stores a single argmax label.

Corpus and language. Trained on 58 Android applications from one store, in English, from a single collection window, with text that arrives lowercased and lemmatised with digits masked. Device models, OS versions and numeric quantities are largely destroyed by that preprocessing. Behaviour on raw, cased, non-English, or iOS review text is untested.

Taxonomy. The seven classes are our extension of the four-class Maalej and Nabil taxonomy (bug report, feature request, user experience, rating). There is no established seven-class app-review taxonomy.

The model weights and this card are released under the MIT License. This does not extend to the training data. The model was trained on labels derived from the RRGen corpus (Gao et al., ASE 2019), which is access-gated, states academic use only, and grants no redistribution right. We do not redistribute that corpus. See DATA_NOTICE.md.

Jalal, F., Dhruba, S. T., Tushti, T. Z., Abrar, A., Hasan, M. K., and Mahmud, H. Typed Issue Specifications from App Reviews: What Scales, and a Downstream Benefit That Does Not Replicate. Under review at Information and Software Technology, 2026.

Code: https://github.com/Fabiha-9876/ReviewAgent Data bundle: https://doi.org/10.5281/zenodo.22048738 Corrections: NOTICE_CORRECTION.md in the repository records seventeen numbered corrections.