Reconstruction-based LSTM-Autoencoder for detecting anomalous REST API traffic. Trained exclusively on normal (benign) traffic, so it needs no labelled attack data and can generalize to sequence-dependent attacks — such as Broken Object Level Authorization (BOLA) — that request-isolated, signature-based defences (WAFs) typically miss.
Numeric mode (for network-flow traffic, e.g. CIC-IDS2018, UNSW-NB15): standardized float feature vectors fed directly to the LSTM
Sequence length: sliding-window sessions of W = 5 consecutive requests/flows
Dropout: 0.2 between LSTM layers
Training objective: minimize mean-squared reconstruction error on normal traffic only
Detection rule: a session is flagged anomalous if its reconstruction error exceeds a threshold calibrated at the 95th percentile of normal-traffic reconstruction errors; the threshold can be further refined via an Active Learning (uncertainty-sampling) loop without retraining the model
This repository provides separate checkpoints per dataset, since each uses a different input mode / feature space (see "Checkpoints" below).
Intended Use
Detecting anomalous/attack sessions in REST API or network-flow traffic in settings where labelled attack data is unavailable (the common zero-day scenario). Designed for real-time or near-real-time deployment: inference latency is ~0.044–0.050 ms per session on CPU.
Not intended for: single-request (non-sequential) anomaly detection, traffic types substantially different from the training distributions below without retraining/fine-tuning, or as a sole security control (best used alongside, not instead of, a WAF).
Training Data
Trained separately, per dataset, on normal traffic only:
Dataset
Type
Normal sessions (train)
CSIC 2010
HTTP logs
35,996
CIC-IDS2018
Network flows
1,676,966
UNSW-NB15
Network flows
74,396
See the accompanying dataset card for full dataset details and licensing.
Evaluation
Dataset
Precision
Recall
F1
FPR
Latency (ms/session)
CSIC 2010
0.946
1.000
0.972
0.040
0.050
CIC-IDS2018
0.037†
0.988
0.071†
0.057
0.044
UNSW-NB15
0.979
0.667
0.794
0.128
0.044
† CIC-IDS2018 precision is limited by extreme session-level class imbalance (~454:1 normal:attack) in the test set, not a model deficiency — recall stays high (0.988).
On UNSW-NB15, the model outperforms an Isolation Forest baseline (F1 = 0.663) while running 6.1–6.8× faster.
Ablation: why normal-only training?
A supplementary ablation trained the same architecture on balanced normal/attack data (reconstruction objective and direct supervision). Balanced reconstruction training degraded F1 by 43–78% relative to the normal-only model across all three datasets, confirming that training exclusively on normal traffic is necessary — not incidental — for reconstruction-based anomaly detection to work as intended. Direct supervision achieved near-perfect scores only because it requires labelled attack data at training time, which defeats the zero-day use case this model targets.
How to Use
python
1import torch
2from model import LSTMAutoencoder # see repo for model.py34model = LSTMAutoencoder(mode="numeric", input_dim=23, hidden_size=64)5state_dict = torch.load("lstm_ae_cicids2018.pt", map_location="cpu")6model.load_state_dict(state_dict)7model.eval()89# x: tensor of shape (batch, W=5, input_dim), pre-scaled with the matching StandardScaler10with torch.no_grad():11 x_hat = model(x)12 error =((x - x_hat)**2).mean(dim=(1,2))# per-session MSE anomaly score1314threshold =0.0136# example — use the calibrated threshold shipped for each checkpoint15is_anomalous = error > threshold
For CSIC 2010 (embedding mode), inputs are integer token sequences produced by the matching vocabulary/abstraction pipeline in the source repository — using the model without that exact preprocessing will produce unreliable results, since the abstraction step is load-bearing for this model (see Limitations).
Limitations
CIC-IDS2018 precision is fundamentally constrained by the dataset's extreme class imbalance at the session level, independent of model quality.
Preprocessing-sensitive: the URL-abstraction / feature-scaling pipeline used at inference time must exactly match the one used to build the training vocabulary/scaler. Divergent implementations of the same logic (observed during development) silently inflate false positives.
Session ordering matters for flow-based datasets: sessions must be constructed in timestamp order, not raw file order, to match training conditions.
BOLA is a motivating attack class for the sequence-aware design but was not evaluated with a dedicated, labelled BOLA dataset.
Overlapping sliding-window sessions carry a theoretical train/test leakage risk (adjacent windows share requests), acknowledged as a limitation rather than resolved.
Latency figures are per-session batched inference, not single live-request latency.
Citation
Kandanaarachchi, K.A.D.S.D. "Detecting Anomalous REST API Traffic."
BSc thesis, University of Vavuniya, 2020/ICT/19.
Supervised by Dr. S. Kirushanth.
License
MIT (code/weights). Underlying datasets retain their original licenses — see the dataset card.
