You can deploy this model and start earning money today!

FishAndSheep/
TextFluoroscopy

Name: TextFluoroscopy
Rating: 2.5 (6 reviews)

safetensors

binary_classifier

custom_code

Views

No views yet

🔍 Text Fluoroscopy

Python 3.8+

Text Fluoroscopy: we propose Text Fluoroscopy, a black-box method for LLM-generated text detection through intrinsic features. Our method captures the text’s intrinsic features by identifying the layer with the largest distribution difference from the last and first layers when projected to the vocabulary space.

📋 Overview

Most LLM detection methods focus on either semantic features (from the last layer) or simple linguistic features (from early layers). Text Fluoroscopy takes a different approach by:

Finding the model layer with the largest distribution difference from both first and last layers
Extracting intrinsic text features that reveal the actual "fingerprints" of AI generation
Achieving superior generalization and robustness across different domains and against paraphrase attacks

🚀 Getting Started

Using our pretrained model

python

from __future__ import annotations
from transformers import AutoModel
from transformers import AutoTokenizer, AutoModel,AutoModelForCausalLM
import torch.nn.functional as F
import torch
import json
from tqdm import tqdm
import torch.nn as nn
from sklearn.metrics import roc_auc_score

class TextFluoroscopy(nn.Module):
    def __init__(self, pretrained_model_name_or_path,clf_model, device='cuda'):
        super(TextFluoroscopy, self).__init__()
        self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, trust_remote_code=True)
        self.tokenizer.pad_token = self.tokenizer.eos_token
        self.embedding_model = AutoModelForCausalLM.from_pretrained(pretrained_model_name_or_path, trust_remote_code=True,device_map='auto' )
        self.clf_model = clf_model
        self.clf_model = self.clf_model.to(device=device)
        self.clf_model.eval()
    def last_token_pool(self, last_hidden_states,attention_mask):
        left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
        if left_padding:
            return last_hidden_states[:, -1]
        else:
            sequence_lengths = attention_mask.sum(dim=1) - 1
            batch_size = last_hidden_states.shape[0]
            return last_hidden_states[torch.arange(batch_size, device='cpu'), sequence_lengths]
    def get_embedding(self,input_texts):
        batch_dict = self.tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
        with torch.no_grad():
            outputs = self.embedding_model(**batch_dict,output_hidden_states=True)
            last_logits = self.embedding_model.lm_head(outputs.hidden_states[-1]).squeeze()
            first_logits = self.embedding_model.lm_head(outputs.hidden_states[0]).squeeze()
        kls = []
        for i in range(1,len(outputs.hidden_states)-1):
            with torch.no_grad():
                middle_logits = self.embedding_model.lm_head(outputs.hidden_states[i]).squeeze()
            kls.append(F.kl_div(F.log_softmax(middle_logits, dim=-1), F.softmax(first_logits, dim=-1), reduction='batchmean').item()+
                    F.kl_div(F.log_softmax(middle_logits, dim=-1), F.softmax(last_logits, dim=-1), reduction='batchmean').item())
        max_kl_idx = kls.index(max(kls))
        max_kl_embedding = self.last_token_pool(outputs.hidden_states[max_kl_idx+1].cpu(), batch_dict['attention_mask'])
        return max_kl_embedding
    def forward(self, texts):
        embs = []
        for text in tqdm(texts):
            emb = self.get_embedding([text])
            embs.append(emb)
        embs = torch.cat(embs, dim=0).to(device=device)
        with torch.no_grad():
            outputs = self.clf_model(embs)
            probs = torch.softmax(outputs, dim=1)[:, 1]
            return probs




device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pretrained_model_name_or_path = 'Alibaba-NLP/gte-Qwen1.5-7B-instruct'

clf_model = AutoModel.from_pretrained("FishAndSheep/TextFluoroscopy",trust_remote_code=True)
model = TextFluoroscopy(pretrained_model_name_or_path,clf_model)

labels = torch.tensor([0, 1, 0, 1, 0])
labels = labels.to(device=device)

texts = [
    'Maj Richard Scott, 40, is accused of driving at speeds of up to 95mph (153km/h) in bad weather before the smash on a B-road in Wiltshire. Gareth Hicks, 24, suffered fatal injuries when the van he was asleep in was hit by Mr Scott\'s Audi A6. Maj Scott denies a charge of causing death by careless driving. Prosecutor Charles Gabb alleged the defendant, from Green Lane in Shepperton, Surrey, had crossed the carriageway of the 60mph-limit B390 in Shrewton near Amesbury. The weather was "awful" and there was strong wind and rain, he told jurors. He said Mr Scott\'s car was described as "twitching" and "may have been aquaplaning" before striking the first vehicle; a BMW driven by Craig Reed. Mr Scott\'s Audi then returned to his side of the road but crossed the carriageway again before colliding head-on with a Ford Transit van in which Mr Hicks was a passenger, the court was told. "There is no doubt that when the Audi smashed into the panel van he was on completely the wrong',
    "Maj Richard Scott, 40, is accused of driving at speeds of up to 95mph (153km/h) in bad weather before the fatal crash that claimed the lives of two young children. The incident occurred on the A34 motorway near Newbury, Berkshire, last Saturday.\n\nEyewitnesses reported seeing Maj Scott's car weaving in and out of traffic before losing control and colliding with another vehicle. The impact caused both cars to veer off the road and crash into a nearby tree.\n\nTragically, the two children, aged 3 and 5, who were passengers in the other vehicle, were pronounced dead at the scene. The driver of that car, a 32-year-old woman, sustained serious injuries and is currently in critical condition at a local hospital.\n\nMaj Scott, a decorated military officer, has been charged with dangerous driving causing death and grievous bodily harm. He appeared before the magistrates' court earlier today and was released on bail.\n\nThe incident has sparked a debate regarding the responsibility of drivers, especially those in high-stress professions, to ensure the safety of others on the road. The court", 
    "The ban was backed by local authorities in Urumqi, state media reported. The move is the latest in a campaign against Islamic clothing and symbols in the mainly Muslim region. China's government has blamed Uighur militants for several recent attacks - but activists say the violence has been fuelled by Chinese repression. The Uighurs traditionally practice a moderate form of Islam. Recently, though, the influence of stricter forms of the faith from the Gulf and Pakistan has led to more women covering their faces. According to the BBC's Asia Pacific editor, Charles Scanlon, the Chinese authorities regard wearing an Islamic face veil as an act of defiance at a time of growing violence in the region. In August this year, the northern city of Karamay banned men with long beards from boarding buses. Some experts believe such restrictions could backfire at a time when many Uighurs believe their culture is under attack. James Leibold, an expert on China's ethnic policy at Melbourne's La Trobe University, told Reuters news agency", 
    "The ban was backed by local authorities in Urumqi, state media reported. The move is the latest in a campaign against Islamic clothing and symbols, as the authorities strive to promote secularism and maintain social stability in the region.\n\nThe ban specifically targets clothing and symbols associated with Islam, such as the hijab and the crescent moon and star. This decision follows similar measures implemented in other parts of the country, where the government has been cracking down on religious expression deemed to be a threat to the state's authority.\n\nCritics argue that these bans infringe on the rights of religious minorities, particularly the Muslim community, and exacerbate tensions between different ethnic and religious groups. Supporters, on the other hand, believe that such measures are necessary to combat religious extremism and maintain social harmony.\n\nChina's Muslim population is estimated to be around 23 million people, primarily residing in the western region of Xinjiang. This latest ban further adds to the ongoing debate surrounding religious freedom in China and raises concerns about the",
    'Palmer expects to be fined by the British Horseracing Authority (BHA) for the rule breach during Champions Day. "It was an oversight," Palmer told BBC Sport. "The wrong kit box was left in the lorry and taken to the stables." The Newmarket-based trainer saw Galilieo Gold finish fifth in the Group One Queen Elizabeth II Stakes at Ascot. Horses are only allowed to receive hay-feed or water before a race, but other substances such as electrolytes are permitted post-race. Employees of Palmer and fellow trainers, Francois Rohaut and and Jean-Claude Rouget, as well as the trainers themselves, were interviewed by stewards following a report by the Equine Welfare and Integrity Officer. Palmer is yet to hear what punishment the BHA will hand down. "I imagine there will be a routine fine for a rule that\'s been broken," he told BBC Radio Berkshire. "There\'s no suggestion that anyone was trying to seek an advantage or cheat. It\'s simply what the rule says. The rules are the rules'
]


probs = model(texts)
print(probs,probs.shape)
auroc = roc_auc_score(labels.cpu().numpy(), probs.cpu().numpy())
print('Detection AUROC:',auroc)

⚡ Performance Optimization

Although our dynamic layer-selection method is effective, the computational overhead of examining each layer introduces time delays. To address this limitation, we provide an alternative approach:

Methods	ChatGPT	GPT-4	Claude3
Detection with the Last Layer	0.07s	0.09s	0.08s
Text Fluoroscopy	0.52s	0.51s	0.49s
Detection with the 30-th layer	0.08s	0.08s	0.07s

As shown above:

We identified that using the fixed 30th layer offers an excellent balance between accuracy and speed
This optimization reduces processing time by ~6.5× with less than 0.7% accuracy reduction
The 30th layer approach maintains the core benefits of our method while being nearly as fast as last-layer-only methods

🔄 Applicability Across LLMs

Text Fluoroscopy is not limited to a specific encoder model. We tested our approach with multiple LLMs to demonstrate its broad applicability:

LLM	Layer	ChatGPT				GPT-4				Claude3
		XSum	Writing	PubMed	Avg.	XSum	Writing	PubMed	Avg.	XSum	Writing	PubMed	Avg.
gte-Qwen2-7B	Last	0.9658	0.9710	0.6186	0.8518	0.9711	0.9758	0.6847	0.8772	0.9472	0.9836	0.8011	0.9106
	Middle	0.9988	0.9834	0.7744	0.9189	0.9996	0.9872	0.8416	0.9428	0.9994	0.9953	0.9373	0.9773
stella_en_1.5B_v5	Last	0.8928	0.9802	0.6966	0.8565	0.8996	0.9708	0.7293	0.8666	0.8971	0.9758	0.8646	0.9125
	Middle	1.0000	0.9921	0.6611	0.8844	1.0000	0.9873	0.6955	0.8943	0.9997	0.9834	0.8955	0.9595
GPT-neo-2.7B	Last	0.6270	0.7190	0.6079	0.6513	0.7317	0.7970	0.4883	0.6724	0.9674	0.9981	0.8769	0.9475
	Middle	0.8568	0.8916	0.6079	0.7854	0.9005	0.9137	0.5027	0.7723	0.9945	0.9933	0.9350	0.9743

As shown in the table above:

Our method consistently outperforms the last-layer approach across different encoder architectures
We tested with both advanced models (gte-Qwen2-7B, stella_en_1.5B_v5) and classical models (GPT-neo-2.7B)
The improvement is especially significant when detecting ChatGPT and GPT-4 generated content
This demonstrates that Text Fluoroscopy's core principle of leveraging intrinsic features from middle layers is valid across LLM architectures

📚 Citation

If you find our work useful, please consider citing our paper:

bibtex

@inproceedings{yu2024textfluoroscopy,
    title={Text Fluoroscopy: Detecting LLM-Generated Text through Intrinsic Features},
    author={Yu, Xiao and Chen, Kejiang and Yang, Qi and Zhang, Weiming and Yu, Nenghai},
    booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
    year={2024},
    url={https://aclanthology.org/2024.emnlp-main.885.pdf}
}

📬 Contact

For questions or feedback, please open an issue or contact us at Xiao Yu.