Views
No views yet
EcomBert_NER_V1 is a span-based Named Entity Recognition (NER) model built on top of a BERT encoder with a GlobalPointer-style span classification head.config.jsonpytorch_model.bintransformers.AutoTokenizer.save_pretrained(...)(label, start, end) spans. Overlapping spans are possible.threshold)max_length.checkpoints/hf_export).1python train.py \
2 --splits_dir ./data2/splits \
3 --output_dir checkpoints \
4 --model_name bert-base-chinese \
5 --hf_export_dir hf_exportcheckpoints/hf_export/config.jsoncheckpoints/hf_export/pytorch_model.bincheckpoints/hf_export/tokenizer.*1python infer.py \
2 --model_dir checkpoints/hf_export \
3 --text "Apple released a new iPhone in California."1python infer.py \
2 --model_dir checkpoints/hf_export \
3 --text "Apple released a new iPhone in California." \
4 --threshold 0.551import torch
2from transformers import AutoTokenizer
3from model import EcomBertNER
4
5model_dir = "checkpoints/hf_export"
6
7device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
8model, cfg = EcomBertNER.from_pretrained(model_dir, device=device)
9
10tokenizer = AutoTokenizer.from_pretrained(model_dir)
11text = "Apple released a new iPhone in California."
12
13enc = tokenizer(text, return_tensors="pt", return_offsets_mapping=True)
14input_ids = enc["input_ids"].to(device)
15attention_mask = enc["attention_mask"].to(device)
16
17o = model(input_ids=input_ids, attention_mask=attention_mask)
18logits = o["logits"][0] # (C, L, L)
19probs = torch.sigmoid(logits)
20threshold = float(cfg.get("threshold", 0.5))
21
22hits = (probs > threshold).nonzero(as_tuple=False)
23print(hits[:10])| Label | Description |
|---|---|
MAIN_PRODUCT | Primary product being searched/described |
SUB_PRODUCT | Secondary / accessory product |
BRAND | Brand name |
MODEL | Model number or name |
IP | IP / licensed character / franchise |
MATERIAL | Material composition |
COLOR | Color attribute |
SHAPE | Shape attribute |
PATTERN | Pattern or print |
STYLE | Style descriptor |
FUNCTION | Function or use-case |
ATTRIBUTE | Other product attribute |
COMPATIBILITY | Compatible device / platform |
CROWD | Target audience |
OCCASION | Use occasion or scene |
LOCATION | Geographic / location reference |
MEASUREMENT | Size, dimension, capacity |
TIME | Time reference |
QUANTITY | Count or amount |
SALE | Promotion or sale information |
SHOP | Shop or seller name |
CONJ | Conjunction linking entities |
PREP | Preposition linking entities |
"Nike running shoes for men, breathable mesh upper, size 42"BRAND: "Nike"MAIN_PRODUCT: "running shoes"CROWD: "men"MATERIAL: "breathable mesh"MEASUREMENT: "size 42""iPhone 15 Pro compatible leather case, black, for outdoor use"COMPATIBILITY: "iPhone 15 Pro"MAIN_PRODUCT: "leather case"MATERIAL: "leather"COLOR: "black"OCCASION: "outdoor use""Disney Mickey pattern kids cotton pajamas, 3-piece set, buy 2 get 1 free"IP: "Disney Mickey"PATTERN: "Mickey pattern"CROWD: "kids"MATERIAL: "cotton"MAIN_PRODUCT: "pajamas"QUANTITY: "3-piece set"SALE: "buy 2 get 1 free"evaluate.py for evaluating .pt checkpoints produced during training.