You can deploy this model and start earning money today!

vuiseng9/
bert-base-uncased-squadv1-59.6-sparse

Views

No views yet

A set of unstructured sparse bert-base-uncased models fine-tuned for SQuADv1.
Tensorflow models are created using TFAutoModelForQuestionAnswering.from_pretrained(..., from_pt=True) and model.save_pretrained(tf_pth).
Observed issue - loss in model translation, discrepancy observed in evaluation between pytorch and tensorflow models.
Table below is evaluated in HF's transformers v4.9.2. Sparsity is normalized to dense layers in attention heads and FFNN.
Evaluation cli:
bash
1python run_qa.py \ 2 --model_name_or_path <model identifier> \ 3 --dataset_name squad \ 4 --do_eval \ 5 --per_device_eval_batch_size 384 \ 6 --max_seq_length 68 \ 7 --doc_stride 26 \ 8 --output_dir /tmp/eval-squad

	HF Model Hub Identifier	sparsity	em (pytorch)	em (tf)	f1 (pytorch)	f1 (tf)
0	vuiseng9/bert-base-uncased-squadv1-85.4-sparse	85.4	69.9338	14.2573	77.6861	23.4917
1	vuiseng9/bert-base-uncased-squadv1-72.9-sparse	72.9	74.6358	31.0596	82.2555	39.8446
2	vuiseng9/bert-base-uncased-squadv1-65.1-sparse	65.1	76.1306	43.0274	83.4117	51.4300
3	vuiseng9/bert-base-uncased-squadv1-59.6-sparse	59.6	76.8590	50.4920	84.1267	59.0881
4	vuiseng9/bert-base-uncased-squadv1-52.0-sparse	52.0	78.0038	54.2857	85.2000	62.2914

vuiseng9/bert-base-uncased-squadv1-59.6-sparse