You can deploy this model and start earning money today!

BEE-spoke-data/
BeeTokenizer

BEE-spoke-data/bees-internal

apache-2.0

Views

No views yet

BeeTokenizer

note: this is literally a tokenizer trained on beekeeping text

After minutes of hard work, it is now available.

python

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("BEE-spoke-data/BeeTokenizer")

test_string = "When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination."

output = tokenizer(test_string)
print(f"Test string: {test_string}")
print(f"Tokens ({len(output.input_ids)}):\n\t{output.input_ids}")

Notes

the default tokenizer (on branch main) has a vocab size of 32000
based on the SentencePieceBPETokenizer class

How to Tokenize Text and Retrieve Offsets

To tokenize a complex sentence and also retrieve the offsets mapping, you can use the following Python code snippet:

python

from transformers import AutoTokenizer

# Initialize the tokenizer
tokenizer = AutoTokenizer.from_pretrained("BEE-spoke-data/BeeTokenizer")

# Sample complex sentence related to beekeeping
test_string = "When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination."

# Tokenize the input string and get the offsets mapping
output = tokenizer.encode_plus(test_string, return_offsets_mapping=True)

print(f"Test string: {test_string}")

# Tokens
tokens = tokenizer.convert_ids_to_tokens(output['input_ids'])
print(f"Tokens: {tokens}")

# Offsets
offsets = output['offset_mapping']
print(f"Offsets: {offsets}")

This should result in the following (Feb '24 version):

>>> print(f"Test string: {test_string}")
Test string: When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination.
>>>
>>> # Tokens
>>> tokens = tokenizer.convert_ids_to_tokens(output['input_ids'])
>>> print(f"Tokens: {tokens}")
Tokens: ['When', '▁dealing', '▁with', '▁Varroa', '▁destructor', '▁mites,', "▁it's", '▁cru', 'cial', '▁to', '▁administer', '▁the', '▁right', '▁acar', 'icides', '▁during', '▁the', '▁late', '▁autumn', '▁months,', '▁but', '▁only', '▁after', '▁ensuring', '▁that', '▁the', '▁worker', '▁bee', '▁population', '▁is', '▁free', '▁from', '▁pesticide', '▁contam', 'ination.']
>>>
>>> # Offsets
>>> offsets = output['offset_mapping']
>>> print(f"Offsets: {offsets}")
Offsets: [(0, 4), (4, 12), (12, 17), (17, 24), (24, 35), (35, 42), (42, 47), (47, 51), (51, 55), (55, 58), (58, 69), (69, 73), (73, 79), (79, 84), (84, 90), (90, 97), (97, 101), (101, 106), (106, 113), (113, 121), (121, 125), (125, 130), (130, 136), (136, 145), (145, 150), (150, 154), (154, 161), (161, 165), (165, 176), (176, 179), (179, 184), (184, 189), (189, 199), (199, 206), (206, 214)]

if you compare this to the output of the llama tokenizer (below), you can quickly see which is more suited for beekeeping related language modeling.

>>> print(f"Test string: {test_string}")
Test string: When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination.
>>> # Tokens
>>> tokens = tokenizer.convert_ids_to_tokens(output['input_ids'])
>>> print(f"Tokens: {toke>>> print(f"Tokens: {tokens}")
Tokens: ['<s>', '▁When', '▁dealing', '▁with', '▁Var', 'ro', 'a', '▁destruct', 'or', '▁mit', 'es', ',', '▁it', "'", 's', '▁cru', 'cial', '▁to', '▁admin', 'ister', '▁the', '▁right', '▁ac', 'ar', 'ic', 'ides', '▁during', '▁the', '▁late', '▁aut', 'umn', '▁months', ',', '▁but', '▁only', '▁after', '▁ens', 'uring', '▁that', '▁the', '▁worker', '▁be', 'e', '▁population', '▁is', '▁free', '▁from', '▁p', 'estic', 'ide', '▁cont', 'am', 'ination', '.']
>>> offsets = output['offset_mapping']
>>> print(f"Offsets: {offsets}")
Offsets: [(0, 0), (0, 4), (4, 12), (12, 17), (17, 21), (21, 23), (23, 24), (24, 33), (33, 35), (35, 39), (39, 41), (41, 42), (42, 45), (45, 46), (46, 47), (47, 51), (51, 55), (55, 58), (58, 64), (64, 69), (69, 73), (73, 79), (79, 82), (82, 84), (84, 86), (86, 90), (90, 97), (97, 101), (101, 106), (106, 110), (110, 113), (113, 120), (120, 121), (121, 125), (125, 130), (130, 136), (136, 140), (140, 145), (145, 150), (150, 154), (154, 161), (161, 164), (164, 165), (165, 176), (176, 179), (179, 184), (184, 189), (189, 191), (191, 196), (196, 199), (199, 204), (204, 206), (206, 213), (213, 214)]