Views
No views yet
model.generate() call. You can update the memories using the following sequence of commands:1model.clear_memories()
2model.memory_ids = list_of_new_token_idstrust_remote_code=True to avoid warnings. Pass the memories to the model as a list of token ids.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3ag_wiki_entry = """Alexander Grothendieck (/ˈɡroʊtəndiːk/; German pronunciation: [ˌalɛˈksandɐ ˈɡʁoːtn̩ˌdiːk] (listen); French: [ɡʁɔtɛndik]; 28 March 1928 – 13 November 2014) was a stateless (and then, since 1971, French) mathematician who became the leading figure in the creation of modern algebraic geometry.[7][8] His research extended the scope of the field and added elements of commutative algebra, homological algebra, sheaf theory, and category theory to its foundations, while his so-called "relative" perspective led to revolutionary advances in many areas of pure mathematics.[7][9] He is considered by many to be the greatest mathematician of the twentieth century.[10][11]"""
4
5tokenizer_hf = AutoTokenizer.from_pretrained("normalcomputing/extended-mind-llama-2-70b")
6memories = tokenizer_hf(ag_wiki_entry).input_ids
7
8model_hf = AutoModelForCausalLM.from_pretrained("normalcomputing/extended-mind-llama-2-70b", external_memories=memories, trust_remote_code=True)topk below) by passing new values to the model.generate() method.1inputs = "When did Alexander Grothendieck become a French citizen?"
2inputs = tokenizer(inputs, return_tensors="pt").input_ids
3
4outputs = model.generate(inputs, max_length=40, topk=2)
5tokenizer.decode(outputs_hf['sequences'][0], skip_special_tokens=True)output_retrieved_memory_idx=True in the model.generate() method, you can retrieve the memory indices used during generation. We walk through an example in the demo notebook.memory_type (string, optional, defaults to manual):
Whether to store external memories manually or in a vector database.mask_by_sim (bool, optional, defaults to True):
Whether or not to mask retrieved memories by similarity.sim_threshold (float, optional, defaults to 0.25):
Threshold for masking retrieved memories.tokenizer_all_special_ids (list, optional, defaults to [0, 50278]):
Ids for special tokens to remove from memories.remove_special_tokens (bool, optional, defaults to True):
Remove memories that correspond to tokenizer special ids.generate_cache() method. Smaller strides generate higher-quality representations, while larger strides require fewer computations.