ALM-Qwen Model: ALM-Qwen-0.5B-testing

Model Components

• AttentionLinkedMemory (ALM): A custom PyTorch module for two-level attention-based retrieval from structured memory. (See ALM.py)
• QwenGenerator: Wraps a Hugging Face Qwen model (e.g., Qwen2.5-0.5B-Instruct or Qwen2.5-7B-Instruct) for text generation.
• ALMQwenModel_HF: The main class orchestrating the ALM retrieval and Qwen generation. (See alm_qwen.py)
• Saved Weights & Config:
  • alm_layer_state_dict.pth: Trained weights for the ALM layer.
  • alm_qwen_hf_config.json: Configuration for the ALMQwenModel_HF, including ALM parameters and paths to the Qwen components.
  • qwen_generator/: Contains the saved Hugging Face Qwen model and tokenizer.

How to Use

1. Prerequisites:
   bash

   1pip install torch transformers huggingface_hub sentencepiece accelerate 2# Add other dependencies if any, e.g., bitsandbytes for quantization
2. Clone the repository (or download files manually):
   bash

   1git lfs install # if large files are used, though typically not for these components directly 2git clone https://huggingface.co/moelanoby/ALM-Qwen-0.5B-testing 3cd ALM-Qwen-0.5B-testing
3. Load the model in Python:
   python

   1from alm_qwen import ALMQwenModel_HF # Make sure alm_qwen_hf.py and ALM.py are in your PYTHONPATH 2import torch 3 4# Desired device 5device = "cuda" if torch.cuda.is_available() else "cpu" 6 7# Path to the directory where you cloned/downloaded the model 8model_directory = "." # Or the specific path if you are running from outside the cloned repo 9 10# Load the model 11loaded_model = ALMQwenModel_HF.load_model(model_directory, device=device) 12print("ALM-Qwen model loaded successfully!") 13 14# --- Prepare Dummy Input Data (similar to the example in alm_qwen_hf.py) --- 15# batch_size = 1 16# alm_query_dim = loaded_model.alm_config['query_dim'] 17# alm_memory_dim = loaded_model.alm_config['memory_dim'] 18# num_kb_buckets = 3 # Example 19# max_kb_items_per_bucket = 5 # Example 20 21# query_texts = ["What is the capital of France?"] 22# query_embeddings_for_alm = torch.randn(batch_size, alm_query_dim) 23# memory_item_embeddings = torch.randn(batch_size, num_kb_buckets, max_kb_items_per_bucket, alm_memory_dim) 24# memory_text_items = [[["Paris is the capital of France." for _ in range(max_kb_items_per_bucket)] for _ in range(num_kb_buckets)] for _ in range(batch_size)] 25# memory_mask = torch.ones(batch_size, num_kb_buckets, max_kb_items_per_bucket, dtype=torch.bool) 26# memory_mask[:, :, -1] = False # Example mask 27 28# # Run inference 29# generated_answers, _, _ = loaded_model( 30# query_texts, 31# query_embeddings_for_alm, 32# memory_item_embeddings, 33# memory_text_items, 34# memory_mask 35# ) 36# print(f"Query: {query_texts[0]}") 37# print(f"Answer: {generated_answers[0]}")

Training

Notes

• The Qwen model components can be large. Ensure you have sufficient disk space and network bandwidth.
• The load_model method in alm_qwen_hf.py handles the reconstruction of the composite model.
• If any errors happen use alm_qwen.py directly