LWM is a powerful pre-trained model developed as a universal feature extractor for wireless channels. As the world's first foundation model crafted for this domain, LWM leverages transformer architectures to extract refined representations from simulated datasets, such as DeepMIMO and Sionna, and real-world wireless data.
How is LWM built?
The LWM model’s structure is based on transformers, allowing it to capture both fine-grained and global dependencies within channel data. Unlike traditional models that are limited to specific tasks, LWM employs a self-supervised approach through our proposed technique, Masked Channel Modeling (MCM). This method trains the model on unlabeled data by predicting masked channel segments, enabling it to learn intricate relationships between antennas and subcarriers. Utilizing bidirectional attention, LWM interprets the full context by attending to both preceding and succeeding channel segments, resulting in embeddings that encode comprehensive spatial information, making them applicable to a variety of scenarios.
What does LWM offer?
LWM provides a universal feature extraction framework that can be applied across diverse wireless communication and sensing tasks. It is built to handle complex wireless environments, capturing channel characteristics in a way that facilitates robust performance across different scenarios and conditions.
Trained on hundreds of thousands of wireless channel samples, LWM has been designed to generalize across varied environments—from dense urban areas to synthetic setups, ensuring its adaptability and consistency across a broad spectrum of wireless tasks.
How is LWM used?
LWM is designed to be easily integrated into downstream applications as a source of high-quality embeddings that encapsulate complex channel features. By feeding raw wireless channel data into the pre-trained model, users obtain embeddings that capture essential spatial relationships and interactions within the channel environment.
These embeddings provide a versatile and contextualized representation of wireless data, which can be leveraged across different applications. By utilizing the pre-trained model in this way, users can reduce the need for extensive labeled data while benefiting from embeddings that retain the critical properties of the original channel.
Advantages of Using LWM
Various Tasks: Self-supervised and pre-trained without labels, LWM excels in a wide range of wireless tasks, offering flexibility and performance
Limited Data: With LWM embeddings, downstream tasks achieve high accuracy with less data, cutting reliance on large datasets
Various Environments: Pre-trained on diverse data, LWM excels in various environments from urban to rural areas, ensuring reliable performance
Join the growing community of researchers using LWM for their wireless communication and sensing research, and unlock a new level of performance and insight in your models!
Please cite the following paper if you use the LWM model or any modified parts:
@misc{alikhani2024largewirelessmodellwm,
title={Large Wireless Model (LWM): A Foundation Model for Wireless Channels},
author={Sadjad Alikhani and Gouranga Charan and Ahmed Alkhateeb},
year={2024},
eprint={2411.08872},
archivePrefix={arXiv},
primaryClass={cs.IT},
url={https://arxiv.org/abs/2411.08872},
}
🛠 How to Use
1. Install Conda
First, ensure that you have a package manager like Conda installed to manage your Python environments and packages. You can install Conda via Anaconda or Miniconda.
Anaconda includes a comprehensive scientific package suite. Download it here.
Miniconda is a lightweight version that includes only Conda and Python. Download it here.
Once installed, you can use Conda to manage environments.
2. Create a New Environment
After installing Conda, follow these steps to create a new environment and install the required packages.
Step 1: Create a new environment
To begin, open the Anaconda PowerShell Prompt and create a new Conda environment named lwm_env:
conda create -n lwm_env
Step 2: Activate the environment
Activate the environment:
conda activate lwm_env
3. Install Required Packages
Once the environment is activated, install the necessary packages.
Install CUDA-enabled PyTorch
Although inference can run efficiently on a CPU, you may need a GPU for training more resource-intensive downstream tasks. Visit this page and select the appropriate options based on your system's specifications. The website will generate a tailored installation command.
For instance, on an NVIDIA system, you can use a command like the following with the appropriate CUDA version for your system:
This command installs PyTorch with CUDA support for GPU-accelerated training. Ensure that the specified CUDA version is compatible with your system, adjusting it if necessary.
Note: If you encounter issues installing CUDA-enabled PyTorch, verify your CUDA version compatibility. It might also be due to conflicting installation attempts—try a fresh environment.
The following functions will help you clone specific dataset scenarios from a repository:
python
1import subprocess
2import os
3import shutil
45defclone_dataset_scenario(repo_url, model_repo_dir="./LWM", scenarios_dir="scenarios"):6"""
7 Clones all scenarios from a repository, ensuring all files (small and large) are downloaded.
89 Args:
10 repo_url (str): URL of the Git repository
11 model_repo_dir (str): Path to the model repository
12 scenarios_dir (str): Directory name for storing scenarios
13 """14# Ensure we're in the correct directory structure15 current_dir = os.path.basename(os.getcwd())16if current_dir =="LWM":17 model_repo_dir ="."1819# Create the scenarios directory if it doesn't exist20 scenarios_path = os.path.join(model_repo_dir, scenarios_dir)21 os.makedirs(scenarios_path, exist_ok=True)2223# Store the original working directory24 original_dir = os.getcwd()2526try:27# Clean up any existing temp directory28if os.path.exists(scenarios_path):29 shutil.rmtree(scenarios_path)3031# Clone the entire repository (including all files)32print(f"Cloning entire repository into temporary directory...")33 subprocess.run([34"git","clone",35 repo_url,36 scenarios_path
37], check=True)3839# Navigate to the temporary clone directory40 os.chdir(scenarios_path)4142# Pull all files using Git LFS43print(f"Pulling all files using Git LFS...")44 subprocess.run(["git","lfs","install"], check=True)# Ensure LFS is installed45 subprocess.run(["git","lfs","pull"], check=True)# Pull all LFS files4647print(f"Successfully cloned all scenarios into {scenarios_path}")4849except subprocess.CalledProcessError as e:50print(f"Error cloning scenarios: {str(e)}")51finally:52# Clean up temporary directory53if os.path.exists(scenarios_path):54 shutil.rmtree(scenarios_path)55# Return to original directory56 os.chdir(original_dir)
5. Clone the Model Repository
Now, clone the LWM model repository to your local system.
bash
1# Step 1: Clone the model repository (if not already cloned)2model_repo_url ="https://huggingface.co/wi-lab/lwm"3model_repo_dir ="./LWM"45if not os.path.exists(model_repo_dir):
6 print(f"Cloning model repository from {model_repo_url}...")7 subprocess.run(["git", "clone", model_repo_url, model_repo_dir], check=True)
6. Clone the Desired Dataset Scenarios
You can now clone specific scenarios from the DeepMIMO dataset, as detailed in the table below:
It is important to note that these six datasets were not used during the pre-training of the LWM model, and the high-quality embeddings produced are a testament to LWM’s robust generalization capabilities rather than overfitting.
The operational settings below were used in generating the datasets for both the pre-training of LWM and the downstream tasks. If you intend to use custom datasets, please ensure they adhere to these configurations:
Operational Settings:
Antennas at BS: 32
Antennas at UEs: 1
Subcarriers: 32
Paths: 20
Frequency: 3.5GHz (By the way, our results are consistent across different frequency ranges.)
Clone the Scenarios:
python
1import numpy as np
2dataset_repo_url ="https://huggingface.co/datasets/wi-lab/lwm"# Base URL for dataset repo34# Clone the requested scenarios5clone_dataset_scenario(dataset_repo_url, model_repo_dir)
7. Change the Working Directory to LWM
bash
1if os.path.exists(model_repo_dir):
2 os.chdir(model_repo_dir)3 print(f"Changed working directory to {os.getcwd()}")4else:
5 print(f"Directory {model_repo_dir} does not exist. Please check if the repository is cloned properly.")
8. Tokenize and Load the Model
Before we dive into tokenizing the dataset and loading the model, let's understand how the tokenization process is adapted to the wireless communication context. In this case, tokenization refers to segmenting each wireless channel into patches, similar to how Vision Transformers (ViTs) work with images. Each wireless channel is structured as a 32x32 matrix, where rows represent antennas and columns represent subcarriers.
The tokenization process involves dividing the channel matrix into patches, with each patch containing information from 16 consecutive subcarriers. These patches are then embedded into a 64-dimensional space, providing the Transformer with a richer context for each patch. In this process, positional encodings are added to preserve the structural relationships within the channel, ensuring the Transformer captures both spatial and frequency dependencies.
If you choose to apply Masked Channel Modeling (MCM) during inference (by setting gen_raw=False), LWM will mask certain patches, as it did during pre-training. However, for standard inference, masking isn't necessary unless you want to test LWM's robustness to noisy inputs! The printed LWM loss after inference could show you how well it has predicted the masked patches.
Now, let's move on to tokenize the dataset and load the pre-trained LWM model.
python
1from input_preprocess import tokenizer
2from lwm_model import lwm
3import torch
45scenario_names = np.array([6"city_18_denver","city_15_indianapolis","city_19_oklahoma",7"city_12_fortworth","city_11_santaclara","city_7_sandiego"8])9scenario_idxs = np.array([0,1,2,3,4,5])# Select the scenario indexes10selected_scenario_names = scenario_names[scenario_idxs]1112preprocessed_chs = tokenizer(13 selected_scenario_names=selected_scenario_names,# Selects predefined DeepMIMOv3 scenarios. Set to None to load your own dataset.14 manual_data=None,# If using a custom dataset, ensure it is a wireless channel dataset of size (N,32,32) based on the settings provided above.15 gen_raw=True# Set gen_raw=False to apply masked channel modeling (MCM), as used in LWM pre-training. For inference, masking is unnecessary unless you want to evaluate LWM's ability to handle noisy inputs.16)1718device ='cuda'if torch.cuda.is_available()else'cpu'19print(f"Loading the LWM model on {device}...")20model = lwm.from_pretrained(device=device)
With this setup, you're ready to pass your tokenized wireless channels through the pre-trained model, extracting rich, context-aware embeddings that are ready for use in downstream tasks.
9. Perform Inference
Before running the inference, it's important to understand the benefits of the different embedding types. The CLS embeddings (cls_emb) provide a highly compressed, holistic view of the entire wireless channel, making them ideal for tasks requiring a general understanding, such as classification or high-level decision-making. On the other hand, channel embeddings (channel_emb) capture detailed spatial and frequency information from the wireless channel, making them more suitable for complex tasks like beamforming or channel prediction.
You can now perform inference on the preprocessed data using the LWM model.
python
1from inference import lwm_inference, create_raw_dataset
2input_types =['cls_emb','channel_emb','raw']3selected_input_type = input_types[1]# Change the index to select LWM CLS embeddings, LWM channel embeddings, or the original input channels.45if selected_input_type in['cls_emb','channel_emb']:6 dataset = lwm_inference(preprocessed_chs, selected_input_type, model, device)7else:8 dataset = create_raw_dataset(preprocessed_chs, device)
By selecting either cls_emb or channel_emb, you leverage the pre-trained model's rich feature extraction capabilities to transform raw channels into highly informative embeddings. If you prefer to work with the original raw data, you can choose the raw input type.
10. Generate Labels if Necessary
If your dataset requires labels, you can easily generate them using DeepMIMO data. Here's an example to create labels for either LoS/NLoS classification or beam prediction, depending on the scenario selected:
python
1from input_preprocess import create_labels
2tasks =['LoS/NLoS Classification','Beam Prediction']3task = tasks[1]# Choose 0 for LoS/NLoS labels or 1 for beam prediction labels.4labels = create_labels(task, selected_scenario_names, n_beams=64)# For beam prediction, n_beams specifies the number of beams in the codebook. If you're generating labels for LoS/NLoS classification, you can leave this value unchanged as it doesn't impact the label generation.
11. Leverage the Dataset for Downstream Tasks
LWM, pre-trained on a vast and diverse dataset using self-supervised learning, does not rely on labeled data. During inference, it transforms raw channels into rich embeddings in real time, capturing both general and intricate patterns within the wireless channels. These embeddings can be directly applied to various downstream tasks, offering a more powerful alternative to using the original channel data.
12. Explore the Interactive Demo
To experience LWM interactively, visit our demo hosted on Hugging Face Spaces:
You are now ready to explore the power of LWM in wireless communications! Start processing datasets and generate high-quality embeddings to advance your research or applications.
If you have questions or need assistance, feel free to:
