Views
No views yet
minn=[Your minn], maxn=[Your maxn]), which is particularly beneficial for morphologically rich languages like Urdu. This allows the model to:[Your vector_size].train.txt) where each line represented a sentence or document, and words were separated by spaces.model=cbow). [If you used skipgram, specify that instead and briefly explain why, e.g., "Skip-gram model (model=skipgram), often better for capturing representations of rare words."]dim: [Your vector_size] (Vector dimensionality)ws: [Your window_size] (Context window size)minCount: [Your min_word_count] (Minimum word frequency to be included in vocabulary)epoch: [Your epochs] (Number of training epochs)neg: [Your negative_samples] (Number of negative samples)minn: [Your minn] (Minimum character n-gram length)maxn: [Your maxn] (Maximum character n-gram length)thread: 4 (Number of threads used)1pip install fasttext
2import fasttext
3import numpy as np # For calculating cosine similarity
4
5# Path to the downloaded .bin model file
6model_path = "path/to/your/downloaded/urdu_fasttext.bin"
7
8# Load the fastText model
9try:
10 model = fasttext.load_model(model_path)
11 print("Model loaded successfully!")
12except ValueError as e:
13 print(f"Error loading model: {e}")
14 print("Ensure the file exists and is a valid fastText binary model.")
15 model = None # Set model to None if loading fails
16
17
18if model:
19 # --- Get Word Vector ---
20 word = "پاکستان" # Example Urdu word
21 print(f"\nVector for '{word}':")
22 try:
23 vector = model.get_word_vector(word)
24 print(f"Shape: {vector.shape}")
25 print(f"First 10 dimensions: {vector[:10]}")
26 except ValueError as e:
27 print(f"Error getting vector for '{word}': {e}. Word might be too short or have no valid subwords.")
28
29
30 # --- Find Nearest Neighbors (Similar Words) ---
31 word_for_neighbors = "اردو" # Example Urdu word
32 print(f"\nWords similar to '{word_for_neighbors}':")
33 try:
34 # Get top 10 most similar words
35 neighbors = model.get_nearest_neighbors(word_for_neighbors, k=10)
36 if neighbors:
37 print(neighbors)
38 else:
39 print(f"No similar words found for '{word_for_neighbors}'.")
40 except ValueError as e:
41 print(f"Error finding similar words for '{word_for_neighbors}': {e}. Word might not be valid.")
42
43
44 # --- Calculate Similarity Between Two Words (Manual Cosine Similarity) ---
45 word1 = "علم" # Example word 1
46 word2 = "روشنی" # Example word 2
47 print(f"\nSimilarity between '{word1}' and '{word2}':")
48 try:
49 vec1 = model.get_word_vector(word1)
50 vec2 = model.get_word_vector(word2)
51
52 # Calculate cosine similarity
53 norm1 = np.linalg.norm(vec1)
54 norm2 = np.linalg.norm(vec2)
55
56 if norm1 > 0 and norm2 > 0:
57 cosine_similarity = np.dot(vec1, vec2) / (norm1 * norm2)
58 print(f"Cosine similarity: {cosine_similarity}")
59 else:
60 print("Cannot compute similarity: zero vector detected for one or both words.")
61 except ValueError as e:
62 print(f"Error calculating similarity between '{word1}' and '{word2}': {e}. One or both words might not be valid.")
63
64 # --- Using the .vec file (Optional) ---
65 # The .vec file contains just the word vectors for words in the vocabulary.
66 # It can be loaded by other libraries like Gensim or spaCy.
67 # Note: This method *does not* utilize fastText's subword capabilities for OOV words.
68 # For fastText specific features, use the .bin file.
69 # Example (using gensim - requires gensim installation):
70 # from gensim.models import KeyedVectors
71 # vec_file_path = "path/to/your/downloaded/urdu_fasttext.vec"
72 # try:
73 # # Load vectors in Word2Vec text format
74 # word_vectors = KeyedVectors.load_word2vec_format(vec_file_path, binary=False)
75 # print(f"\nLoaded {len(word_vectors.key_to_index)} vectors from .vec file using Gensim.")
76 # # Example: Find similar words using Gensim
77 # # print(word_vectors.most_similar("اردو"))
78 # except Exception as e:
79 # print(f"Error loading .vec file with Gensim: {e}")
80
81
82else:
83 print("\nModel could not be loaded. Usage examples are skipped.")
84
85
86**Steps after creating the Model Card content:**
87
881. **Create a Model Repository on Hugging Face:** Go to huggingface.co, log in, click your profile picture -> "New model".
892. **Name your Model:** Choose a descriptive name (e.g., `urdu-fasttext-word-embeddings`).
903. **Set Visibility:** Choose Public or Private.
914. **Create Model:** This creates an empty repository.
925. **Upload Files:** Go to the "Files" tab of your new repository. You can either:
93 * Click "Add file" and upload `urdu_fasttext.bin`, `urdu_fasttext.vec`, and your training script file.
94 * Or, clone the repository locally and push the files using Git.
956. **Edit Model Card:** Go to the "Model card" tab. This is where you paste and format the content prepared above. You can edit it directly in the browser using Markdown.
967. **Fill in Placeholders:** Go through the content and replace all `[ ... ]` placeholders with your specific details (vector size, epochs, dataset source, license, your name, etc.).
978. **Format with Markdown:** Use the formatting options (headers, bold, code blocks) to make the card readable.
989. **Save Model Card:** Save the changes.
99
100Your model will then be available on Hugging Face with the documentation you've provided.