Views
No views yet
1
2Note: You need Python 3.10 or above.
3
41. Create a virtual environment (venv):
5 ```sh
6 python -m venv path-to-venv
7 ```
8 `venv` stands for Virtual Environment.
9
102. Activate the virtual environment:
11 ```sh
12 source path-to-venv/bin/activate
13 ```
14 On Windows:
15 ```sh
16 .\venv\Scripts\activate
17 ```
18
193. Install requirements:
20 ```sh
21 pip install -r requirements.txt
22 ```
23
24## Running
25
26## Data
27
281. Kaggle : https://www.kaggle.com/datasets/dilshaansandhu/international-jobs-dataset/data
29
302. O-net : https://www.onetcenter.org/database.html#individual-files
31
32### Technical Overview
33
34 Excel Processing: Utilizes UnstructuredExcelLoader for extracting text from Excel documents, CSVLoader for extracting text from csv documents, PyPDFLoader for extracting text from pdf documents.
35 Text Chunking: Employs the RecursiveCharacterTextSplitter from LangChain for dividing the extracted text into manageable chunks.
36 Vector Store Creation: Uses chromadb for creating a searchable vector store from text chunks.
37 Answer Generation: Leverages gpt-4-turbo-preview model from OpenAI for generating answers to user queries using the context provided by the uploaded documents.
38
39
40### Indexing
41
42Helping Functions:
43
44`from_web(url)`
45 Description: This function retrieves documents from a web page specified by the given URL.
46 Purpose: It's used to scrape text content from web pages for further processing and analysis.
47 Implementation: It utilizes BeautifulSoup (bs4) for web scraping and the langchain library for document loading.
48
49`from_excel(file_address)`
50
51 Description: This function loads documents from Excel files located either at the specified file address or within a directory specified by the file address.
52 Purpose: It's designed to extract text data from Excel files, which may contain structured or tabular data.
53 Implementation: It utilizes the langchain_community library for loading Excel files.
54
55`from_csv(file_address):`
56
57 Description: This function loads documents from a CSV file located at the specified file address.
58 Purpose: It's used for extracting text data from CSV files, which are commonly used for storing tabular data.
59 Implementation: It utilizes the langchain_community library for loading CSV files.
60
61`from_pdf(file_address):`
62
63 Description: This function loads documents from a PDF file located at the specified file address.
64 Purpose: It's intended for extracting text data from PDF documents.
65 Implementation: It utilizes the langchain_community library for loading PDF files.
66
67`from_text_files(file_address):`
68
69 Description: This function loads documents from text files (.txt) located within a directory specified by the file address.
70 Purpose: It's used to extract text data from multiple text files within a directory.
71 Implementation: It utilizes the langchain_community library for loading text files.
72
73`retriever_from_docs(docs):`
74
75 Description: This function processes the retrieved documents, splitting them into smaller chunks, generating embeddings for each chunk, and storing them in a Chroma vector store.
76 Purpose: It's responsible for preprocessing and embedding the text data for further analysis or retrieval.
77 Implementation: It utilizes various components such as text splitters, embeddings, and Chroma vector stores provided by the langchain library.
78
79
801. Go to `bot/rag_indexing`
812. Run:
82 ```sh
83 python indexing.py <data-sources>
84 ```
85 For example:
86 ```sh
87 python indexing.py <path/to/data/files>/botstreamlit run bot-langchain-chat.py