Information Retrieval (IR) System

An implementation of a custom Information Retrieval Engine and Document Clustering System based on the Vector Space Model (VSM). This project was developed as part of the Information Retrieval laboratory course (Winter Semester 2025–2026) at the Department of Computer Engineering and Informatics (CEID), University of Patras.

The engine is evaluated on the classic Cystic Fibrosis (C.F.) dataset, which comprises a collection of 1,239 medical documents, 20 test queries, and a ground-truth relevance list (rated by medical specialists).

---

Table of Contents

• Project Specifications (Assignment Questions)
• Project Architecture
• File Tree & Descriptions
• System Features
• Requirements
• Installation
• How to Run
• Evaluation Metrics

---

Project Specifications (Assignment Questions)

The project is structured around the five specific questions outlined in the university assignment:

1. Question 1 - Collection Parsing & Inverted Index (1 point): Parse the provided Cystic Fibrosis dataset to create an inverted index. This involves applying necessary text preprocessing, structuring the index effectively for future retrieval, and handling edge cases like missing document IDs to ensure alignment with relevance lists.

2. Question 2 - Vector Space Model (VSM) Implementation (3 points): Develop a custom Vector Space Model (VSM) engine from scratch. Represent both documents and queries as vectors, implementing two different term-weighting schemes: a classic TF-IDF formulation and an alternative normalized scheme. Retrieve and rank the most relevant documents for each query using Cosine Similarity.

3. Question 3 - Evaluation Metrics (1 point): Manually implement the core mathematical evaluation metrics for an Information Retrieval system. Specifically, write custom functions to compute Precision, Recall, F1-score, and Precision@k. Additionally, implement a function to generate Precision-Recall Curves.

4. Question 4 - Hyperparameter Tuning & Comparisons (3 points): Utilize Scikit-Learn's TfidfVectorizer to find the optimal IR model through grid search parameter tuning (testing >30 configurations over parameters like ngram_range, sublinear_tf, min_df, max_df, and norm). Compare the custom-built models from Question 2 against this optimized library implementation using the metrics from Question 3, and evaluate indexing/retrieval execution times.

5. Question 5 - Document Clustering & Dimensionality Reduction (2 points): Perform document clustering to contrast two different representation spaces: sparse TF-IDF vectors (from the optimized model in Q4) and dense semantic embeddings (using Hugging Face's sentence-transformers). Determine the optimal number of clusters ($k$) using methods like the Silhouette Score and the Elbow Method, and visualize the final clustered spaces in 2D using Principal Component Analysis (PCA).

---

Project Architecture

The system is built upon five core experimental phases outlined in the project specification:

1. Document Preprocessing & Indexing: Building a custom Inverted Index mapping terms to documents and calculating Term Frequencies (TF) and Inverse Document Frequencies (IDF).
2. Custom Vector Space Models (VSM): Implementing two distinct tf-idf based term weighting systems (Classic TF-IDF vs. Cosine-Normalized tfc-nfx variant) and ranking using custom Cosine Similarity.
3. Custom Evaluation Metrics: Building a mathematical evaluation library from scratch (Precision, Recall, F1-score, Precision@k, and Precision-Recall Curves).
4. Scikit-Learn TF-IDF Tuning: Hyperparameter optimization via grid-search (>30 combinations) to compare custom VSM modules against a tuned TfidfVectorizer.
5. Semantic Document Clustering: Semantic grouping using K-Means Clustering applied to both TF-IDF sparse space and dense SentenceTransformer embeddings, optimized via Silhouette Scores and visualized using PCA dimensionality reduction.

---

File Tree & Descriptions

informationRetrieval/
├── docs/                                  # Collection of 1,239 document text files (Cystic Fibrosis collection)
├── pyFiles/                               # Secondary/modular files split for Phase 2 functions
│   ├── analyshErwthsewn1.py               # Computes query tf-idf weights for VSM Method 1
│   ├── analyshErwthsewn2.py               # Computes query tf-idf weights for VSM Method 2
│   ├── analyshEurethriou1.py              # Generates document tf-idf vectors for VSM Method 1
│   ├── analyshEurethriou2.py              # Generates document tf-idf vectors for VSM Method 2
│   ├── findDocumentRanks1.py              # Computes Cosine Similarity and ranks docs for Method 1
│   ├── findDocumentRanks2.py              # Computes Cosine Similarity and ranks docs for Method 2
│   ├── printRelevancy1.py                 # Interactive terminal UI to print top matches for Method 1
│   └── printRelevancy2.py                 # Interactive terminal UI to print top matches for Method 2
├── textFiles/                             # Holds text datasets, index files, and experimental outputs
│   ├── Queries.txt                        # List of the 20 evaluation queries
│   ├── Relevant.txt                       # Ground-truth relevance mapping (Query ID -> Relevant Document IDs)
│   ├── inverted_index.json                # Inverted index file containing vocabulary details, TF, and IDF
│   ├── tfidfVectors.json / tfidfVectors2.json  # Precomputed VSM sparse vectors for Method 1 & 2
│   ├── queryVector.json / queryVector2.json    # Precomputed VSM query vectors for Method 1 & 2
│   ├── sortedRelevant1.json / sortedRelevant2.json # Retrieval rankings for each query
│   ├── bestModel.json                     # Hyperparameter details of the optimal Scikit-Learn VSM
│   └── results.txt                        # Detailed benchmark comparisons (Metrics & Processing times)
├── clustering.py                          # KMeans clustering pipeline comparing TF-IDF vs. Dense Sentence Embeddings
├── createEurethrio.py                     # Script to read documents, parse terms, and build the Inverted Index
├── evaluationMetricsFunctions.py         # Custom library implementing evaluation metrics and plotting PR curves
├── tfidf_tuning.py                        # Execution runner for hyperparameter grid search and multi-model benchmarking
├── vectorizer1.py                         # Single-file orchestrator for Custom VSM Method 1
├── vectorizer2.py                         # Single-file orchestrator for Custom VSM Method 2 (tfc-nfx weighting)
├── requirements.txt                       # List of required python libraries
└── README.md                              # Project documentation (this file)

Key Python Script Roles:

• createEurethrio.py (Q1): Reads the document text corpus from docs/, sanitizes tokens, resolves potential indexing gaps caused by missing document IDs, calculates overall term frequencies and IDFs, and dumps the structure to inverted_index.json.
• vectorizer1.py & vectorizer2.py (Q2): Implement custom vector representations.
  • Method 1 (vectorizer1.py): Uses standard tf-idf formulation.
  • Method 2 (vectorizer2.py): Uses the normalized query formulation (0.5 + 0.5 * (tf / max_tf)) * idf combined with Cosine Normalization for document term weights (represented as the classic tfc-nfx scheme). Both rank matching documents per query using a custom-implemented Cosine Similarity.
• evaluationMetricsFunctions.py (Q3): Implements mathematical evaluation indicators without external model dependencies, including custom Precision, Recall, F1-score, Precision@k, and a dynamic plot routine for the Precision-Recall curve.
• tfidf_tuning.py (Q4): Performs grid search tuning over 48 permutations of Scikit-Learn's TfidfVectorizer parameters (ngram_range, sublinear_tf, min_df, max_df, norm) to find the best configuration based on Mean Precision@10. Saves comparative reports (custom models vs. scikit-learn best model including indexing/retrieval execution times) in results.txt.
• clustering.py (Q5): Applies the KMeans clustering algorithm to the document corpus under two feature spaces: sparse TF-IDF and dense embeddings extracted from Hugging Face's sentence-transformers/all-MiniLM-L6-v2. Computes optimal clusters ($k \in [2..10]$) using Silhouette Scores, reduces dimensionality to 2D using PCA, and plots the results.

---

System Features

• Zero-Dependency Core Metrics: Core IR ranking, similarity calculations, and evaluation performance plots are written using base Python, Math, and NumPy to comply with academic strictness.
• Advanced Term Weighting Schemes: Comparative assessment of custom implementations (Classic vs. Cosine-Normalized tfc-nfx) and optimized library setups.
• Deep Semantic Clustering: Directly compares traditional lexicon-based sparse VSM clustering against modern, pre-trained transformer-based deep contextual embeddings.
• Extensive Benchmarking: High-fidelity timing instrumentation for both indexing and retrieval execution.

---

Requirements

• Python 3.10+
• Dependencies listed in requirements.txt:
  • scikit-learn
  • numpy
  • sentence-transformers
  • matplotlib

---

Installation

1. Clone this repository:

   git clone https://github.com/jimfil/informationRetrieval/
   cd informationRetrieval

2. Install the necessary libraries:

   pip install -r requirements.txt

---

How to Run

Step 1: Create the Inverted Index

Initialize the system vocabulary and document associations:

python createEurethrio.py

Step 2: Run Custom Retrieval & Similarity Engine

You can run either custom VSM schemes. Run the files and press Enter to proceed, or q to quit:

• For Classic TF-IDF (Method 1):

  python vectorizer1.py

• For Optimized tfc-nfx Weighting (Method 2):

  python vectorizer2.py

Step 3: Run Hyperparameter Tuning & Cross-Model Benchmarking

Execute the multi-model pipeline comparing custom implementations against scikit-learn variations:

python tfidf_tuning.py

Outputs optimal grid parameters to textFiles/bestModel.json and a full evaluation scorecard to textFiles/results.txt.

Step 4: Perform Document Clustering

Run KMeans clustering to compare TF-IDF vs. transformer-based representation spaces:

python clustering.py

This will open a detailed matplotlib window plotting PCA projections of the optimal clusters for both representations.

---

Evaluation Metrics

Each execution of tfidf_tuning.py computes:

• Precision@10: Ratio of relevant items within the top 10 retrieved results.
• Recall: Proportion of all truly relevant items that were successfully retrieved.
• F1-Score: Harmonic mean of precision and recall.
• Precision-Recall Curve: Visualizing the precision-recall trade-off across different thresholds.