GitHub - SadiqCodex/Next_Word_Prediction_Model: LSTM-based NLP application for next-word prediction and sentence completion, built with TensorFlow, Keras, and Streamlit using a custom quote dataset.

Overview
Features
Project Preview
How It Works
Dataset
NLP Pipeline
Model Architecture
Model Specifications
Inference Pipeline
Temperature & Top-K Sampling
Application Modes
Project Structure
Tech Stack
Installation
Usage
Model Artifacts
Training
Reproducibility
Limitations
Future Improvements
Deployment
Contributing
License
Author

Overview

This project implements a next-word and sentence completion system using a custom-trained LSTM neural network. The model was trained on a dataset of 3,038 famous quotes, learning sequential word patterns through n-gram sequence modeling.

The core problem: given a partial text input, predict the most likely word(s) that follow. This is a classic language modeling task, approached here with a recurrent architecture rather than a modern transformer, making it a practical demonstration of foundational NLP and deep learning concepts.

Conceptual pipeline:

User Input Text
      ↓
Lowercase + Tokenize
      ↓
Convert Words → Token IDs
      ↓
Pad Sequence to Fixed Length
      ↓
LSTM Model → Probability Distribution (8,979 values)
      ↓
Temperature Scaling
      ↓
Top-K Sampling
      ↓
Predicted Word / Sentence Completion

The application runs entirely locally — no external LLM API is called during inference.

Features

All features listed below are verified from the actual app.py implementation.

Feature	Description
⚡ Auto Completion	Ghost-text style next-word suggestion; accept with TAB or the Accept button
💬 Auto Suggestion	Multiple full sentence completions shown as clickable buttons
🎯 Top-K Control	Slider to control how many candidate words are considered (1–10)
🌡️ Temperature	Slider to control prediction creativity/randomness (0.1–2.0)
📏 Prediction Length	Control how many words are generated per suggestion (3–20 words)
🔄 Auto Predict Toggle	Enable or disable real-time prediction as you type
📊 Show Probabilities	Display confidence percentage of the top prediction
🔢 Word & Character Counter	Live word and character count in the writing editor
📈 Model Analytics Dashboard	Displays vocabulary size, parameter count, architecture layers, and current mode
🔒 Local Inference	All predictions run on your machine — no external API calls

Project Preview

The application features a clean glassmorphism-style UI with a central writing editor, a mode switcher between Auto Completion and Auto Suggestion, a live stats bar showing vocabulary size and model parameters, and a sidebar for adjusting prediction settings.

How It Works

Training Phase

Raw Quotes (CSV)
      ↓
Lowercasing
      ↓
Punctuation Removal
      ↓
Keras Tokenizer (word → integer ID)
      ↓
N-Gram Sequence Generation
      ↓
Pre-Padding to max_len - 1
      ↓
LSTM Model Training
      ↓
Save: model (.h5) + tokenizer (.pkl) + max_len (.pkl)

Inference Phase

User Input
      ↓
Lowercase + Tokenize
      ↓
Pad Sequence
      ↓
LSTM Forward Pass
      ↓
Softmax Probability Distribution
      ↓
Temperature Scaling
      ↓
Top-K Sampling
      ↓
Next Word / Sentence Completion

Dataset

Property	Value
File	`qoute_dataset.csv`
Rows	3,038 quotes
Columns	`quote`, `Author`
Content	Famous inspirational quotes from authors including Albert Einstein, J.K. Rowling, Marilyn Monroe, Jane Austen, and others

The dataset is used exclusively as a text corpus for training. Only the quote column is used; the Author column is not used during training or inference.

N-gram sequence example (illustrative):

Quote: "the future of"

Training pairs generated:
  [the]              → future
  [the, future]      → of
  [the, future, of]  → ...

Every prefix subsequence of every tokenized quote becomes a supervised training sample, producing a large number of (input sequence → next word) pairs from the 3,038 quotes.

NLP Pipeline

Lowercasing

All quotes are converted to lowercase using str.lower() to reduce vocabulary size and normalize word representations.

Punctuation Removal

Python's str.maketrans with string.punctuation strips all punctuation characters from the text before tokenization.

Tokenization

Keras Tokenizer is fitted on the preprocessed corpus. Each unique word is assigned an integer ID. The vocabulary size is 8,978 unique words. The top 10 most frequent words (verified from notebook output) are:

the(1), you(2), to(3), and(4), a(5), i(6), is(7), of(8), that(9), it(10)

N-Gram Sequence Generation

For each tokenized quote, every possible prefix subsequence is extracted as an input, with the next token as the label. This creates a large set of variable-length supervised training pairs.

Padding

All input sequences are padded to a fixed length of max_len - 1 using pre padding (zeros prepended to the left), ensuring uniform input shape for the LSTM.

Model Architecture

Input Token IDs  (padded sequence of length max_len - 1)
        │
        ▼
Embedding Layer  (8,979 vocab → 50-dimensional dense vectors)
        │
        ▼
LSTM Layer       (128 units — learns sequential word dependencies)
        │
        ▼
Dense Layer      (8,979 units + Softmax activation)
        │
        ▼
Probability Distribution over Vocabulary

Layer	Configuration	Purpose
Embedding	vocab=8,979, output_dim=50	Maps token IDs to dense vector representations
LSTM	units=128	Learns temporal/sequential patterns in word sequences
Dense	units=8,979, activation=softmax	Produces a probability for each vocabulary word

Model Specifications

Property	Value
Architecture	LSTM Neural Network
Framework	TensorFlow 2.21 / Keras 3.14
Vocabulary Size	8,979 (word_index size + 1)
Embedding Dimension	50
LSTM Units	128
Output Dimension	8,979 (softmax over full vocabulary)
Loss Function	Categorical Cross-Entropy
Optimizer	Adam
Total Parameters	Displayed dynamically in the app via`model.count_params()`
Training Platform	Apache Databricks
Model File	`quote_lstm_model.h5`

Note: The exact parameter count is computed at runtime from the loaded model. The app displays it live in the sidebar and analytics dashboard.

Inference Pipeline

When a user types text in the editor, the following steps occur in predict_next():

Normalize — strip and lowercase the input text
Tokenize — convert words to integer IDs using the saved tokenizer; unknown words are silently dropped
Pad — pad the sequence to max_len - 1 using pre-padding
Predict — run a forward pass through the LSTM model to get a probability distribution over 8,979 vocabulary entries
Temperature scaling — if temperature ≠ 1.0, apply log-scaling to sharpen or flatten the distribution
Top-K selection — sort by probability, return the top-K words with their probabilities

For sentence generation (generate_sentence()), this process repeats iteratively, appending the top-1 predicted word to the current text until the desired length is reached or a sentence-ending token is encountered.

Temperature & Top-K Sampling

Temperature

Temperature controls how the probability distribution is shaped before sampling.

preds = np.log(np.clip(preds, 1e-10, 1.0)) / max(temperature, 0.01)
preds = np.exp(preds) / np.sum(np.exp(preds))

Temperature	Effect
Low (0.1 – 0.5)	Distribution sharpens — model picks the most probable word more consistently
Medium (~1.0)	Distribution unchanged — balanced between focus and diversity
High (1.5 – 2.0)	Distribution flattens — more varied and creative (but less coherent) output

Top-K Sampling

Top-K limits the candidate pool to the K highest-probability words before returning results. A lower K (e.g., 1) always returns the single most likely word. A higher K (e.g., 10) returns more diverse candidates for the suggestion mode.

In Auto Suggestion mode, the app generates multiple sentence completions by branching from each of the top-K next words, then extending each branch using greedy decoding at temperature 0.75.

Application Modes

⚡ Auto Completion Mode

The model predicts a single next word displayed as ghost text below the editor. The user can:

Press TAB or click "Accept Prediction" to append the word to their text
The model immediately predicts the next word after acceptance, chaining completions

💬 Auto Suggestion Mode

The model generates multiple full sentence completions (up to Top-K suggestions). Each suggestion is displayed as a clickable button. Clicking a suggestion inserts the full completion into the editor.

Project Structure

Next_Word_Prediction_Model/
│
├── app.py                   ← Streamlit web application (UI + inference logic)
├── word_pred_model.ipynb    ← Model training notebook (executed on Apache Databricks)
├── qoute_dataset.csv        ← Dataset: 3,038 famous quotes
├── quote_lstm_model.h5      ← Trained LSTM model weights (HDF5 format)
├── tokenizer.pkl            ← Fitted Keras Tokenizer (word ↔ integer mapping)
├── max_len.pkl              ← Maximum sequence length used during training
├── Word_Predictor_Model.png ← Application screenshot
├── requirements.txt         ← Python dependencies (pinned versions)
├── LICENSE                  ← MIT License
├── README.md                ← This file
└── .gitignore               ← Git ignore rules

Tech Stack

Machine Learning

TensorFlow 2.21 / Keras 3.14
LSTM (Long Short-Term Memory) architecture

NLP

Keras Tokenizer (word-level tokenization)
N-gram sequence generation
Pre-padding with pad_sequences

Data Processing

Pandas 3.0
NumPy 2.4

Application

Streamlit 1.58

Training Environment

Apache Databricks (cloud-based notebook execution)

Installation

1. Clone the repository

git clone https://github.com/SadiqCodex/Next_Word_Prediction_Model.git
cd Next_Word_Prediction_Model

2. Create and activate a virtual environment

Windows:

python -m venv venv
venv\Scripts\activate

Linux / macOS:

python3 -m venv venv
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

The requirements.txt includes pinned versions of all dependencies including TensorFlow 2.21, Keras 3.14, Streamlit 1.58, NumPy 2.4, and Pandas 3.0. Python 3.12 was used in the training environment (Databricks). Python 3.10+ is recommended for local use.

Usage

Run the application

streamlit run app.py

The app will open at: http://localhost:8501

Workflow

Open the app in your browser
Select a prediction mode: Auto Completion or Auto Suggestion
Type a few words in the writing editor (e.g., "the future of")
Auto Completion: ghost text appears below the editor — press TAB or click "Accept Prediction" to insert it
Auto Suggestion: multiple sentence completions appear as buttons — click any to insert
Use the sidebar to adjust Temperature, Top-K, and Prediction Length
Toggle Show Probabilities to see the confidence score of the top prediction

Example input: "the future of" — the model will predict the next word based on patterns learned from the quotes dataset. Actual output depends on the trained model weights and the selected temperature/top-k settings.

Model Artifacts

Three files are required for inference. All three must be present in the project root.

`quote_lstm_model.h5`

The trained LSTM model saved in HDF5 format. Contains all layer weights and the model architecture. Loaded via tensorflow.keras.models.load_model.

`tokenizer.pkl`

The fitted Keras Tokenizer object serialized with pickle. Contains the complete word-to-integer mapping (word_index) and the reverse mapping (index_word) used to convert predicted token IDs back to words.

`max_len.pkl`

The maximum sequence length determined during training. Used to pad all input sequences to a consistent length (max_len - 1) before passing them to the model.

All three artifacts are loaded once at startup using @st.cache_resource to avoid reloading on every user interaction.

Training

The model was trained using the notebook word_pred_model.ipynb on Apache Databricks.

Training steps (verified from notebook):

Load qoute_dataset.csv with Pandas
Extract the quote column (3,038 rows)
Lowercase all text
Remove punctuation using str.maketrans + string.punctuation
Fit a Keras Tokenizer with num_words=8978 on the cleaned corpus
Convert all quotes to integer sequences
Generate all prefix n-gram subsequences as (X, Y) training pairs
Pad all X sequences to max_len - 1 using pre-padding
One-hot encode Y labels (categorical)
Build and compile the model: Embedding → LSTM → Dense(softmax)
Train with categorical_crossentropy loss and adam optimizer
Save model as quote_lstm_model.h5, tokenizer as tokenizer.pkl, max_len as max_len.pkl

Specific training hyperparameters (epochs, batch size, validation split) are not recorded in the notebook output and are therefore not stated here.

Reproducibility

The repository includes the original training notebook (word_pred_model.ipynb) and all serialized model artifacts. To reproduce training:

The notebook was executed on Apache Databricks with Python 3.12 and TensorFlow 2.21
The dataset file qoute_dataset.csv is included in the repository
Exact reproduction of model weights may vary due to random initialization and hardware differences
For inference only, the pre-trained artifacts (quote_lstm_model.h5, tokenizer.pkl, max_len.pkl) are sufficient — no retraining is needed

Limitations

Small, domain-specific dataset: 3,038 quotes is a relatively small corpus. The model's vocabulary and style are constrained to the language patterns found in inspirational quotes.
Out-of-vocabulary words: Words not seen during training are silently dropped by the tokenizer, which can degrade prediction quality for inputs containing uncommon words.
No transformer architecture: This model uses a single LSTM layer. It does not have the attention mechanisms or long-range contextual understanding of modern transformer-based language models.
Greedy/top-K decoding only: The inference pipeline uses top-K sampling without beam search, which can produce locally optimal but globally suboptimal completions.
Punctuation stripped at training time: Since punctuation was removed during preprocessing, the model does not generate punctuation in its predictions.
Generated text quality: Output quality is directly bounded by the training data. Predictions reflect the style and vocabulary of the quotes dataset.

Future Improvements

Larger and more diverse training corpus
Bidirectional LSTM or stacked LSTM layers
Attention mechanism over the LSTM hidden states
Transformer-based architecture (e.g., GPT-style decoder)
Beam search decoding for higher-quality sentence completions
Proper handling of punctuation in both training and inference
Formal model evaluation metrics (perplexity, BLEU)
Experiment tracking with MLflow or Weights & Biases
Docker containerization for reproducible deployment
CI/CD pipeline for automated testing and deployment

Deployment

The application is deployed and publicly accessible at:

https://next-word-predictor-ai.streamlit.app/

The application can also be self-hosted on any platform that supports Streamlit applications, including:

Platform	Notes
Streamlit Community Cloud	Recommended — free, connects directly to GitHub
Hugging Face Spaces	Supports Streamlit apps with model file hosting
Render	Simple Python web service deployment
Railway	Fast deployment via GitHub integration
AWS EC2	Full control for production-grade hosting

Contributing

Contributions are welcome.

Fork → Create Branch → Make Changes → Test Locally → Commit → Pull Request

Please ensure any changes to app.py are tested with the existing model artifacts before submitting a pull request.

License

This project is licensed under the MIT License — see LICENSE for details.

Author

Sadik Mohammad

GitHub: @SadiqCodex

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
Word_Predictor_Model.png		Word_Predictor_Model.png
app.py		app.py
max_len.pkl		max_len.pkl
qoute_dataset.csv		qoute_dataset.csv
quote_lstm_model.h5		quote_lstm_model.h5
requirements.txt		requirements.txt
tokenizer.pkl		tokenizer.pkl
word_pred_model.ipynb		word_pred_model.ipynb

Latest commit

History

Folders and files

Repository files navigation

Table of Contents