News Topic Classification

This is a natural language processing project (NLP). Classifying English news headlines into four topics: Business, Science and Technology, Sports, and World News. The project compares two ways of turning text into features, TF-IDF and Skip-gram Word2Vec, against eight model types, and it does so across three preprocessing regimes. Macro F1 is the main metric because the training data is imbalanced.

The notebook in notebooks/ is the original record of the work, with all cell outputs kept. The src/ folder holds the same pipeline refactored into Python modules so the experiments can be run from the command line.

Dataset

Training comes from Training_data_3.csv: 102,002 headlines across four classes. The classes are uneven.

Class	Count	Share
Science and Technology	43,961	43.1%
Business	26,834	26.3%
Sports	18,287	17.9%
World News	12,920	12.7%

Final evaluation uses a separate file, Test_data.csv, with 12,000 headlines split evenly at 3,000 per class. The test set is only touched once, after the model is chosen on validation. An early check of the raw corpus found that stopwords make up 30.49% of all tokens, which motivated the preprocessing choices below.

Preprocessing

Three versions of the text are built and compared:

• No preprocessing. HTML is stripped and whitespace is normalised, nothing more.
• Extreme. Lowercase, tokenize, keep alphabetic non-stopword tokens, then stem and lemmatize.
• Optimum. Lowercase, remove URLs and non-letters, drop stopwords but keep "no", "not", and "nor", then lemmatize and keep tokens longer than one character.

Models and features

Two representations were paired with the models. TF-IDF (20,000 features, unigrams and bigrams) was used with Logistic Regression and with a dense neural network whose first hidden layer has more than 128 units. Skip-gram Word2Vec (100 dimensions, window 5, trained on the training headlines) initialised the embedding layer of six sequence models: SimpleRNN, GRU, LSTM, and the bidirectional version of each. Hyperparameters were tuned by hand on the validation split, with macro F1 as the selection metric.

Results

The strongest validation model was a Skip-gram Bidirectional LSTM. The final chosen model, trained on optimum preprocessing, was then evaluated once on the held-out test set.

Held-out test set (Skip-gram + Bidirectional LSTM, optimum preprocessing)

Accuracy 0.9222, macro F1 0.9222.

Class	Precision	Recall	F1	Support
Business	0.89	0.90	0.89	3,000
Science and Technology	0.89	0.92	0.91	3,000
Sports	0.97	0.98	0.97	3,000
World News	0.95	0.89	0.92	3,000

Validation highlights

• Best experiment: Skip-gram Bidirectional LSTM, accuracy 0.9327, macro F1 0.9289.
• Weakest experiment: Skip-gram SimpleRNN on extreme preprocessing, accuracy 0.5644, macro F1 0.3778.

The full validation comparison across all preprocessing versions, representations, and models is saved to results/model_comparison.csv and plotted in results/figures/.

What the numbers show. Optimum and light preprocessing beat extreme preprocessing on these headlines. Headlines are short, so aggressive stemming and heavy cleaning strip out words that carry the topic. The gap is widest for the weaker models: SimpleRNN drops sharply under extreme preprocessing, while GRU, LSTM, and the bidirectional variants stay strong. Bidirectional models generally edge out their one-directional counterparts because they read each headline forwards and backwards.

Selected figures:

Repository layout

news-topic-classification/
├── notebooks/   original notebook with all outputs preserved
├── src/         the pipeline as Python modules
│   ├── config.py          paths, seeds, and every hyperparameter
│   ├── preprocessing.py   text cleaning, the three versions, label encoding
│   ├── features.py        TF-IDF, Word2Vec, tokenizer, embedding matrix
│   ├── models.py          the DNN and the sequence model builders
│   ├── evaluate.py        metric printout and confusion-matrix figures
│   └── train.py           runs the full experiment set end to end
├── report/      NLP.pdf, the IEEE-format write-up
└── results/     comparison table and figures

Reproduce

1. Create an environment and install the dependencies.

   pip install -r requirements.txt
   

2. Put Training_data_3.csv and Test_data.csv in a data/ folder, or point the TRAIN_CSV and TEST_CSV environment variables at them.

3. Run the pipeline.

   python src/train.py
   

The run trains every experiment, writes the comparison table and figures to results/, saves the trained artifacts, and prints the final test metrics. Training the dense network on TF-IDF converts the sparse matrix to a dense array, which needs a fair amount of memory on a corpus this size.

Report

report/NLP.pdf is the IEEE-format paper for this project, with the full method, tables, and discussion.