Sentiment Analysis on Tweets with Spark, Kafka, and Spark NLP

Multiclass sentiment classification of tweets across four classes (positive, negative, uncertainty, litigious), implemented as a batch training plus streaming inference pipeline: Apache Spark trains a TF-IDF and Logistic Regression model in batch, then Apache Kafka and Spark Structured Streaming score the held-out test set in micro-batches.

---

Architecture

The system separates model training from model serving.

        Kaggle dataset (1M tweets, CSV)
                    |
     Spark batch: clean, filter EN, split 80/20
                    |
      +-------------+----------------+
      |                              |
 train set (697,501)          test set (173,809)
      |                              |
 fit Pipeline:                exported as JSON
 Spark NLP -> TF-IDF                 |
 -> Logistic Regression       Python Kafka producer
      |                              |
      |                    Kafka topic "tweets"
      |                       (4 partitions)
      |                              |
      +------> Spark Structured Streaming
                    |
        model.transform on each micro-batch
                    |
         predictions appended to Parquet
                    |
   MLlib evaluation: accuracy, precision, recall

Training is expensive and runs offline. Inference is cheap and runs on the stream. Sending the test set through Kafka means the evaluation measures the whole system, not just the model.

---

Tech stack

Component	Version	Role
Apache Spark	3.4.1	Distributed compute (PySpark, local mode)
Spark NLP	5.3.3	Text preprocessing (tokenization, normalization)
Apache Kafka	3.7.0	Stream transport, KRaft mode (no ZooKeeper)
Spark MLlib	3.4.1	Feature extraction, classifier, evaluation
Python	3.x	Notebook, Kafka producer

---

Dataset

Sentiment Dataset with 1 Million Tweets (Kaggle).

Cleaning kept English tweets only, dropped nulls and empty texts, and retained the text and label columns.

Split	Records	Share
Cleaned dataset	871,310	100%
Positive	248,516	28.5%
Negative	244,146	28.0%
Uncertainty	198,586	22.8%
Litigious	180,062	20.7%
Training (80%)	697,501
Test (20%)	173,809

Fixed random seed (42) for reproducibility.

---

Pipeline

Nine stages combined into a single Spark ML Pipeline, so training and streaming inference share the exact same transformations:

1. DocumentAssembler (Spark NLP)
2. Tokenizer (Spark NLP)
3. Normalizer (Spark NLP): lowercasing, punctuation removal
4. Finisher (Spark NLP)
5. StopWordsRemover (MLlib)
6. HashingTF (MLlib): 10,000 features via hashing trick
7. IDF (MLlib)
8. StringIndexer (MLlib)
9. LogisticRegression (MLlib): multinomial, maxIter=20, regParam=0.01

---

Results

Evaluation on 173,809 held-out tweets streamed through Kafka, computed with pyspark.mllib.evaluation.MulticlassMetrics.

Metric	Value
Accuracy	0.9567
Weighted precision	0.9567
Weighted recall	0.9567
Weighted F1	0.9567

Per-class performance:

Class	Precision	Recall	F1
Positive	0.958	0.966	0.962
Negative	0.955	0.953	0.954
Uncertainty	0.946	0.951	0.948
Litigious	0.969	0.956	0.962

Confusion matrix rows sum to 173,809, confirming the streaming job scored the full test set.

---

How to run

Open the notebook in Google Colab and run cells top to bottom.

The first section installs Java, the Python dependencies, and downloads the Kafka binaries. Section 2 starts a single-broker Kafka in KRaft mode and creates the tweets topic. Total runtime is approximately 15 to 25 minutes on a standard Colab VM.

Requirements are handled inside the notebook:

• Java 17
• PySpark 3.4.1
• Spark NLP 5.3.3
• kafka-python 2.0.2
• kagglehub

---

Project structure

.
├── Big Data Tweet Sentiment Analysis.ipynb   # Main notebook
├── README.md
└── .gitignore

---

Author

Hakeem Anibi