Language-Model

A collection of language modelling experiments and NLP utilities, covering smoothing techniques, evaluation, data extraction, and syntactic parsing.

Contents

1. Language Model with Smoothing Techniques

Implementation of n-gram language models with different smoothing techniques to handle unseen n-grams and improve probability estimates.

2. Language Model Evaluation

Scripts to evaluate trained language models, e.g. using perplexity and other standard evaluation metrics.

3. Wikipedia Data Extraction

Tools/scripts to extract and preprocess text data from Wikipedia dumps for use as training corpora.

4. Tutorial Slides

Slide deck(s) covering the theory and practical implementation of the language models in this repository.

5. POS and Dependency Parser

Part-of-speech tagging and dependency parsing utilities used for syntactic analysis of the text corpora.

6. Google Ngram

Utilities for working with the Google Ngram dataset, including data loading and analysis.

Getting Started

Clone the repository and explore each module — see the comments/README within each subfolder for usage details specific to that component.

git clone <repo-url>
cd Language-Model

Notes

Additional details for each component are documented within the corresponding scripts/notebooks.