NLP Pipeline Explained – Text Preprocessing, Tokenisation and Word Embeddings

Every natural language processing application — sentiment analysis, chatbots, document classification, named entity recognition, question answering — begins with the same fundamental challenge: converting unstructured human language into numerical representations that machine learning models can process. The NLP pipeline is the sequence of transformations that performs this conversion. Understanding each stage deeply, and knowing which components to include or omit for a given task, is a core NLP competency. This guide walks through the complete pipeline from raw text to model-ready features, covering both classical and modern (transformer-based) approaches.

This guide is the technical foundation for the interview questions in our NLP Interview Q&A and connects to the deep learning architectures that power modern NLP, covered in our Neural Network Architectures guide. The feature engineering perspective on text data is covered in our Feature Engineering guide, and the transfer learning techniques for fine-tuning language models are in our Transfer Learning and Fine-Tuning guide.

Stage 1 — Text Cleaning and Normalisation

Raw text from the web, social media, or documents contains noise that degrades model performance: HTML tags, special characters, inconsistent casing, contractions, excessive whitespace, and boilerplate. Text cleaning is the first and most task-dependent pipeline stage — what you remove matters and should be driven by your use case. Removing punctuation is appropriate for topic classification; it destroys information for sentiment analysis (exclamation marks signal intensity) and named entity recognition (periods distinguish sentence boundaries from abbreviations).

import re
import unicodedata

def clean_text(text, task='classification'):
    # 1. Decode HTML entities and strip tags
    text = re.sub(r'&', '&', text)
    text = re.sub(r'<', '<', text)
    text = re.sub(r'<[^>]+>', ' ', text)

    # 2. Unicode normalisation
    text = unicodedata.normalize('NFC', text)

    # 3. Handle contractions
    contractions = {
        "don't": "do not", "can't": "cannot", "won't": "will not",
        "it's": "it is", "i'm": "i am", "you're": "you are",
        "isn't": "is not", "aren't": "are not", "wasn't": "was not",
    }
    for contraction, expansion in contractions.items():
        text = text.replace(contraction, expansion)

    # 4. Lowercasing (skip for NER)
    if task != 'ner':
        text = text.lower()

    # 5. Remove URLs and emails
    text = re.sub(r'http\S+|www\.\S+', ' URL ', text)
    text = re.sub(r'\S+@\S+\.\S+', ' EMAIL ', text)

    # 6. Remove special characters
    if task == 'classification':
        text = re.sub(r'[^a-z0-9\s]', ' ', text)

    # 7. Normalise whitespace
    text = re.sub(r'\s+', ' ', text).strip()
    return text

Stage 2 — Tokenisation

Tokenisation splits text into the basic units (tokens) that the model will process. The choice of tokenisation strategy profoundly affects model performance and vocabulary size. The three main families are word-level, character-level, and subword tokenisation.

Word-level tokenisation splits on whitespace and punctuation. Simple and interpretable but suffers from out-of-vocabulary (OOV) words and vocabulary explosion for morphologically rich languages.

Subword tokenisation is the standard in modern NLP — it splits words into meaningful subunits based on corpus frequency. A vocabulary of 30,000–50,000 subword tokens can represent any text without OOV.

from transformers import AutoTokenizer

# BERT tokeniser (WordPiece)
bert_tok = AutoTokenizer.from_pretrained('bert-base-uncased')
tokens   = bert_tok.tokenize('unhappiness is counterproductive')
print(tokens)
# ['un', '##happiness', 'is', 'counter', '##productive']

# Batch encode with padding and truncation
batch = bert_tok(
    ['This is sentence one.', 'Shorter.', 'A longer third sentence here.'],
    padding=True, truncation=True, max_length=128,
    return_tensors='pt'
)
print(batch['input_ids'].shape)    # (3, max_seq_len_in_batch)
print(batch['attention_mask'])     # 1 for real tokens, 0 for padding

Stage 3 — Stemming, Lemmatisation and Stop Word Removal

Stemming crudely removes suffixes using rules: “running” becomes “run”, “studies” becomes “studi” (not a valid word). Fast but loses meaning. Lemmatisation uses morphological analysis and a lexicon to return the dictionary base form: “studies” becomes “study”, “better” becomes “good”. More accurate but slower. Stop word removal deletes high-frequency words (the, a, is, in) — appropriate for TF-IDF and topic modelling, but harmful for tasks where function words matter (machine translation, sentiment where “not” is critical).

import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.corpus import stopwords

nltk.download('wordnet'); nltk.download('stopwords')

stemmer    = PorterStemmer()
lemmatizer = WordNetLemmatizer()
stop_words = set(stopwords.words('english'))

words = ['running', 'studies', 'better', 'geese', 'wolves']
for w in words:
    stem  = stemmer.stem(w)
    lemma = lemmatizer.lemmatize(w, pos='v')
    print(w, '->', 'stem:', stem, '| lemma:', lemma)

# running -> stem: run   | lemma: run
# studies -> stem: studi | lemma: study
# geese   -> stem: gees  | lemma: goose

Stage 4 — Text Representations

Bag of Words and TF-IDF: TF-IDF (Term Frequency-Inverse Document Frequency) reweights word counts by how rare a term is across the corpus: TF-IDF(t,d) = TF(t,d) × log(N / df(t)). Rare, discriminative terms get high scores; common words are downweighted. TF-IDF remains competitive with deep learning on many text classification tasks with small datasets and has the advantage of interpretability.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

tfidf = TfidfVectorizer(
    max_features=50000,
    ngram_range=(1, 2),     # unigrams + bigrams
    min_df=2,
    max_df=0.95,
    sublinear_tf=True,      # apply log(1+tf)
    strip_accents='unicode'
)

# TF-IDF + Logistic Regression: a strong NLP baseline
pipeline = Pipeline([
    ('tfidf', tfidf),
    ('clf',   LogisticRegression(C=1.0, max_iter=1000))
])
pipeline.fit(X_train, y_train)
print(pipeline.score(X_test, y_test))

Contextual embeddings — BERT and beyond: Transformer-based models produce contextual embeddings — the same word gets a different vector depending on its context. “Bank” in “river bank” and “bank account” produce different BERT embeddings. This context-sensitivity is why BERT-based models dramatically outperform Word2Vec on most NLP tasks. For an end-to-end pipeline with BERT, our Transfer Learning guide covers fine-tuning strategies. For NLP-specific interview questions on embeddings, attention, and transformers, our NLP Interview Q&A has 40+ questions. The statistics behind similarity measures connects to the foundations in our Statistics Interview Q&A. The pandas and text-processing Python patterns used in NLP pipelines are covered in our Pandas and NumPy Mastery guide and Python Interview Q&A.