Regular expressions (regex) are a mini-language for pattern matching in text. They are indispensable for data cleaning, log parsing, form validation, and text extraction. Python’s built-in re module provides a full regex engine. This guide covers everything from basic patterns to advanced lookaheads, with real-world data science examples throughout.
Core re Functions
import re
text = "DataExpertise published 42 articles in 2025 and 58 articles in 2026."
# re.search — find first match anywhere in string
m = re.search(r'\d+', text)
print(m.group()) # '42'
print(m.start(), m.end()) # 24 26
# re.match — match only at the START of string
m = re.match(r'Data', text) # matches
m = re.match(r'\d+', text) # None — doesn't start with digit
# re.fullmatch — entire string must match
re.fullmatch(r'\d{4}', '2026') # matches
re.fullmatch(r'\d{4}', '2026x') # None
# re.findall — return list of all matches
numbers = re.findall(r'\d+', text)
print(numbers) # ['42', '2025', '58', '2026']
# re.finditer — return iterator of match objects
for m in re.finditer(r'\d+', text):
print(f'Found {m.group()} at position {m.start()}-{m.end()}')
# re.sub — replace matches
clean = re.sub(r'\d+', 'N', text)
print(clean) # 'DataExpertise published N articles in N and N articles in N.'
# re.split — split on pattern
parts = re.split(r'\s+', 'Hello World Python')
print(parts) # ['Hello', 'World', 'Python']
Pattern Syntax Reference
# Character classes
r'\d' # any digit [0-9]
r'\D' # any non-digit
r'\w' # word character [a-zA-Z0-9_]
r'\W' # non-word character
r'\s' # whitespace (space, tab, newline)
r'\S' # non-whitespace
r'.' # any character except newline (with re.DOTALL it matches newline too)
# Quantifiers
r'a*' # 0 or more a's (greedy)
r'a+' # 1 or more a's (greedy)
r'a?' # 0 or 1 a
r'a{3}' # exactly 3 a's
r'a{2,5}'# 2 to 5 a's
r'a*?' # 0 or more a's (non-greedy / lazy)
r'a+?' # 1 or more a's (lazy)
# Anchors
r'^abc' # abc at start of string (or start of line with re.MULTILINE)
r'abc$' # abc at end of string
r'word' # word boundary — matches 'word' but not 'wording'
# Alternation and grouping
r'cat|dog' # cat or dog
r'(cat|dog)s?' # cat or dog, optionally followed by s
r'(?:cat|dog)' # non-capturing group
Groups and Named Groups
log = '2026-09-08 14:32:01 ERROR user_id=42 message=Connection timeout'
# Numbered groups
m = re.search(r'(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}) (\w+)', log)
print(m.group(0)) # full match
print(m.group(1)) # '2026-09-08'
print(m.group(2)) # '14:32:01'
print(m.group(3)) # 'ERROR'
print(m.groups()) # ('2026-09-08', '14:32:01', 'ERROR')
# Named groups — more readable
pattern = r'(?P\d{4}-\d{2}-\d{2}) (?P Lookaheads and Lookbehinds
text = 'Price: ₹1,250.00 USD: $99.99 EUR: €85.50'
# Positive lookahead (?=...) — match X only if followed by Y
prices_after_rupee = re.findall(r'(?<=₹)[\d,]+\.?\d*', text)
print(prices_after_rupee) # ['1,250.00']
# Positive lookbehind (?<=...) — match X only if preceded by Y
dollars = re.findall(r'(?<=\$)[\d.]+', text)
print(dollars) # ['99.99']
# Negative lookahead (?!...) — match X only if NOT followed by Y
words_not_followed_by_comma = re.findall(r'\w+(?!,)', 'cat, dog, bird fish')
# Extract prices regardless of currency symbol
all_prices = re.findall(r'(?<=[\$₹€])[\d,.]+', text)
print(all_prices) # ['1,250.00', '99.99', '85.50']
Compiled Patterns for Performance
# Compile once, use many times — 10-50x faster for large datasets
email_re = re.compile(
r'^[\w.+\-]+@[a-zA-Z0-9\-]+\.[a-zA-Z]{2,}$'
)
phone_re = re.compile(r'(?:\+91|0)?[6-9]\d{9}')
# Apply to a pandas column (use str.contains or apply)
import pandas as pd
df = pd.DataFrame({'email': ['user@example.com', 'bad-email', 'hello@data.in']})
df['valid_email'] = df['email'].str.match(r'^[\w.+\-]+@[a-zA-Z0-9\-]+\.[a-zA-Z]{2,}$')
# Vectorised — fastest for large DataFrames
df['has_digit'] = df['email'].str.contains(r'\d', regex=True)
# Extract groups with str.extract
df2 = pd.DataFrame({'log': ['ERROR: disk full', 'INFO: started', 'WARN: low memory']})
df2[['level', 'msg']] = df2['log'].str.extract(r'^(\w+): (.+)$')
print(df2)
Real-World Patterns
# Email
r'^[\w.+\-]+@[a-zA-Z0-9\-]+\.[a-zA-Z]{2,}$'
# Indian mobile number
r'(?:\+91|0)?[6-9]\d{9}'
# URL
r'https?://(?:www\.)?[-\w@:%._+~#=]{1,256}\.[a-zA-Z]{2,6}(?:[-\w@:%_+.~#?&/=]*)'
# IPv4 address
r'(?:\d{1,3}\.){3}\d{1,3}'
# Date YYYY-MM-DD
r'\d{4}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12]\d|3[01])'
# HTML tag stripper
re.sub(r'<[^>]+>', '', html_content)
# Remove extra whitespace
re.sub(r'\s+', ' ', text).strip()
# Extract JSON-like key-value from logs
re.findall(r'(\w+)=([^\s,]+)', log_line)
# Find all words that appear in ALL CAPS
re.findall(r'[A-Z]{2,}', text)
# Remove non-ASCII characters
re.sub(r'[^ -]+', '', text)
Flags
# re.IGNORECASE — case-insensitive matching
re.findall(r'python', text, flags=re.IGNORECASE)
# re.MULTILINE — ^ and $ match start/end of each line
re.findall(r'^\w+', multiline_text, flags=re.MULTILINE)
# re.DOTALL — . matches newline too
re.search(r'.+?', html, flags=re.DOTALL)
# re.VERBOSE — write readable patterns with comments
email_pattern = re.compile(r'''
^ # start of string
[\w.+\-]+ # username: word chars, dots, plus, hyphen
@ # literal @
[a-zA-Z0-9\-]+ # domain name
\. # literal dot
[a-zA-Z]{2,} # TLD (2+ letters)
$ # end of string
''', re.VERBOSE | re.IGNORECASE)
Conclusion
Regular expressions reward time invested in learning them — even basic knowledge of \d, \w, \s, and the quantifiers solves the majority of text cleaning tasks. Use re.compile() for patterns you apply many times. Prefer non-capturing groups (?:...) when you do not need to extract sub-matches. Use named groups when a pattern has many captures. Test your patterns interactively on regex101.com before embedding them in production code — it visualises what each part matches, which saves hours of debugging.
