Data scientists work with personal data every day — names, emails, location history, medical records, financial transactions. But most data science courses skip the legal and ethical frameworks that govern how this data can be used. Ignorance is not a defence: GDPR fines can reach €20 million or 4% of global turnover. This guide gives data scientists the practical knowledge they need to work with personal data responsibly and legally.
Key Regulations to Know
GDPR (EU General Data Protection Regulation) applies to any organisation handling personal data of EU residents, regardless of where the organisation is based. India’s DPDP Act (Digital Personal Data Protection Act, 2023) similarly governs personal data of Indian residents. Both require: lawful basis for processing data, explicit consent for sensitive categories, data minimisation (collect only what you need), purpose limitation (don’t use data for different purposes than collected), and the right of individuals to access, correct, or delete their data.
PII Detection – Identifying Personal Data
pip install presidio-analyzer presidio-anonymizer spacy
python -m spacy download en_core_web_lg
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
text = "John Smith's email is john.smith@gmail.com and his phone is +91-9876543210."
results = analyzer.analyze(text=text, language='en')
for r in results:
print(f"Entity: {r.entity_type}, Score: {r.score:.2f}, Text: {text[r.start:r.end]}")
# Anonymize
anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
print(anonymized.text)
# Output: 's email is and his phone is .
Data Anonymisation Techniques
import pandas as pd
import hashlib, re
df = pd.DataFrame({
'name': ['Alice Kumar', 'Bob Singh'],
'email': ['alice@example.com', 'bob@example.com'],
'phone': ['9876543210', '9812345678'],
'age': [28, 35],
'salary': [85000, 120000],
})
# 1. Pseudonymisation — replace with consistent hash
def hash_pii(value: str, salt: str = "secret_salt") -> str:
return hashlib.sha256(f"{salt}{value}".encode()).hexdigest()[:12]
df['user_id'] = df['email'].apply(hash_pii)
df = df.drop(columns=['name', 'email', 'phone'])
# 2. Generalisation — reduce precision of quasi-identifiers
df['age_band'] = pd.cut(df['age'], bins=[0,25,35,50,100],
labels=['18-25','26-35','36-50','51+'])
df['salary_band'] = pd.cut(df['salary'], bins=[0,60000,100000,200000],
labels=['<60K','60-100K','>100K'])
df = df.drop(columns=['age', 'salary'])
# 3. K-anonymity check (each record should be indistinguishable
# from at least k-1 others on quasi-identifiers)
quasi_ids = ['age_band', 'salary_band']
counts = df.groupby(quasi_ids).size()
k_value = counts.min()
print(f"k-anonymity: {k_value}") # should be >= 5 for most contexts
Differential Privacy with Google DP Library
pip install google-dp
import dp_accounting
# Differentially private mean — adds calibrated noise to protect individuals
def dp_mean(data, epsilon=1.0, sensitivity=1.0):
import numpy as np
true_mean = np.mean(data)
noise = np.random.laplace(0, sensitivity / epsilon)
return true_mean + noise
salaries = [85000, 95000, 78000, 120000, 92000]
true_avg = sum(salaries) / len(salaries)
dp_avg = dp_mean(salaries, epsilon=1.0, sensitivity=50000)
print(f"True average: ₹{true_avg:,.0f}")
print(f"DP average: ₹{dp_avg:,.0f}")
# Similar but not identical — individual values can't be inferred
Data Minimisation in Practice
# Bad: collect everything "in case we need it later"
full_profile = pd.read_sql("SELECT * FROM users", conn)
# Good: collect only what the model needs
feature_query = '''
SELECT user_id, account_age_days, num_transactions,
avg_transaction_value, days_since_last_login
FROM users
WHERE created_at >= '2024-01-01'
'''
minimal_features = pd.read_sql(feature_query, conn)
# Never store PII in model training data
# Never log PII in application logs
# Never put PII in Git repositories
Model Governance – What to Document
For every model that processes personal data, you should document: the lawful basis for processing (consent, legitimate interest, contract), the categories of personal data used, how long data is retained, who has access to the model and its outputs, what safeguards prevent misuse, how individuals can exercise their rights (access, erasure, correction), and the results of a Data Protection Impact Assessment (DPIA) for high-risk processing. The EU AI Act (effective 2026) adds further requirements for high-risk AI systems — mandatory transparency, human oversight, and registration in an EU database.
Practical Checklist for Data Scientists
Before any project involving personal data: confirm you have a lawful basis for using this data, check if a DPIA is required, ensure data is accessed on a need-to-know basis, document what data is used and why. During development: don’t use real PII in development — use synthetic or anonymised data, never commit datasets with PII to Git, store data in access-controlled environments, log data access for audit purposes. Before deployment: document the model in a model card, ensure individuals can opt out, implement monitoring for fairness issues, set a data retention policy and stick to it.
Conclusion
Data governance is not bureaucracy — it’s what separates responsible data science from reckless data science. GDPR and India’s DPDP Act are enforceable law with significant penalties. Beyond compliance, responsible data handling builds user trust, reduces security risk, and prevents the kind of ML bias that occurs when data is collected without care for who is and isn’t represented. Build these habits early — they’re much harder to retrofit into a system after the fact.



