Web scraping is the process of automatically extracting data from websites. For data scientists, it’s an essential technique when the data you need doesn’t come in a convenient CSV or API. Job listings, product prices, news articles, sports statistics, social media posts — all of this data can be collected programmatically with Python’s BeautifulSoup and Requests libraries. This guide walks you through the complete workflow from fetching a page to storing clean, structured data.
Setting Up: Requests and BeautifulSoup
You need two main libraries: requests to fetch web pages and beautifulsoup4 to parse the HTML. Install them with pip:
pip install requests beautifulsoup4 lxmlThe basic pattern is always the same — fetch the page content, parse it as HTML, then navigate the tree to extract what you want:
import requests
from bs4 import BeautifulSoup
import time
def get_page(url, headers=None):
# Fetch a page with error handling and polite delays
default_headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '
'AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/120.0.0.0 Safari/537.36'
}
headers = headers or default_headers
try:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status() # raise exception for 4xx/5xx
return response
except requests.RequestException as e:
print(f"Error fetching {url}: {e}")
return None
url = "https://example.com/articles"
response = get_page(url)
if response:
soup = BeautifulSoup(response.text, 'lxml')
print(soup.title.text)
Always set a User-Agent header. Without it, many sites will block your requests immediately, assuming you’re a bot. The User-Agent tells the server what browser you’re pretending to be.
Navigating HTML with BeautifulSoup
BeautifulSoup lets you navigate HTML using element names, CSS classes, IDs, and attributes. The two most important methods are find() (returns the first match) and find_all() (returns all matches as a list). Here’s a practical example scraping job listings:
import requests
from bs4 import BeautifulSoup
import pandas as pd
def scrape_jobs(url):
response = get_page(url)
if not response:
return []
soup = BeautifulSoup(response.text, 'lxml')
jobs = []
# Find all job listing cards
job_cards = soup.find_all('div', class_='job-card')
for card in job_cards:
# Extract specific fields - use .get_text(strip=True) to clean whitespace
title_elem = card.find('h2', class_='job-title')
company_elem = card.find('span', class_='company-name')
location_elem = card.find('span', class_='location')
salary_elem = card.find('span', class_='salary')
job = {
'title': title_elem.get_text(strip=True) if title_elem else 'N/A',
'company': company_elem.get_text(strip=True) if company_elem else 'N/A',
'location': location_elem.get_text(strip=True) if location_elem else 'N/A',
'salary': salary_elem.get_text(strip=True) if salary_elem else 'Not listed',
'url': card.find('a', href=True)['href'] if card.find('a') else ''
}
jobs.append(job)
return jobs
CSS selectors via soup.select() are often cleaner than chained find() calls. soup.select('div.job-card h2.title') finds all h2 elements with class “title” inside divs with class “job-card”.
Handling Pagination
Most real scraping jobs require navigating multiple pages. Two patterns cover 90% of cases — URL-based pagination (where the page number is in the URL) and “next page” link scraping:
def scrape_all_pages(base_url, max_pages=10):
# Scrape multiple pages with polite delays
all_data = []
for page_num in range(1, max_pages + 1):
url = f"{base_url}?page={page_num}"
print(f"Scraping page {page_num}...")
response = get_page(url)
if not response:
break
soup = BeautifulSoup(response.text, 'lxml')
page_data = extract_data(soup) # your extraction function
if not page_data:
print("No data found — reached end of results")
break
all_data.extend(page_data)
# Polite delay between requests (IMPORTANT — be respectful)
time.sleep(2) # 2 seconds between requests
return all_data
def get_next_page_url(soup, current_url):
# Returns the URL of the next page, or None if there is no next page
next_btn = soup.find('a', {'rel': 'next'}) or soup.find('a', string='Next')
if next_btn and next_btn.get('href'):
from urllib.parse import urljoin
return urljoin(current_url, next_btn['href'])
return None
Storing Scraped Data and Avoiding Blocks
Save your scraped data progressively — don’t wait until you’ve scraped everything before saving. If the script crashes halfway through, you lose nothing:
import pandas as pd
import json
from datetime import datetime
def save_data(data, filename_prefix='scraped_data'):
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
# Save as CSV for easy analysis
df = pd.DataFrame(data)
csv_path = f"{filename_prefix}_{timestamp}.csv"
df.to_csv(csv_path, index=False)
print(f"Saved {len(df)} records to {csv_path}")
return df
To avoid getting blocked: always add delays between requests (2-5 seconds minimum), rotate User-Agents if scraping at scale, respect the site’s robots.txt file, and never hammer a server with parallel requests unless you have explicit permission. For JavaScript-rendered sites where BeautifulSoup returns empty divs, use Playwright or Selenium instead — those tools actually execute JavaScript before returning the page content.
Frequently Asked Questions
Is web scraping legal?
It depends. Scraping publicly available data is generally legal in most countries, but scraping behind login walls, violating a site’s Terms of Service, or using data commercially can be legally problematic. Always check robots.txt and the site’s ToS, and when in doubt, look for an official API instead.
When should I use an API instead of scraping?
Always use an API when one exists. APIs are more stable (the data format doesn’t change when the website redesigns), faster, and explicitly permitted. Scraping is a fallback for when no API is available.
How do I scrape JavaScript-rendered sites?
BeautifulSoup only sees the raw HTML — it can’t execute JavaScript. For React, Vue, or Angular sites that load content dynamically, use Playwright (pip install playwright) or Selenium. Playwright is the modern choice — it’s faster, has an async API, and is easier to use than Selenium.
