Web Scraping with Python: requests + BeautifulSoup

Data lives in web pages. Fetch, parse and extract it cleanly — and do it politely.

Python2026-08-128 min readPython · requests · BeautifulSoup

In this guide

  1. Fetch a page
  2. Find elements
  3. Extract attributes and text
  4. Handle pagination
  5. When scraping gets hard

Most of the data you need isn't in a tidy CSV — it's in a web page. Scraping bridges that gap. We use it constantly for datasets, research and monitoring. Here's the clean, responsible way.

Fetch a page

bash
pip install requests beautifulsoup4 lxml
python
import requests
from bs4 import BeautifulSoup

resp = requests.get(
    "https://books.toscrape.com/",
    headers={"User-Agent": "Mozilla/5.0 (compatible; tutorial/1.0)"},
)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "lxml")

Find elements

python
# CSS selectors are usually the cleanest
titles = [h3.a["title"] for h3 in soup.select("h3")]
prices = [p.text.strip() for p in soup.select(".price_color")]
for t, p in zip(titles, prices):
    print(t, "—", p)

soup.select() accepts any CSS selector; .find() and .find_all() are the older API. Prefer selectors.

Extract attributes and text

python
link = soup.select_one("h3 a")
href = link["href"]            # attribute access
text = link.get_text(strip=True)

Handle pagination

python
import time

def scrape_page(url):
    r = requests.get(url, headers={"User-Agent": "tutorial/1.0"})
    return BeautifulSoup(r.text, "lxml")

page = scrape_page("https://books.toscrape.com/")
while True:
    nxt = page.select_one("li.next a")
    if not nxt:
        break
    time.sleep(1)               # be polite
    page = scrape_page("https://books.toscrape.com/" + nxt["href"])
Etiquette matters. Respect robots.txt, throttle yourself, and don't hammer a small site. Scraping responsibly is the difference between a useful tool and a nuisance.

When scraping gets hard

  • Dynamic pages rendered by JS → use Playwright or a headless browser.
  • Login walls → use a session and persist cookies.
  • Rate limits → rotate proxies and back off exponentially.

Master these basics and you can turn any public page into structured data.

Keep reading

More from the lab

Web & Backend

Build a REST API with FastAPI and SQLite

From zero to a documented, validated CRUD API in one file — the fastest Python back end there is.

FastAPI · SQLite · REST · 9 min DevOps

Docker for Beginners: Containers Explained

What a container actually is, how a Dockerfile works, and how to ship an app that runs everywhere.

Docker · Containers · 8 min DevOps

A Clean Git Branching Strategy for Teams

Keep main green and releases boring with a lightweight trunk-based Git workflow.

Git · Workflow · 6 min
Your move

Want us to build this for you?

We turn tutorials like these into production systems every day. Tell us what you need.

Start a project →