Web Scraping with Python, Requests, and Beautiful Soup

ByJu Yeon Eum

Websites often contain valuable information that is not available as a downloadable dataset or through an API. Copying hundreds of records manually into a spreadsheet would be slow, repetitive, and error-prone.

Web scraping automates this process. Python can download a web page with the Requests library, parse its HTML with Beautiful Soup, and extract selected information into a structured dataset.

This article covers:

  • What web scraping is
  • How HTML documents are structured
  • Beautiful Soup objects
  • Navigating the HTML tree
  • Using find() and find_all()
  • Extracting data from HTML tables
  • Scraping a live web page
  • Saving extracted data with pandas
  • Responsible and reliable scraping practices

What Is Web Scraping?

Web scraping is the automated extraction of information from web pages.

A typical scraping workflow has four stages:

  1. Send an HTTP request to a web server.
  2. Download the page’s HTML.
  3. locate the required HTML elements.
  4. Extract, clean, and store their contents.

For example, a scraper could collect:

  • Product names and prices
  • Sports statistics
  • Real-estate listings
  • Article titles
  • Public financial tables
  • Job listings
  • Links to documents
  • Public research data

Web scraping should generally be used when an appropriate API or downloadable dataset is unavailable. If a website provides a stable API, that will usually be more reliable than parsing its HTML.

Requests and Beautiful Soup

Two Python libraries are commonly combined for basic web scraping:

  • Requests downloads the web page.
  • Beautiful Soup parses and searches its HTML.

Install the required packages with:

python -m pip install requests beautifulsoup4 pandas

Then import them:

import requests
from bs4 import BeautifulSoup

Beautiful Soup does not download web pages by itself. It processes HTML or XML content that has already been obtained from a file, string, or HTTP client.

Understanding HTML Structure

HTML documents are composed of nested elements called tags.

Consider this simplified page:

<html>
    <head>
        <title>Basketball Players</title>
    </head>

    <body>
        <section class="player">
            <h3><b>LeBron James</b></h3>
            <p class="salary">$48,728,845</p>
        </section>

        <section class="player">
            <h3><b>Stephen Curry</b></h3>
            <p class="salary">$55,761,216</p>
        </section>
    </body>
</html>

The elements form a tree:

html
├── head
│   └── title
└── body
    ├── section
    │   ├── h3
    │   │   └── b
    │   └── p
    └── section
        ├── h3
        │   └── b
        └── p

Understanding this hierarchy helps us locate the required information.

Creating a Beautiful Soup Object

Pass an HTML document and a parser name to the BeautifulSoup constructor:

from bs4 import BeautifulSoup

html = """
<html>
    <head>
        <title>Basketball Players</title>
    </head>

    <body>
        <section class="player">
            <h3><b>LeBron James</b></h3>
            <p class="salary">$48,728,845</p>
        </section>

        <section class="player">
            <h3><b>Stephen Curry</b></h3>
            <p class="salary">$55,761,216</p>
        </section>
    </body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

The resulting soup object represents the document as a searchable parse tree.

Python’s built-in html.parser is sufficient for many projects. Other parsers, such as lxml, can offer different performance and error-recovery characteristics.

Beautiful Soup Object Types

Beautiful Soup works with several important object types.

BeautifulSoup

The BeautifulSoup object represents the parsed document as a whole:

print(type(soup))

Tag

A Tag represents an HTML or XML tag:

title_tag = soup.find("title")

print(type(title_tag))
print(title_tag)

Result:

<title>Basketball Players</title>

NavigableString

Text located inside a tag is represented by a string-like object called a NavigableString.

title_text = title_tag.string

print(type(title_text))
print(title_text)

For most extraction tasks, get_text() is more flexible than accessing .string directly.

print(title_tag.get_text(strip=True))

Finding the First Matching Element

The find() method returns the first matching element:

first_heading = soup.find("h3")

print(first_heading)

Result:

<h3><b>LeBron James</b></h3>

The following shorthand also returns the first matching element:

first_heading = soup.h3

However, explicit find() calls are often easier to read and extend:

first_player = soup.find("section", class_="player")

Notice that Beautiful Soup uses class_ instead of class because class is a reserved Python keyword.

Extracting Text from a Tag

Use get_text() to extract the visible text contained in an element:

player_name = first_heading.get_text(strip=True)

print(player_name)

Result:

LeBron James

The strip=True argument removes surrounding whitespace.

If nested text fragments should be separated, supply a separator:

text = first_player.get_text(" ", strip=True)

print(text)

Navigating Down the Tree

Tags can contain other tags.

The first h3 element contains a b element:

heading = soup.find("h3")
bold_tag = heading.find("b")

print(bold_tag.get_text(strip=True))

You can also access the nested tag using tag-name notation:

bold_tag = heading.b

For direct children, use .children:

for child in heading.children:
    print(repr(child))

The .contents attribute returns direct children as a list:

print(heading.contents)

Navigating Up the Tree

Use .parent to move from an element to its parent:

bold_tag = soup.find("b")
heading = bold_tag.parent

print(heading.name)

Result:

h3

To inspect multiple ancestors, use .parents:

for parent in bold_tag.parents:
    if parent.name:
        print(parent.name)

Navigating Between Siblings

Elements at the same level of the tree are siblings.

heading = soup.find("h3")
next_element = heading.find_next_sibling()

print(next_element)

Result:

<p class="salary">$48,728,845</p>

find_next_sibling() is often more convenient than .next_sibling because raw HTML frequently contains newline and whitespace nodes between tags.

For example, this may return a newline rather than the next tag:

print(repr(heading.next_sibling))

For element-oriented navigation, prefer:

salary_tag = heading.find_next_sibling("p")

Accessing HTML Attributes

HTML attributes are available through a dictionary-like interface.

Consider this element:

<a href="/players/23" class="profile-link">Player Profile</a>

Parse and inspect it:

html_link = """
<a href="/players/23" class="profile-link">
    Player Profile
</a>
"""

link = BeautifulSoup(
    html_link,
    "html.parser"
).find("a")

print(link["href"])
print(link.get("class"))

Result:

/players/23
['profile-link']

Using get() is safer when an attribute might not exist:

href = link.get("href")

if href is not None:
    print(href)

Direct access raises a KeyError when the attribute is missing:

href = link["href"]

Finding All Matching Elements

The find_all() method searches an element’s descendants and returns all matches.

players = soup.find_all("section", class_="player")

print(len(players))

Each result is a Tag object:

for player in players:
    print(player.get_text(" ", strip=True))

Beautiful Soup’s current documentation defines filters such as the tag name, attributes, string content, recursion behavior, and a result limit for find_all().

Filtering by Tag

headings = soup.find_all("h3")

Filtering by Class

player_sections = soup.find_all(
    "section",
    class_="player"
)

Filtering by Attribute

links = soup.find_all("a", href=True)

This returns only a elements that have an href attribute.

Limiting Results

first_two = soup.find_all(
    "section",
    class_="player",
    limit=2
)

Filtering Text

import re

matches = soup.find_all(
    string=re.compile("James", re.IGNORECASE)
)

Extracting Player Names and Salaries

The player information can now be converted into Python dictionaries:

records = []

for player in soup.find_all(
    "section",
    class_="player"
):
    name_tag = player.find("h3")
    salary_tag = player.find("p", class_="salary")

    if name_tag is None or salary_tag is None:
        continue

    records.append({
        "name": name_tag.get_text(strip=True),
        "salary": salary_tag.get_text(strip=True)
    })

print(records)

Result:

[
    {
        "name": "LeBron James",
        "salary": "$48,728,845"
    },
    {
        "name": "Stephen Curry",
        "salary": "$55,761,216"
    }
]

Checking for None prevents the program from failing when an expected element is missing.

Extracting Data from an HTML Table

Tabular information is commonly represented using:

  • <table> for the complete table
  • <tr> for rows
  • <th> for header cells
  • <td> for data cells

Consider this HTML:

table_html = """
<table id="restaurants">
    <tr>
        <th>Restaurant</th>
        <th>City</th>
        <th>Rating</th>
    </tr>
    <tr>
        <td>North Pizza</td>
        <td>Chicago</td>
        <td>4.7</td>
    </tr>
    <tr>
        <td>Central Pizza</td>
        <td>New York</td>
        <td>4.5</td>
    </tr>
</table>
"""

table_soup = BeautifulSoup(
    table_html,
    "html.parser"
)

table = table_soup.find(
    "table",
    id="restaurants"
)

Find all rows:

rows = table.find_all("tr")

Extract every cell:

for row in rows:
    cells = row.find_all(["th", "td"])
    values = [
        cell.get_text(strip=True)
        for cell in cells
    ]

    print(values)

Result:

['Restaurant', 'City', 'Rating']
['North Pizza', 'Chicago', '4.7']
['Central Pizza', 'New York', '4.5']

Converting a Table to a pandas DataFrame

Separate the header from the data rows:

import pandas as pd

rows = table.find_all("tr")

headers = [
    cell.get_text(strip=True)
    for cell in rows[0].find_all("th")
]

data = []

for row in rows[1:]:
    values = [
        cell.get_text(strip=True)
        for cell in row.find_all("td")
    ]

    if len(values) == len(headers):
        data.append(values)

df = pd.DataFrame(data, columns=headers)

print(df)

Result:

       Restaurant      City Rating
0     North Pizza   Chicago    4.7
1   Central Pizza  New York    4.5

Save it as a CSV file:

df.to_csv(
    "restaurants.csv",
    index=False
)

For simple, conventional tables, pandas may be able to extract them directly:

tables = pd.read_html(table_html)
df = tables[0]

Beautiful Soup is more useful when the page requires custom element selection or cleaning.

Using CSS Selectors

Beautiful Soup also supports CSS selectors.

Find all player sections:

players = soup.select("section.player")

Find the salary within the first player section:

salary = soup.select_one(
    "section.player p.salary"
)

Extract the text:

print(salary.get_text(strip=True))

Common selectors include:

SelectorMeaning
pEvery p element
.salaryElements with class salary
#playersElement with ID players
section.playersection elements with class player
table trRows located inside tables
a[href]Links containing an href attribute

For complex page structures, select() and select_one() can be easier to read than a chain of nested find() calls.

Scraping a Live Web Page

The Requests library can download a page before Beautiful Soup parses it.

import requests
from bs4 import BeautifulSoup

url = "https://example.com/"

headers = {
    "User-Agent": (
        "DataSphere-Learning-Scraper/1.0 "
        "(contact: your-email@example.com)"
    )
}

response = requests.get(
    url,
    headers=headers,
    timeout=(3.05, 15)
)

response.raise_for_status()

soup = BeautifulSoup(
    response.text,
    "html.parser"
)

title = soup.find("title")

if title is not None:
    print(title.get_text(strip=True))

Important parts of this request include:

  • An HTTPS URL
  • A descriptive user agent
  • An explicit timeout
  • raise_for_status() for HTTP errors
  • A check that the expected tag exists

Requests does not apply a default timeout, so production requests should explicitly provide one. Its documentation recommends using timeouts and explains that raise_for_status() raises an exception for unsuccessful HTTP responses.

A Reusable Scraping Function

The following function retrieves and parses a page:

import requests
from bs4 import BeautifulSoup

def download_page(url):
    headers = {
        "User-Agent": (
            "DataSphere-Learning-Scraper/1.0 "
            "(contact: your-email@example.com)"
        )
    }

    try:
        response = requests.get(
            url,
            headers=headers,
            timeout=(3.05, 15)
        )

        response.raise_for_status()

    except requests.exceptions.Timeout as error:
        raise RuntimeError(
            f"Request timed out: {url}"
        ) from error

    except requests.exceptions.HTTPError as error:
        raise RuntimeError(
            f"HTTP error while retrieving {url}: {error}"
        ) from error

    except requests.exceptions.RequestException as error:
        raise RuntimeError(
            f"Unable to retrieve {url}: {error}"
        ) from error

    return BeautifulSoup(
        response.text,
        "html.parser"
    )

Use it as follows:

soup = download_page(
    "https://example.com/"
)

heading = soup.find("h1")

if heading is not None:
    print(heading.get_text(strip=True))

Handling Missing Elements

Web pages are not guaranteed to contain every expected element.

This code can fail:

salary = player.find(
    "p",
    class_="salary"
).get_text(strip=True)

If no matching element exists, find() returns None, and calling get_text() raises an error.

A safer approach is:

salary_tag = player.find(
    "p",
    class_="salary"
)

salary = (
    salary_tag.get_text(strip=True)
    if salary_tag
    else None
)

For multiple fields, a helper function is useful:

def extract_text(element, selector):
    match = element.select_one(selector)

    if match is None:
        return None

    return match.get_text(" ", strip=True)

Example:

record = {
    "name": extract_text(player, "h3"),
    "salary": extract_text(player, "p.salary")
}

Cleaning Extracted Values

Scraped data is usually text and may need additional cleaning.

Convert a salary string into an integer:

salary_text = "$48,728,845"

salary = int(
    salary_text
    .replace("$", "")
    .replace(",", "")
)

print(salary)

Result:

48728845

For more robust currency cleaning:

import re

salary_text = "$48,728,845"

digits = re.sub(
    r"[^\d.-]",
    "",
    salary_text
)

salary = float(digits)

Other common cleaning tasks include:

  • Removing extra whitespace
  • Converting dates
  • Standardizing units
  • Resolving relative URLs
  • Removing duplicate records
  • Handling missing values
  • Converting numeric columns

Resolving Relative URLs

A page may contain a relative link:

<a href="/players/23">Profile</a>

Use urljoin() to convert it into an absolute URL:

from urllib.parse import urljoin

base_url = "https://example.com/"
relative_url = "/players/23"

absolute_url = urljoin(
    base_url,
    relative_url
)

print(absolute_url)

Result:

https://example.com/players/23

Why Some Pages Cannot Be Scraped with Requests Alone

Requests downloads the HTML returned directly by the server. It does not execute JavaScript.

Some websites initially return an almost empty HTML document and load their visible information later through JavaScript. In that situation, Beautiful Soup cannot find data that was never present in the downloaded HTML.

Possible alternatives include:

  • Finding the API used by the page
  • Looking for embedded JSON in the HTML
  • Using an official data export
  • Using a browser-automation tool when permitted

Before choosing browser automation, inspect the source HTML and browser network requests. An underlying JSON endpoint is often more stable and efficient.

Responsible Web Scraping

The technical ability to retrieve a page does not automatically grant permission to collect or reuse its information.

Before scraping a site:

  • Read its terms of service.
  • Review its robots.txt file.
  • Prefer an official API or data export.
  • Respect copyright and database rights.
  • Avoid collecting private or sensitive information.
  • Do not bypass authentication or access controls.
  • Limit request frequency.
  • Identify the scraper appropriately.
  • Cache pages when repeated downloads are unnecessary.
  • Follow applicable privacy and data-protection laws.

Python provides urllib.robotparser for checking whether a user agent is permitted to retrieve a URL according to a site’s robots.txt rules. The module exposes methods such as can_fetch() and crawl_delay(). Python documents the complete interface.

A basic check looks like this:

from urllib.robotparser import RobotFileParser

robots_url = "https://example.com/robots.txt"
target_url = "https://example.com/data"
user_agent = "DataSphere-Learning-Scraper"

parser = RobotFileParser()
parser.set_url(robots_url)
parser.read()

allowed = parser.can_fetch(
    user_agent,
    target_url
)

print(allowed)

A robots.txt file communicates crawler preferences, but it is not a substitute for reviewing the site’s terms, permissions, and applicable law.

Avoiding Excessive Requests

Add delays when requesting multiple pages:

import time

for url in urls:
    soup = download_page(url)

    # Extract the required information here.

    time.sleep(2)

For large collections, also consider:

  • Retry limits
  • Exponential backoff
  • Response caching
  • Duplicate URL detection
  • Checkpointing
  • Request and error logging
  • Resumable processing

Aggressive parallel scraping can overload a site and may cause the client to be blocked.

Common Web-Scraping Problems

The selector returns nothing

The page structure may have changed, or the element may be generated by JavaScript.

element = soup.select_one(".salary")

if element is None:
    print("Salary element not found.")

The server returns 403

The site may reject automated access or require authorization. Do not attempt to bypass access controls. Look for an official API or request permission.

The script receives different HTML

Sites may serve different content based on location, cookies, language, authentication, user agent, or experiments.

The scraper suddenly stops working

HTML is a presentation format rather than a stable data contract. Classes, nesting, and page templates can change without notice.

Extracted data contains duplicates

Pages may contain repeated listings, featured records, mobile markup, or pagination overlap. Identify a stable record key before removing duplicates.

Complete Table-Scraping Example

The following example combines parsing, validation, cleaning, and DataFrame creation:

from bs4 import BeautifulSoup
import pandas as pd

html = """
<table id="players">
    <tr>
        <th>Player</th>
        <th>Team</th>
        <th>Salary</th>
    </tr>
    <tr>
        <td>LeBron James</td>
        <td>Lakers</td>
        <td>$48,728,845</td>
    </tr>
    <tr>
        <td>Stephen Curry</td>
        <td>Warriors</td>
        <td>$55,761,216</td>
    </tr>
</table>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

table = soup.find("table", id="players")

if table is None:
    raise ValueError("Player table not found.")

rows = table.find_all("tr")
records = []

for row in rows[1:]:
    cells = row.find_all("td")

    if len(cells) != 3:
        continue

    name = cells[0].get_text(strip=True)
    team = cells[1].get_text(strip=True)

    salary_text = cells[2].get_text(strip=True)
    salary = int(
        salary_text
        .replace("$", "")
        .replace(",", "")
    )

    records.append({
        "player": name,
        "team": team,
        "salary": salary
    })

df = pd.DataFrame(records)

print(df)

This produces a structured dataset whose salary column contains numbers rather than formatted text.

Key Takeaways

  • Web scraping automatically extracts information from web pages.
  • Requests downloads HTML from a server.
  • Beautiful Soup converts HTML into a searchable tree.
  • find() returns the first matching element.
  • find_all() returns all matching descendants.
  • CSS selectors can be used through select() and select_one().
  • get_text(strip=True) extracts and cleans text.
  • HTML tables consist primarily of table, tr, th, and td tags.
  • Missing elements should be handled explicitly.
  • Requests should include timeouts and HTTP error handling.
  • Requests and Beautiful Soup do not execute JavaScript.
  • Site rules, permissions, privacy, and request frequency must be considered before scraping.

Conclusion

Requests and Beautiful Soup provide an effective foundation for collecting public information from conventional HTML pages. Requests handles HTTP communication, while Beautiful Soup offers tools for searching tags, navigating document relationships, extracting attributes, and converting unstructured markup into records.

A dependable scraper requires more than successful extraction. It must tolerate missing elements, clean the resulting data, handle network failures, accommodate reasonable page changes, and operate respectfully within the site’s rules and applicable legal requirements.

One-sentence summary: Python web scraping combines Requests for downloading web pages with Beautiful Soup for locating, extracting, cleaning, and structuring information contained in HTML.

Similar Posts

Questions, corrections, or additional insights?