How to Scrape Wikipedia Tables into DataFrames with Python

Direct answer: use pandas.read_html() to load Wikipedia’s HTML tables, inspect the returned list, deliberately select the intended DataFrame, then normalize headers and convert values before analysis.

import pandas as pd

url = "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"
tables = pd.read_html(url)

print(f"Found {len(tables)} tables")
for i, table in enumerate(tables):
    print(f"\\nTable {i}")
    print(table.head())
    print(table.columns)

df = tables[0]  # Choose only after inspecting the candidates
print(df.head())

read_html searches HTML <table> elements and returns a list of DataFrame objects, even if the page contains one table. See the official pandas API reference and the HTML I/O guide.

1. Install pandas and an HTML parser

Install pandas and at least one supported parser in the environment that runs your script:

python -m pip install pandas lxml

Pandas documents lxml, html5lib, and bs4 parser flavors. If one parser cannot handle a page or is missing, install the dependencies for another supported flavor and pass flavor= explicitly.

2. Inspect every returned table

Never assume table zero is the table you need. Wikipedia pages often contain navigation, chronology, rankings, references, and data tables together.

import pandas as pd

url = "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"
tables = pd.read_html(url)

for i, df in enumerate(tables):
    print(f"Table {i}: shape={df.shape}")
    print("Columns:", list(df.columns))
    print(df.head(3).to_string(index=False))

# After inspection, select by a documented decision.
df = tables[0]

Check the shape, column labels, first few rows, and whether the values represent the subject you intended. Keep the selection rule in your code or configuration so a later rerun is auditable.

3. Select the right Wikipedia table

Filter by visible text with `match`

tables = pd.read_html(
    url,
    match="Population",
)

if not tables:
    raise ValueError("No table matched the requested text")
df = tables[0]

match filters tables by text found in the table. Use a distinctive heading or phrase, then still inspect the returned DataFrames.

Target a valid HTML attribute with `attrs`

tables = pd.read_html(
    url,
    attrs={"class": "wikitable"},
)
for i, candidate in enumerate(tables):
    print(i, candidate.shape, candidate.columns)

attrs accepts valid HTML attributes such as an element’s id or class. A class can match several tables, so combine it with inspection or match.

Use a selection helper

def choose_table(tables, required_columns):
    required = set(required_columns)
    for index, candidate in enumerate(tables):
        columns = {str(column).strip() for column in candidate.columns}
        if required.issubset(columns):
            return index, candidate
    raise ValueError(f"No table contains columns: {sorted(required)}")

index, df = choose_table(tables, ["Country", "Population"])
print(f"Selected table {index}")

Column names may be tuples when the source has multi-row headers. Inspect them before writing strict matching logic.

4. Important `read_html` options

Option	Use it for	Example
`match`	Filtering tables by visible text	`match="Population"`
`attrs`	Filtering by valid HTML attributes	`attrs={"class":"wikitable"}`
`header`	Choosing the row used for column labels	`header=0`
`index_col`	Using one or more columns as the index	`index_col=0`
`skiprows`	Skipping title, note, or preamble rows	`skiprows=[0, 1]`
`parse_dates`	Parsing date columns during import	`parse_dates=["Date"]`
`thousands`	Declaring thousands separators	`thousands=","`
`decimal`	Declaring the decimal marker	`decimal="."`
`converters`	Applying a function to selected columns	`converters={"Rank": int}`
`na_values`	Adding source strings treated as missing	`na_values=["—", "N/A"]`
`keep_default_na`	Controlling pandas’ default missing markers	`keep_default_na=True`
`displayed_only`	Including or excluding hidden HTML elements	`displayed_only=False`
`extract_links`	Preserving links from cells	`extract_links="all"`
`flavor`	Selecting a parser implementation	`flavor="lxml"`

Use options to describe the source you have inspected; do not add them blindly. Header spans, footnotes, hidden rows, and presentation markup frequently require a cleanup pass.

5. Clean headers and values

Normalize column labels

def flatten_column(column):
    if isinstance(column, tuple):
        parts = [str(part).strip() for part in column if str(part).strip() != "Unnamed: 0"]
        return "_".join(parts)
    return str(column).strip()

df.columns = [flatten_column(column) for column in df.columns]
df.columns = (
    pd.Index(df.columns)
    .str.lower()
    .str.replace(r"[^a-z0-9]+", "_", regex=True)
    .str.strip("_")
)
print(df.columns.tolist())

Inspect the result before renaming columns. A multi-row header can produce a MultiIndex, and a careless flattening rule can lose meaning.

Convert numbers safely

import re

number_column = "population"
df[number_column] = (
    df[number_column]
    .astype("string")
    .str.replace(r"\\[[^]]*\\]", "", regex=True)  # footnote markers
    .str.replace(",", "", regex=False)
    .str.strip()
)
df[number_column] = pd.to_numeric(df[number_column], errors="coerce")

invalid = df[number_column].isna().sum()
print(f"Values that could not be converted: {invalid}")

errors="coerce" makes unparseable values missing instead of crashing. Count and review those missing values; do not silently treat conversion loss as valid data.

Parse dates after checking the displayed format

df["date"] = pd.to_datetime(
    df["date"],
    errors="coerce",
    dayfirst=False,
)
if df["date"].isna().any():
    print("Some dates did not parse; inspect the source format and failed rows.")

You can also pass parse_dates to read_html, or use a column-specific function through converters. Verify whether the page uses day-first dates, month-first dates, years only, or mixed display formats.

Handle missing values explicitly

tables = pd.read_html(
    url,
    na_values=["—", "N/A", "No data"],
    keep_default_na=True,
)
df = tables[0]
print(df.isna().sum())

Keep a distinction between a true zero, an unavailable value, and a suppressed value. If that distinction matters, preserve the raw column before converting it.

6. Preserve hyperlinks when they matter

By default, the table is optimized for displayed values. If you need the links behind country names or references, request them explicitly:

tables = pd.read_html(
    url,
    extract_links="all",
)
df = tables[0]
print(df.iloc[0].to_dict())

Link extraction can change cell values into pairs containing displayed text and URLs. Normalize those pairs deliberately before analysis or export.

7. A reusable, auditable loader

from datetime import datetime, timezone
from pathlib import Path
import pandas as pd


def load_wikipedia_table(url, *, match=None, attrs=None, required_columns=None):
    read_kwargs = {
        "na_values": ["—", "N/A", "No data"],
        "keep_default_na": True,
    }
    if match is not None:
        read_kwargs["match"] = match
    if attrs is not None:
        read_kwargs["attrs"] = attrs

    tables = pd.read_html(url, **read_kwargs)
    if not tables:
        raise ValueError("Wikipedia returned no matching tables")

    for index, candidate in enumerate(tables):
        print(index, candidate.shape, list(candidate.columns))

    if required_columns:
        required = {str(column) for column in required_columns}
        for candidate in tables:
            columns = {str(column) for column in candidate.columns}
            if required.issubset(columns):
                df = candidate.copy()
                break
        else:
            raise ValueError(f"No table contains {sorted(required)}")
    else:
        df = tables[0].copy()

    df.attrs["source_url"] = url
    df.attrs["retrieved_at_utc"] = datetime.now(timezone.utc).isoformat()
    return df


url = "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population"
df = load_wikipedia_table(
    url,
    match="Population",
    attrs={"class": "wikitable"},
    required_columns=["Country", "Population"],
)
Path("table.csv").write_text(df.to_csv(index=False), encoding="utf-8")

Store the URL and retrieval timestamp with your output. If the page changes, you can identify which source produced a file and rerun the selection and validation steps.

8. When an API is a better interface

read_html is convenient for ordinary rendered tables. It depends on the page’s HTML structure, so it is less suitable when markup changes often, headers use complicated spans, or the data is available through a structured endpoint.

MediaWiki publishes an official API. For structured Wikimedia data, evaluate that API instead of coupling a long-lived pipeline to presentation HTML. An API can also provide clearer field names, pagination, and machine-oriented responses, depending on the dataset and endpoint.

9. Troubleshooting

Symptom	Likely cause	Fix
`read_html` returns many tables	The page contains navigation and reference tables as well as data tables.	Use `match` or valid `attrs`, print every candidate, and select by columns.
No tables found	The filter text or attribute does not match the rendered table.	Remove filters, inspect all tables, then use text or attributes visible in the HTML.
Parser or dependency error	The selected flavor is not installed or cannot parse this markup.	Install the dependencies for `lxml`, `bs4`/`html5lib`, or try another supported `flavor`.
Unexpected `Unnamed` columns	Header cells span multiple rows or contain blank cells.	Inspect `df.columns`; set `header`, `skiprows`, or flatten the resulting MultiIndex.
Numbers are objects or strings	Separators, footnote markers, percent signs, or other presentation text remain.	Clean the raw strings, then call `pd.to_numeric(..., errors="coerce")` and count failures.
Dates are wrong or missing	The displayed format is ambiguous or mixed.	Inspect raw values, set the correct day/month interpretation, and review rows that became `NaT`.
Links disappeared	Only displayed text was extracted.	Use `extract_links="all"` and normalize the resulting text/URL values.
The result changed after a rerun	Wikipedia’s rendered markup or data changed.	Save retrieval metadata, validate expected columns and row counts, and consider the MediaWiki API.
Hidden rows are missing	Only displayed HTML was considered.	Evaluate `displayed_only=False`, then verify that hidden content belongs in your dataset.

10. Performance, reliability, and cost

Performance: fetch only the page and tables you need. Filtering with match or attrs reduces post-processing, but the page still has to be retrieved and parsed.
Reliability: validate required columns, expected data types, and a reasonable row count before publishing downstream results. Keep a raw copy or checksum when reproducibility matters.
Change detection: record the source URL and UTC retrieval time. Treat a changed header, missing table, or sudden row-count shift as a pipeline error that needs review.
Parser choice: use a supported parser that is installed and stable in your deployment. Pin dependencies in production and document the selected flavor.
Cost: pandas itself is open source, but your runtime, network, storage, and any hosted execution environment can have their own costs. A direct MediaWiki API request may be simpler when rendered HTML is unnecessary.

Or skip the browser setup

If your workflow also needs a visual snapshot of the source page, ScreenshotNeo provides a single GET request that returns a PNG, JPEG, WebP, or PDF. It is separate from extracting cell data with pandas, but it can give your pipeline an auditable visual artifact.

Before capture, ScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. It also provides an MCP server with take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.

See the ScreenshotNeo API documentation for all options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population -o wikipedia.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={
        "access_key": "YOUR_API_KEY",
        "url": "https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population",
    },
    timeout=90,
)
r.raise_for_status()
open("wikipedia.webp", "wb").write(r.content)

const q = new URLSearchParams({
  access_key: 'YOUR_API_KEY',
  url: 'https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot failed: ${res.status}`);
const image = Buffer.from(await res.arrayBuffer());
require('fs').writeFileSync('wikipedia.webp', image);

ScreenshotNeo includes full-page and element capture, custom CSS and JavaScript, waits, blocking controls, headers, cookies, user agents, timezone and geolocation, caching, signed links, async jobs, bulk capture, usage information, and PDF controls on every plan. Pricing starts with 1,000 screenshots per month free with no card; paid plans start at $5 for 3,000 shots. Sign up free for ScreenshotNeo.

FAQ

Why does `pd.read_html` return a list?

A page can contain multiple HTML tables, so pandas returns a list of DataFrames. Selecting tables[0] is your choice, not a guarantee that it is the intended table.

Can I scrape a table without downloading the whole page?

read_html reads HTML from a URL, path-like object, or file-like object. If you already have the relevant HTML, pass that object; otherwise the URL fetch retrieves the page needed for parsing.

Should I use BeautifulSoup instead?

Use read_html when the table structure is conventional and you want a DataFrame quickly. Use targeted HTML parsing when you need unusual cell-level rules, and prefer the MediaWiki API when structured data is available and page markup is unstable.

How do I make a scraper reproducible?

Record the source URL and retrieval time, pin parser dependencies, validate columns and types, preserve raw values before cleaning, and review changes rather than silently accepting a different table.