Capture a dynamic web table after its rows load asynchronously
Wait for the table’s actual content—not just page navigation—then extract its rows or capture a screenshot. See Playwright and Selenium examples.
To capture a dynamic web table, wait for a condition tied to its actual rows or cell content, then read the rendered rows or take a screenshot. Page navigation completing does not mean JavaScript has finished fetching and rendering the table.
This guide shows how to wait for table content with Playwright and Selenium, extract the rows, and capture the populated table. Adapt the selectors and expected values to the site you are automating.
1. Choose a table readiness condition
First identify what proves that the table is ready for your task. Prefer a positive signal connected to the data you need:
- A known row or cell becomes visible.
- A cell contains an expected value.
- A result count reaches the expected value.
- A loading indicator disappears, ideally alongside a check that rows are present.
Avoid relying on navigation completion alone. JavaScript can update a page after the document reaches its selected ready state. A row may not exist when navigation returns. Likewise, networkidle means no network connections for at least 500 ms; it does not prove the table contains the desired data. Playwright discourages using that state as a general test readiness signal. Use the condition that describes the state you actually need. Selenium’s waits documentation and the Playwright Page API describe these waiting behaviors.
2. Playwright: wait, extract rows, and take a screenshot
Install Playwright Test in a Node.js project with npm init playwright@latest, following the setup prompts, or add the packages directly with npm install -D @playwright/test. The following example is a Playwright Test test. Replace the URL, selector, and expected cell text with values from the page.
import { test, expect } from '@playwright/test';
test('capture the populated results table', async ({ page }) => {
await page.goto('https://example.com/results');
const table = page.locator('table#results');
const expectedRow = table.locator('tbody tr', {
hasText: 'Expected value',
});
// Playwright Test retries this assertion until it passes or times out.
await expect(expectedRow).toBeVisible({ timeout: 15000 });
const rows = await table.locator('tbody tr').evaluateAll((elements) =>
elements.map((row) =>
Array.from(
row.querySelectorAll('th, td'),
(cell) => cell.textContent?.trim() ?? ''
)
)
);
console.log(rows);
await table.screenshot({ path: 'results-table.png' });
});Run it with npx playwright test. Playwright locator actions and assertions auto-wait where appropriate. The locator screenshot scrolls the target into view and waits for it to be ready for the action. For a focused screenshot, table.screenshot() captures the table element; use page.screenshot() if you need the entire viewport or page instead.
The example waits for one expected row. If the task requires a known number of rows, wait for that count as well, for example with await expect(table.locator('tbody tr')).toHaveCount(20). Only use an exact count when it is a real completion condition for that page; a changing or paginated dataset may not have a fixed count.
3. Selenium: wait for a row or cell condition
Install Selenium with python -m pip install selenium and configure a browser driver supported by your Selenium installation. This Python example waits until a matching row is visible, then extracts its cells. Adapt the selector and expected text to the target page.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
URL = "https://example.com/results"
options = webdriver.ChromeOptions()
# Uncomment for a headless run:
# options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
driver.get(URL)
wait = WebDriverWait(driver, 15)
table = wait.until(
EC.presence_of_element_located((By.CSS_SELECTOR, "table#results"))
)
expected_row = wait.until(
EC.visibility_of_element_located((
By.XPATH,
"//table[@id='results']//tbody/tr[contains(., 'Expected value')]"
))
)
rows = []
for row in table.find_elements(By.CSS_SELECTOR, "tbody tr"):
cells = row.find_elements(By.CSS_SELECTOR, "th, td")
rows.append([cell.text.strip() for cell in cells])
print(rows)
table.screenshot("results-table.png")
finally:
driver.quit()presence_of_element_located only establishes that an element exists in the DOM. The second wait checks a visible row containing the expected text, which ties readiness to the rendered content. Selenium explicit waits poll for the condition you specify. Do not combine implicit and explicit waits: Selenium warns that the resulting wait duration can be unpredictable. See the Selenium waiting documentation.
4. Select the right capture and extraction scope
Extracting structured data
After the readiness condition passes, map each rendered row to its header and cell text. The Playwright and Selenium examples read the currently rendered tbody tr elements. If the table includes nested markup, buttons, or hidden columns, inspect the DOM and adjust the cell selector or extraction logic. For machine-readable results, preserve column headers alongside each row so values remain interpretable.
Capturing an image
Use an element screenshot when the table itself is the deliverable. Use a page screenshot if surrounding headings, filters, or context matter. Make sure the target is visible and not covered by a sticky header, modal, or consent prompt; those can affect what appears in the image.
Pagination, virtualization, and scrolling
A DOM read or element screenshot includes only what the page has rendered. On paginated tables, move through each page, wait for that page’s data condition, and collect or capture it before continuing. On virtualized tables, rows outside the visible area may not exist in the DOM yet. Scroll the table to trigger additional rendering, wait for newly rendered rows, and repeat. Do not describe a viewport screenshot as the complete dataset unless you have verified that all required rows were rendered and captured.
Infinite-scroll tables need a stopping condition such as a known last row, an end-of-results marker, or a verified total count. Without one, a script can stop early or continue indefinitely.
5. Troubleshoot common failures
| Symptom | Likely cause | Fix |
|---|---|---|
| Wait times out before a row appears | The selector or expected text does not match, data failed to load, or the page needs authentication or another interaction. | Inspect the live DOM and page state; confirm the selector, expected value, and prerequisites. Keep the timeout finite and report a timeout as a readiness failure. |
| Navigation succeeds but the table is empty | Navigation completion was mistaken for data readiness. | Wait for a row, cell value, result count, or other table-specific completion condition. |
networkidle passes but rows are missing |
Network quiet is not equivalent to table completion; data may be delayed or rendered through another mechanism. | Wait for expected content directly instead of treating network quiet as proof. |
| Extracted rows are incomplete | The table is paginated, virtualized, or scroll-loaded. | Navigate pages or scroll to render each segment, waiting for its content before extraction. |
| Screenshot is clipped or misses the table | The locator targets the wrong element, the table is not in view, or only the current rendered portion exists. | Use a stable table locator, capture the element after readiness, and handle scrolling or pagination explicitly. |
| Selenium takes much longer than the configured wait | Implicit and explicit waits may be interacting. | Remove the implicit wait and use explicit waits for the conditions needed by the workflow. |
6. Reliability, speed, and cost
Use stable selectors owned by the application or based on accessible roles and names where available. Positional selectors such as “the third table on the page” break when layout changes. Choose a timeout long enough for the application’s normal response, but finite so a genuine failure is visible. A timeout is not evidence that the table is complete; treat it as a failed capture and retain enough context to diagnose the page state.
Waiting for one meaningful row is often more direct than waiting for broad page activity to stop. If completeness matters, add a count or end-of-results check. Screenshots and DOM extraction are separate outputs: a screenshot documents appearance, while extracted rows support downstream data work. Browser automation cost depends on the browser runtime and how many pages, interactions, and captures your workflow runs; avoid rerunning expensive browser setup when a simpler capture meets the need.
7. Or skip the browser setup
For a rendered page screenshot through an API, ScreenshotNeo accepts a URL in one GET request and returns an image or PDF. This does not extract table cells into structured data, and a URL-only request cannot wait on a page-specific expected row. Use Playwright or Selenium when that content condition or structured row extraction is required. See the ScreenshotNeo API documentation.
curl -G "https://api.screenshotneo.com/v1/shot" \
-d access_key=YOUR_API_KEY \
--data-urlencode url=https://example.com/results \
-o table.webpimport requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://example.com/results"},
timeout=90,
)
r.raise_for_status()
open("table.webp", "wb").write(r.content)const q = new URLSearchParams({
access_key: 'YOUR_API_KEY',
url: 'https://example.com/results',
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
const image = Buffer.from(await res.arrayBuffer());
await import('node:fs/promises').then(({ writeFile }) =>
writeFile('table.webp', image)
);ScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each of those steps can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and response headers report the page verdict and billing status. An MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents. The free plan includes 1,000 screenshots each month with no card; paid plans start at $5 for 3,000 shots. Every feature is on every plan. Create a free ScreenshotNeo account.
8. Frequently asked questions
Should I use Playwright or Selenium?
Use the framework that fits your project’s language, browser setup, and existing selectors. Both support waiting for a page-specific condition; neither is universally better for every table.
Can I capture a table without extracting its rows?
Yes. Wait for the table’s expected content, then take a screenshot of the table locator. This preserves its rendered appearance but does not produce structured data.
Does waiting for a row prove every result loaded?
No. It proves only the condition you checked. For complete results, also verify a count, final page, or end-of-results marker that matches the application’s behavior.
Why can a screenshot API not guarantee my asynchronous table is ready?
A URL-based capture does not know which row or cell proves completion for your application. Use browser automation when capture must depend on that page-specific condition.
