ScreenshotNeo
Sign in Get a free key

BlogEngineering

How to Process Screen-Recording Frames in Node.js with FFmpeg

Extract, sample, process, and stream screen-recording frames in Node.js with FFmpeg, including reliable code, formats, timing, backpressure, and troubleshooting.

By the ScreenshotNeo team1 October 20268 min read

Direct answer: Use FFmpeg’s image2 muxer to decode the recording and write numbered images, then orchestrate FFmpeg from Node.js with child_process.spawn(). A filter such as fps=1 extracts one frame per second without loading the entire video into memory.

import { spawn } from 'node:child_process';
import { mkdir } from 'node:fs/promises';

await mkdir('frames', { recursive: true });

const args = [
  '-hide_banner',
  '-loglevel', 'error',
  '-i', 'recording.mp4',
  '-vf', 'fps=1',
  '-start_number', '0',
  'frames/frame-%05d.png'
];

const ffmpeg = spawn('ffmpeg', args, {
  stdio: ['ignore', 'ignore', 'pipe']
});

let diagnostics = '';
ffmpeg.stderr.setEncoding('utf8');
ffmpeg.stderr.on('data', chunk => { diagnostics += chunk; });

const exitCode = await new Promise((resolve, reject) => {
  ffmpeg.once('error', reject);
  ffmpeg.once('close', resolve);
});

if (exitCode !== 0) {
  throw new Error(`ffmpeg failed (${exitCode}): ${diagnostics}`);
}

console.log('Frames written to frames/');

This creates frame-00000.png, frame-00001.png, and so on. The input is decoded incrementally by FFmpeg; Node.js only supervises the process and collects diagnostics.

1. Install FFmpeg and verify the input

Install FFmpeg using your operating system’s package manager or the official builds, then verify that it is available to the account running Node.js:

ffmpeg -version
ffprobe -v error -show_format -show_streams recording.mp4

The ffprobe output helps you confirm duration, dimensions, frame rate, codec, and whether the file contains a video stream. FFmpeg’s documentation covers decoding and image extraction in its ffmpeg command-line documentation and the formats and muxers documentation.

2. Choose the sampling behavior

Sampling rate, seek position, duration, and frame count are separate controls. Decide which one describes your job before assembling the argument array.

Goal FFmpeg arguments Notes
One frame each second -vf fps=1 Filter-based sampling; output count follows the selected time range.
Five frames each second -vf fps=5 Produces more images and increases decoding, processing, and storage work.
Start at a position -ss 00:02:00 Places the seek before or after -i depending on the desired speed and precision.
Limit the time window -t 00:00:10 Stops after ten seconds of output.
Limit the number of frames -frames:v 1 Useful for a single representative image.

Use -r when you specifically need output frame-rate behavior. For extraction, -vf fps=... makes the sampling rule explicit and is usually easier to reason about.

3. Runnable Node.js examples

Extract one frame at a timestamp

import { spawn } from 'node:child_process';

function runFfmpeg(args) {
  return new Promise((resolve, reject) => {
    const child = spawn('ffmpeg', args, { stdio: ['ignore', 'ignore', 'pipe'] });
    let stderr = '';
    child.stderr.setEncoding('utf8');
    child.stderr.on('data', chunk => { stderr += chunk; });
    child.once('error', reject);
    child.once('close', code => {
      if (code === 0) resolve();
      else reject(new Error(`FFmpeg exited with ${code}: ${stderr}`));
    });
  });
}

await runFfmpeg([
  '-hide_banner', '-loglevel', 'error',
  '-ss', '00:00:12.500',
  '-i', 'recording.mp4',
  '-frames:v', '1',
  'frame-at-12-5s.png'
]);

Use this for thumbnails, audit checkpoints, or a single representative frame. FFmpeg documents -ss seeking and -frames:v 1 output limiting.

Extract a bounded interval as WebP

import { mkdir } from 'node:fs/promises';
import { spawn } from 'node:child_process';

await mkdir('frames', { recursive: true });

const child = spawn('ffmpeg', [
  '-hide_banner', '-loglevel', 'error',
  '-ss', '00:02:00',
  '-i', 'recording.mp4',
  '-t', '00:00:10',
  '-vf', 'fps=2',
  'frames/frame-%05d.webp'
], { stdio: ['ignore', 'ignore', 'pipe'] });

let stderr = '';
child.stderr.setEncoding('utf8');
child.stderr.on('data', chunk => { stderr += chunk; });
child.once('error', err => { throw err; });
child.once('close', code => {
  if (code !== 0) console.error(stderr);
});

Ten seconds at two frames per second produces approximately twenty images. The exact count can vary at stream boundaries, so treat the output directory as the source of truth.

Use a wrapper: fluent-ffmpeg

fluent-ffmpeg reduces command assembly and exposes lifecycle events and screenshot helpers:

import ffmpeg from 'fluent-ffmpeg';

ffmpeg('recording.mp4')
  .outputOptions(['-vf', 'fps=1'])
  .on('error', error => console.error(error.message))
  .on('end', () => console.log('frames complete'))
  .save('frames/frame-%05d.png');

The wrapper is convenient for sparse thumbnails and timemarks. Direct spawn keeps every FFmpeg argument visible, makes stderr handling explicit, and avoids hiding argument-order details. The fluent-ffmpeg screenshot recipe documents options such as count, timemarks, output folder, filename tokens, size, and fast seeking.

4. Naming, formats, and timestamps

A numbered pattern such as frame-%05d.png is handled by FFmpeg’s image2 muxer. The five digits provide stable lexical ordering through frame 99,999. Use -start_number 0 or another starting index to match your application.

  • PNG: lossless pixels; useful for OCR, pixel comparison, and archival screenshots.
  • JPEG: smaller files for photographic or screen content where some loss is acceptable.
  • WebP: often a practical size/quality compromise when your downstream tools support it.

When filenames must represent presentation timestamps rather than sequence numbers, investigate FFmpeg’s timestamp-oriented options such as frame_pts and strftime. Do not assume that a frame number equals wall-clock time when the source has variable timing, a nonstandard time base, or dropped frames.

5. Process each frame without retaining the whole recording

For most applications, write frames to a temporary directory and process them as a bounded queue:

  1. Start extraction with a controlled rate and time window.
  2. Read one completed frame path.
  3. Run OCR, computer vision, compression, or another analysis step.
  4. Persist the result.
  5. Delete or archive the frame before taking the next item.

A simple sequential worker avoids an unbounded array:

import { readdir, unlink } from 'node:fs/promises';
import path from 'node:path';

async function analyzeFrame(filePath) {
  // Replace this with OCR or image analysis.
  return { filePath, processedAt: new Date().toISOString() };
}

const names = (await readdir('frames'))
  .filter(name => name.endsWith('.png'))
  .sort();

for (const name of names) {
  const filePath = path.join('frames', name);
  const result = await analyzeFrame(filePath);
  console.log(result);
  await unlink(filePath);
}

For very large recordings, consider a producer-consumer queue with a fixed concurrency limit. The limit is part of your backpressure policy: if analysis is slower than decoding, pause or bound the producer rather than allowing buffers and temporary files to grow without limit.

Streaming raw frames

FFmpeg can write a pipe instead of image files, but raw video has no self-describing frame boundaries. Your Node.js process must know the width, height, pixel format, and therefore the exact bytes per frame. Parse complete frame-sized chunks, retain incomplete remainder bytes between reads, and apply backpressure when the consumer is slow. File output is simpler when you do not need a zero-copy pipeline.

6. Reliability and observability

  • Listen for both the child’s error event (for example, an absent executable) and its close event (the process finished).
  • Capture stderr even with -loglevel error; it contains codec, demuxer, permission, and output-path diagnostics.
  • Check the exit code before consuming output as if the job succeeded.
  • Use unique temporary directories for concurrent recordings.
  • Pin the FFmpeg binary and version in deployment, and log the exact argument array.
  • Clean partial output after a failed job so a retry cannot mistake old frames for new ones.

For long-running services, add a job timeout and terminate the child process if the input is corrupt or a decoder stalls. Keep retries bounded; retrying a deterministic “unknown decoder” error will not fix the input.

7. Performance, storage, and cost considerations

There is no universal throughput or memory number. Work depends on codec, resolution, duration, sampling rate, filters, storage, and hardware, so measure with the recordings and deployment environment you actually use.

Control Effect
Lower fps Fewer decodes, analyses, files, and bytes.
Shorten -t Bounds work and temporary storage.
Use JPEG/WebP Reduces storage at the expense of format-specific quality or compatibility trade-offs.
Resize during extraction Reduces downstream CPU and storage when full resolution is unnecessary.
Process with bounded concurrency Prevents a fast extractor from overwhelming OCR or CV workers.

Seek placement also affects speed and precision. A fast seek may begin near a keyframe, while accurate extraction can require decoding from an earlier point. Validate the selected frame when timestamp accuracy matters.

8. Troubleshooting

Symptom Likely cause Fix
spawn ffmpeg ENOENT FFmpeg is not installed or is absent from the service PATH. Install it, use an absolute configured path, and verify with ffmpeg -version under the same account.
“Invalid data found when processing input” Corrupt, truncated, unsupported, or mislabeled input. Run ffprobe, obtain a complete recording, or transcode it to a supported format.
No output images The seek and duration select an empty interval, or the file has no video stream. Inspect stream metadata and test without -ss/-t.
Frames are unexpectedly dark or blank Decoder or color conversion behavior differs from the source. Try another output format, inspect the source stream, and compare a frame produced by a media player.
Only a few frames appear Low source frame rate, an overly short duration, or an aggressive seek. Check duration and frame-rate metadata; adjust -vf fps, -ss, and -t.
Node process uses too much memory Frames or pipe chunks are accumulated without bounds. Process files sequentially or enforce a fixed-size queue and consume complete pipe frames.
Output is overwritten Concurrent jobs share a directory or filename pattern. Use a unique directory per job and clean it after completion.
Wrapper emits an opaque error The generated command or FFmpeg stderr is hidden. Enable command logging, capture lifecycle errors, or reproduce with direct spawn.

9. Or skip the browser setup

If what you really need is a current screenshot of a web page rather than frames from a local recording, ScreenshotNeo provides a single HTTP request that returns PNG, JPEG, WebP, or PDF. See the ScreenshotNeo API documentation for all options.

const q = new URLSearchParams({
  access_key: 'YOUR_API_KEY',
  url: 'https://stripe.com'
});
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`ScreenshotNeo failed: ${res.status}`);
const image = Buffer.from(await res.arrayBuffer());
await writeFile('shot.webp', image);

Equivalent requests:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)

Cookie banners, newsletter popups, and chat widgets are removed before the shot. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed; response headers report the page verdict and billing status. An MCP server lets Claude, Cursor, and other MCP clients call take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots per month with no card, and paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.

10. FAQ

Can I extract every original frame?

Yes. Omit the sampling filter when you need the decoder’s frame sequence, but expect substantially more CPU, storage, and processing work than a sampled extraction.

Should I use PNG for OCR?

PNG preserves pixels and avoids additional lossy compression. Validate your OCR engine with the actual screen content; format alone does not guarantee recognition quality.

Is a frame number a timestamp?

Not necessarily. Variable timing, time bases, dropped frames, and seeking can make the relationship nontrivial. Store or derive timestamps explicitly when they matter.

When is fluent-ffmpeg preferable?

Use it when its helpers and event model make common screenshot jobs easier to maintain. Use direct spawning when exact flags, argument order, stderr, and dependency control are priorities.

How do I prevent a failed run from being treated as successful?

Reject on the child error event, require an exit code of zero, inspect stderr, and verify that expected output files exist before publishing results.

Keep reading