Usage

Minimal example: scrape Hacker News top stories

This is the canonical example from the official repo. It loads Hacker News, passes the page content to GPT-4o, and extracts the top 5 stories as a typed object.

import { chromium } from 'playwright'
import { z } from 'zod'
import { Output } from 'ai'
import { openai } from '@ai-sdk/openai'
import LLMScraper from 'llm-scraper'

const browser = await chromium.launch()
const llm = openai('gpt-4o')
const scraper = new LLMScraper(llm)

const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')

const schema = z.object({
  top: z.array(
    z.object({
      title: z.string(),
      points: z.number(),
      by: z.string(),
      commentsURL: z.string(),
    })
  ).length(5),
})

const { data } = await scraper.run(page, Output.object({ schema }))
console.log(data.top)

await browser.close()

Content format modes

By default LLM Scraper sends the page as markdown. You can override the format:

// Send a screenshot instead (requires a multimodal model)
const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'image',
})

// Send raw HTML
const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'html',
})

// Send plain text extracted by Readability.js
const { data } = await scraper.run(page, Output.object({ schema }), {
  format: 'text',
})

Available formats: html, raw_html, markdown, text, image, custom

Streaming output

For large pages, stream partial results as the LLM generates them:

const { stream } = await scraper.stream(page, Output.object({ schema }))

for await (const partial of stream) {
  console.log(partial)
}

Using Anthropic Claude instead of OpenAI

import { anthropic } from '@ai-sdk/anthropic'
import LLMScraper from 'llm-scraper'

const llm = anthropic('claude-3-5-sonnet-20240620')
const scraper = new LLMScraper(llm)
// rest of the code is identical

Using a local Ollama model

import { ollama } from 'ollama-ai-provider-v2'
import LLMScraper from 'llm-scraper'

const llm = ollama('llama3')
const scraper = new LLMScraper(llm)

Ollama must be running locally (ollama serve) with the target model pulled (ollama pull llama3).