Skip to content
Website & Domain

Website Scraper

Any URL as clean Markdown and metadata, ready for LLM and RAG pipelines.

POST/v1/website-scraper
10 credits / pageBulk supportedGlobal
Request
curl -X POST https://api.1lookup.dev/v1/website-scraper \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/pricing"}'
Response
{
  "success": true,
  "data": {
    "url": "https://example.com/pricing",
    "final_url": "https://example.com/pricing",
    "status_code": 200,
    "title": "Pricing | Example",
    "description": "Simple plans for clinics of every size. Start free, upgrade when you need more seats.",
    "markdown": "# Pricing\n\nSimple plans for clinics of every size.\n\n## Starter\n\n$19 per seat per month. Online booking, reminders and one location.\n\n## Practice\n\n$49 per seat per month. Everything in Starter, plus multiple locations and reporting.\n\n[Contact sales](https://example.com/contact)",
    "links": [
      {
        "text": "Contact sales",
        "url": "https://example.com/contact"
      },
      {
        "text": "Documentation",
        "url": "https://example.com/docs"
      },
      {
        "text": "Terms",
        "url": "https://example.com/terms"
      }
    ],
    "metadata": {
      "language": "en",
      "canonical_url": "https://example.com/pricing",
      "og_image": "https://example.com/og/pricing.png",
      "word_count": 36
    }
  },
  "meta": {
    "credits_used": 10,
    "credits_remaining": 999,
    "cached": false,
    "request_id": "req_8f3a2c1e9b7d"
  }
}

Authorization

Authorizationstring<bearer>required

Your API key as a Bearer token. Live keys start with 1lk_live_. Test keys start with 1lk_test_ and return this endpoint's sample response without spending credits. See authentication.

Body

Send the fields as a JSON object with Content-Type: application/json.

urlstring<uri>required

URL. Example: https://example.com/pricing

freshboolean

Skip the 7-day cache and run a new lookup. Charged as a normal lookup.

referencestring

Your own identifier, echoed back in meta and shown in lookup history.

Response

A 200 response carries the result in data and the cost of the call in meta. These are the fields most integrations read.

successboolean

true when the lookup ran.

data.markdownstring

The main page content as Markdown, with headings, lists and links preserved.

data.titlestring

The page title.

data.descriptionstring

The meta description, when the page sets one.

data.linksarray

Links found on the page, each with its anchor text and absolute URL.

data.metadataobject

Language, canonical URL, social preview image and word count.

data.status_codenumber

The status the page returned, so you can tell a 404 from real content.

data.final_urlstring

Where the request ended up after redirects.

meta.credits_usednumber

Credits charged for this call.

meta.credits_remainingnumber

Your balance after the call.

meta.cachedboolean

true when the result came from the 7-day cache.

meta.request_idstring

Quote this when you contact support.

Errors

A failed request returns success: false with an error object, and is never charged. The full list is on the errors page.

HTTPCodeMeaning
401unauthorizedThe key is missing, malformed or revoked.
402insufficient_creditsYour credits are used up and pay as you go is off or at its limit.
422invalid_inputA field is missing or malformed. The field property names it.
429rate_limitedToo many requests for this key. Wait for the Retry-After interval.
500internal_errorSomething failed on our side. Safe to retry.
503source_unavailableAn upstream source did not answer in time. Safe to retry.

Billing

10 credits / page. Charged when the lookup runs and returns a result. Errors are never charged, and repeating the same input within 7 days is served from cache at no charge.

Product overview and use cases