Web Intelligence API

Extract, audit, and train from any page on the web.

Crawlee fetches pages through a TLS-spoofing client, fingerprints the underlying platform, picks the right extraction strategy, and returns clean structured data. No browser. No headless overhead.

View endpoints Authentication

15 Platform detectors

7 Extraction strategy families

3 Core endpoints

36 Regression tests

Endpoints

Three workflows. One base URL.

POST

Adaptive Scrape

/api/v1/scrape

Detects the site platform, selects the matching extraction strategy, and returns markdown, internal and external links, images, and optional token-aware chunks. Falls back deterministically, never silently.

POST

Security Audit

/api/v1/security-audit

Reads raw HTML before any extractor cleans it. Checks for hidden CSS injections, off-screen text, HTML comment vectors, hostile attributes, script-embedded secrets, and zero-width obfuscation.

POST

Dataset Generator

/api/v1/dataset

Parses documentation-style pages into synthetic Q&A pairs. Exports in OpenAI ChatML, Alpaca, ShareGPT, and DPO preference formats. Includes quality scores and token estimates per pair.

Quick start

Copy and run.

Pick an endpoint and language. Your API key goes in the X-Api-Key header.

curl -X POST https://dock.sluxia.com/api/v1/scrape \
  -H 'Content-Type: application/json' \
  -H 'X-Api-Key: YOUR_KEY' \
  -d '{
    "url": "https://stripe.com/docs",
    "fit_markdown": true,
    "chunk_size": 500,
    "chunk_overlap": 80
  }'

const res = await fetch('https://dock.sluxia.com/api/v1/scrape', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-Api-Key': 'YOUR_KEY'
  },
  body: JSON.stringify({
    url: 'https://stripe.com/docs',
    fit_markdown: true,
    chunk_size: 500,
    chunk_overlap: 80
  })
});
const data = await res.json();

import requests

data = requests.post(
    'https://dock.sluxia.com/api/v1/scrape',
    headers={
        'Content-Type': 'application/json',
        'X-Api-Key': 'YOUR_KEY'
    },
    json={
        'url': 'https://stripe.com/docs',
        'fit_markdown': True,
        'chunk_size': 500,
        'chunk_overlap': 80
    }
).json()

curl -X POST https://dock.sluxia.com/api/v1/security-audit \
  -H 'Content-Type: application/json' \
  -H 'X-Api-Key: YOUR_KEY' \
  -d '{
    "url": "https://example.com"
  }'

const res = await fetch('https://dock.sluxia.com/api/v1/security-audit', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-Api-Key': 'YOUR_KEY'
  },
  body: JSON.stringify({ url: 'https://example.com' })
});
const data = await res.json();

import requests

data = requests.post(
    'https://dock.sluxia.com/api/v1/security-audit',
    headers={
        'Content-Type': 'application/json',
        'X-Api-Key': 'YOUR_KEY'
    },
    json={'url': 'https://example.com'}
).json()

curl -X POST https://dock.sluxia.com/api/v1/dataset \
  -H 'Content-Type: application/json' \
  -H 'X-Api-Key: YOUR_KEY' \
  -d '{
    "url": "https://developers.cloudflare.com/fundamentals/",
    "min_confidence": 0.85
  }'

const res = await fetch('https://dock.sluxia.com/api/v1/dataset', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-Api-Key': 'YOUR_KEY'
  },
  body: JSON.stringify({
    url: 'https://developers.cloudflare.com/fundamentals/',
    min_confidence: 0.85
  })
});
const data = await res.json();

import requests

data = requests.post(
    'https://dock.sluxia.com/api/v1/dataset',
    headers={
        'Content-Type': 'application/json',
        'X-Api-Key': 'YOUR_KEY'
    },
    json={
        'url': 'https://developers.cloudflare.com/fundamentals/',
        'min_confidence': 0.85
    }
).json()

Authentication

One header. Every request.

Pass your key in the X-Api-Key header on every call. The health endpoint is always open for uptime monitoring. Everything else is gated.

GET /api/v1/health Always open. No key required.

POST /api/v1/scrape Requires X-Api-Key header.

POST /api/v1/security-audit Requires X-Api-Key header.

POST /api/v1/dataset Requires X-Api-Key header.

# Missing key
HTTP 401 {"detail": "Missing X-Api-Key header"}

# Wrong key
HTTP 403 {"detail": "Invalid API key"}

# Correct key
HTTP 200 {"success": true, ...}