Query Firecrawl search results directly with DuckDB

Scrapes web search results via the Firecrawl API and pipes the JSON directly into DuckDB for SQL analysis.

$ curl -sX POST https://api.firecrawl.dev/v1/search \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "duckdb release notes"}' \
  | duckdb -c "SELECT unnest(data).title AS title, unnest(data).url AS url FROM read_json_auto('/dev/stdin');"

1. [01] curlcurl -sX POST https://api.firecrawl.dev/v1/search
   Calls the Firecrawl API search endpoint to retrieve web results.
2. [02] curl-H "Authorization: Bearer $FIRECRAWL_API_KEY"
   Authenticates the request using your Firecrawl API key.
3. [03] curl-d '{"query": "duckdb release notes"}'
   Passes the search query as a JSON payload.
4. [04] duckdb| duckdb -c "..."
   Pipes the JSON response into DuckDB for immediate querying.
5. [05] duckdbSELECT unnest(data).title AS title, unnest(data).url AS url FROM read_json_auto(…
   Unnests the 'data' array from the JSON response and extracts the title and URL fields.

┌──────────────────────────────┬────────────────────────────────────────┐
│            title             │                  url                   │
│           varchar            │                varchar                 │
├──────────────────────────────┼────────────────────────────────────────┤
│ DuckDB 0.10.0 Release Notes  │ https://duckdb.org/2024/02/13/annou... │
│ Releases · duckdb/duckdb     │ https://github.com/duckdb/duckdb/re... │
└──────────────────────────────┴────────────────────────────────────────┘

• Footgun: DuckDB's `read_json_auto` might fail if the Firecrawl API returns an error message instead of the expected schema (e.g., rate limit exceeded).
• Cost/Permission: Requires a valid Firecrawl API key; searches consume credits from your Firecrawl account.