Extract and Group Page Links by Domain using Firecrawl

Scrapes a webpage using the Firecrawl API to extract all links, then uses jq to parse, group, and count the occurrences of each domain.

$ curl -sX POST https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com", "formats": ["links"]}' \
  | jq -r '.data.links | map(sub("https?://";"") | split("/")[0]) | group_by(.) | map({domain: .[0], count: length}) | sort_by(-.count)[] | "\(.count)\t\(.domain)"'

1. [01] curlcurl -sX POST https://api.firecrawl.dev/v1/scrape -d '{"url": "...", "formats": …
   Calls the Firecrawl API to scrape the target URL, requesting only the extracted links format to minimize payload size.
2. [02] jq.data.links
   Extracts the raw array of URLs from the Firecrawl JSON response.
3. [03] jqmap(sub("https?://";"") | split("/")[0])
   Strips the http/https protocol and path from each URL, leaving just the base domain name.
4. [04] jqgroup_by(.) | map({domain: .[0], count: length})
   Groups identical domains together and maps them into objects containing the domain name and its frequency count.
5. [05] jqsort_by(-.count)[] | "\(.count)\t\(.domain)"
   Sorts the domains by count in descending order and formats the output as tab-separated text.

142	news.ycombinator.com
4	github.com
2	en.wikipedia.org
1	www.nytimes.com
1	blog.cloudflare.com

• Footgun: Relative links scraped by Firecrawl might not contain the domain, which could result in empty or malformed domain strings in the jq output depending on the target site's structure.
• Cost/Permission: Requires a valid Firecrawl API key. Scraping large or multiple sites may consume your API credits quickly.