Extract and Count Link Domains using Firecrawl

Scrapes a webpage using Firecrawl to extract all outgoing links and aggregates them by domain using jq.

$ firecrawl scrape "https://news.ycombinator.com" --json | jq -r '.data.linksOnPage[]? | select(test("^https?://")) | capture("https?://(?<domain>[^/]+)").domain' | sort | uniq -c | sort -nr | head -n 5

1. [01] firecrawlfirecrawl scrape "https://news.ycombinator.com" --json
   Scrapes the target URL and outputs the extracted page data and metadata as JSON.
2. [02] jqjq -r '.data.linksOnPage[]? | select(test("^https?://")) | capture("https?://(?<…
   Parses the JSON output, filters for valid HTTP/HTTPS URLs, and uses a regex capture to extract just the domain name.
3. [03] sortsort | uniq -c
   Alphabetically sorts the extracted domains to group identical ones, then counts the occurrences of each.
4. [04] headsort -nr | head -n 5
   Sorts the counted domains in descending numerical order and limits the output to the top 5 most frequently linked domains.

  45 news.ycombinator.com
  12 github.com
   5 en.wikipedia.org
   3 twitter.com
   2 www.nytimes.com

• Requires a valid Firecrawl API key set in your environment variables; scraping large sites may consume your API credits quickly.
• The regex in jq assumes standard http/https URLs and will drop relative paths or malformed links.