Extract and Group Outbound Links via Firecrawl

Scrapes a webpage using the Firecrawl API to extract all links, then uses jq and shell utilities to find the most frequently linked domains.

$ curl -sX POST https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com", "formats": ["links"]}' \
  | jq -r '.data.links[] | select(startswith("http")) | split("/")[2]' \
  | sort | uniq -c | sort -nr \
  | head -n 5

1. [01] curlcurl -sX POST https://api.firecrawl.dev/v1/scrape ... -d '{"url": "...", "format…
   Calls the Firecrawl API to scrape the target URL, explicitly requesting only the extracted links in the JSON response.
2. [02] jqjq -r '.data.links[] | select(startswith("http")) | split("/")[2]'
   Iterates over the links array, filters for absolute URLs, and splits the string by '/' to isolate the domain name.
3. [03] sortsort | uniq -c | sort -nr
   Groups the extracted domains, counts their occurrences, and sorts the results in descending order by frequency.
4. [04] headhead -n 5
   Limits the final output to the top 5 most frequently linked domains on the page.

  14 news.ycombinator.com
   3 github.com
   2 en.wikipedia.org
   1 www.nytimes.com
   1 arxiv.org

• Footgun: Relative links (e.g., `/about`) are filtered out by the `startswith("http")` selector, so internal link counts may be underrepresented if the site uses relative paths.
• Cost/Permission: Requires a valid Firecrawl API key; scraping large sites or using advanced extraction formats consumes your monthly API credits.