← All one-liners·#077·scraping·self contained·power

Scrape a live page with CSS selectors — no LLM, no API key

htmlq applies jq-style querying to HTML: give it a CSS selector, get back matched text or attributes. No Firecrawl call, no headless browser, no rendering — just curl piped into a real HTML parser, the same selector syntax you'd type into browser devtools.

Setup
  • → cargo install htmlq # Rust; no apt or npm package as of Sep 2026
Cost per run
free
The one-liner
$ curl -sS "https://pkg.go.dev/search?q=cli" | htmlq -a data-clicked-package '.SearchSnippet h2 a' | sort -u
What each stage does
  1. [01] curlcurl -sS "https://pkg.go.dev/search?q=cli"
    Fetch the rendered search-results HTML for the query 'cli' from Go's official package index — a real server-rendered page, not a JSON API.
  2. [02] htmlqhtmlq -a data-clicked-package '.SearchSnippet h2 a'
    CSS-select every `<a>` inside an `<h2>` inside a `.SearchSnippet` block, then print its `data-clicked-package` attribute instead of the visible link text — `-a` switches htmlq from text extraction to attribute extraction.
  3. [03] sortsort -u
    Dedupe and alphabetize the extracted Go import paths.
Expected output (sample)
code.cloudfoundry.org/cli/util/configv3
github.com/apex/log/handlers/cli
github.com/cli/go-gh/v2
github.com/cloudflare/cfssl/cli
github.com/cometbft/cometbft/libs/cli
github.com/cosmos/cosmos-sdk/x/auth/client/cli
github.com/desertbit/grumble
github.com/docker/cli/cli/command
github.com/jawher/mow.cli
github.com/micro/cli/v2
github.com/minio/cli
github.com/mitchellh/cli
... (25 unique packages total)
Caveats & tips
  • Verified on htmlq 0.4.0 (built via `cargo install htmlq`) against the live pkg.go.dev search page on 2026-09-02.
  • htmlq panics with a BrokenPipe backtrace if a downstream command closes the stream early (e.g. piping straight into `head`) — harmless to the extracted data already printed, but pipe into something that consumes the full stream (sort, wc, grep) instead.
  • CSS class names are markup-version-dependent: any site, pkg.go.dev included, can rename them on a redesign — re-inspect with devtools if a selector stops matching.
  • Only works on server-rendered HTML; a client-side-rendered (JS-hydrated) page needs something that executes JS first — this pattern is the no-JS-needed half of the spectrum.