Rewrite github-cli as a keyless reader of github.com - #3

Node.js 20 is being deprecated in the Actions runtime.

  actions/checkout              → v7.0.0
  browser-actions/setup-chrome  → v2.1.2
  golangci/golangci-lint-action → v9.2.1
  goreleaser/goreleaser-action  → v7.2.2
  docker/setup-qemu-action      → v4.1.0
  docker/setup-buildx-action    → v4.1.0
  docker/login-action           → v4.2.0
  sigstore/cosign-installer     → v4.1.2
  anchore/sbom-action           → v0.24.0

The foundation of the rewrite: URI parsing, the surface table, the
per-field merge, the repository page decoder, and every search type that
answers without a session.

Search is the widest surface here. Nine of the ten types return real
results to an anonymous client. The tenth is code search, which answers
200 with an empty result set, so it exits 7 with a message saying why
rather than reporting no matches.

Three things the live probes settled:

The sidebarAbout sections block is mostly booleans meaning "this box is
on the page", with only releases and usedBy arriving as objects. A typed
struct there loses the whole block the first time a member is true
instead of {}, so each member is decoded on its own.

There is no keyless language histogram. /graphs/languages 301s back to
the repository page, show_partial and the other guesses 404, and the
sidebar language bar is a loading skeleton on a cold fetch. The primary
language comes from a repository search instead, one hop, recorded in
_via so nobody mistakes it for a byte count.

Tree entry ids use owner/name@ref/path with a slash. A colon there put
the filename inside the ref and produced tree URLs nobody could follow.

Profiles are the one part of the site with no JSON at all. Asking for
application/json on /{login} returns the 2012 deprecation notice for the
v2 API, and the page carries no react-app.embeddedData block, so every
field comes from microdata, a microformat class, a data attribute, or a
counted link.

A user profile and an org page are different templates, not one template
with parts hidden. Which one arrived decides Type, because nothing about
the string "golang" says organization.

Two things worth writing down. The user vcard labels each detail row
with an itemprop naming what it holds, so one selector plus the label
covers company, location, website, social links, and email instead of
five selectors that each break separately. And the org tab counters are
empty spans carrying title="Not available": those counts are behind a
session, so an org record leaves them nil rather than inventing a zero,
and the live test asserts they stay absent.

The three thread types ship on three different planes and this file says so
out loud. An issue preloads its whole GraphQL node into the page, so labels,
milestone, reactions and the first timeline items all come from one fetch. A
pull request only has route props, and the props for the conversation route
are a delta against the mounted layout, so the page HTML is the only place
the merge metadata lives. A discussion is still Rails, with a schema.org
QAPage block that carries the body and the accepted answer.

A pull request body is 200 characters of open graph text and no more. GitHub
renders the conversation client-side and serves none of it logged out, so
that is the ceiling. Via records where it came from rather than pretending
the text is complete.

Provenance moves out of Extra into its own Via field. Extra means GitHub sent
something nobody modelled and the suite asserts it is empty, so writing our
own bookkeeping in there was making that assertion fail for the wrong reason.

The tree walk pins itself to the commit oid the first directory
resolved to, so a recursive listing of a moving branch cannot end up
half from one commit and half from the next.

Blobs are the awkward one. GitHub's symbol analyser answers null about
half the time and the same full list a second later, on either surface,
with any headers, so a blob read retries across the page cache and then
once more with our own entry dropped before it reports the symbols as
unavailable. That is a real state and the record says so rather than
pretending the language is unsupported.

Six surfaces answer history questions and each is incomplete in a
different way, so the choice is written down per function: route JSON for
the commit list and for one commit, the git upload-pack advertisement for
a complete ref list, the release pages for the labels the feed has no room
for, and the plain-text patch mailbox for a range.

The compare page turned out not to be a JSON route at all. It answers 227
KB of Rails HTML with no react-app payload on it, while the same range
with a .patch suffix is a git-format-patch stream carrying every commit's
author, date, subject and diff in a format GitHub does not own and cannot
restyle. CompareRefs reads that instead.

Two release findings worth recording. Assets are lazy behind an
include-fragment, so the 238 KB release page alone never sees them. And
download counts are gone for a logged-out client: the fragment publishes a
sha256 digest where the count used to be, which is a better trade than it
looks.

page.Find was returning the last match rather than the first. Walk only
stops the subtree it is told to stop, so the result kept being overwritten,
and on a template that renders a dialog repeating the content's own tags
that quietly answered with the dialog. It has a test now, and so does Sel.

This is the layer that was missing between the readers and the binary. gh/domain.go
declares the domain to kit, including the raw and gist hosts so a pasted link
resolves here, and gh/ops.go registers every verb: the single-record reads, the
nine searches, the contents commands, the history commands, and the two meta
commands. cli/ adds the byte plane that does not belong on a record, so cat,
readme, archive, and diff write to stdout instead of into a field.

The old github/ package and pkg/render are gone. Both predate the rewrite and
nothing pointed at them any more.

Four things came out of running it against the live site.

Release authors decoded as whole URLs. GitHub writes the same link two ways: the
release list uses absolute hrefs and the per-tag page relative ones, and the
decoder only trimmed a leading slash. hrefPath now reduces either form to a path
and releaseAuthor is shared by both decoders, which also gets the /apps/ case
right for a release cut by a workflow.

Bare references like cli/cli/blob/trunk/go.mod classified as wiki pages. The URL
parser already knew what they meant, so classifyBare hands them over instead of
guessing.

The symbols error printed a title-cased path. The path moved out of first
position in the sentence, since that is the word the renderer capitalises.

readme printed nothing, because Repo.ReadmeText was never filled. page.Text was
the wrong tool for it: collapsing a twenty-kilobyte README onto one line is not
a rendering of anything. page.BlockText keeps one line per block, one blank line
between paragraphs, and the interior whitespace of a pre block, and the release,
discussion, and profile bodies use it too. The organization profile readme has
no class of its own, so that lookup falls back to the markdown article.

This is the long tail of the read surface: the profile tabs, the
organization roster, gists, the contribution calendar, activity feeds,
trending, topic pages, fork networks, and the contributor graph.

Two findings worth naming. The contributor graph has a data route of its
own, /graphs/contributors-data, which answers 202 with an empty body
while GitHub computes the numbers, so the reader polls rather than
treating that as a failure. And the language histogram does have a
keyless source after all: /{owner}/{repo}/_sidebar is the fragment the
repository page's own front end waits for, it needs no credential, and
it carries the contributor count and the dependent count alongside the
languages. repo.go used to say the histogram had no keyless source. It
does now, and --deep uses it.

The calendar fragment is HTML and answers 406 to a request that asks for
JSON, which the client reports as a response rather than an error, so it
reads on the HTML surface and not the XHR one.

Every reader so far produced records. This turns those records into a graph:
one extractor per type that says what a record points at, five trust levels so a
caller can ask for only the edges that came from an id rather than from prose,
and a walk that follows them.

The crawler is sequential on purpose. Doc 04 gives it a Concurrency knob and it
would be a lie here, because every request already queues through one rate
limiter, so workers would only queue deeper behind the same pacer while making
the output order unpredictable.

deps and dependents are the two dependency graph pages. They are the most
valuable keyless surface on the site and the least reliable one, and they need
two readers rather than one: the rows have different shapes, and so do the two
pagers, ?page=N with a rel="next" anchor on one side and an opaque cursor in a
button on the other.

The empty-page retry in rowPage is not defensive coding, it is the fix for a
real truncation. GitHub answers a cursor page with a 200, the right title, and
no rows often enough to hit on the first repo tried, and the walk cannot tell
that from the end of the list, so it stopped at sixty rows out of two hundred
and then cached the empty page and kept stopping there for fifteen minutes.

rdf and export are byte-plane commands for the same reason cat is: N-Triples is
a serialisation with its own rules, not a record, and pushing it through the
record renderer would produce something that is neither.

page prints the whole extraction for one URL and drops nothing. It is the escape
hatch for a field no record models yet, it is the way to tell "the page did not
carry it" apart from "the decoder dropped it", and it is how a recorded fixture
is read back. Every reader in the package now works from the same Client.Page,
so what this prints is what they see rather than a second opinion.

doctor answers the question people actually ask when a command comes back wrong.
The check that matters is the first one: a token in the environment does nothing
here, and the failure that causes is invisible, because the tool keeps working
and stays exactly as rate limited as it was, so the obvious conclusion is that
the token is wrong. Now it says so.

Brings the url output format and the table column tags the records rely on.

A container package is named after the repository and the thing inside
it, so its name carries a slash, and GitHub wants that slash as %2F.
Locate was writing it plain, which 404s, and the parser was taking the
last path segment as the name, which lost the first half. Both directions
round trip now.

The rest is wording. Every message here leads with a word rather than
with the reference it is about, because the renderer title-cases the
first token and a title-cased URL reads like the tool mangled the input.
An unknown kind now lists the kinds, which is the difference between an
error a reader can act on and one that sends them to the source, and
knownKind reads that same list so a kind cannot be accepted here and
left out of the list.

The renderer capitalises the first token of an error. A message that
started with a path came back as Golang/Go/Blob/Master, which reads like
the tool broke the input rather than like the page was missing, and one
that started with the argument did the same to whatever was typed. So
every message now opens with a plain word and puts the reference after
it.

Two other things fell out of reading them all at once. Articles did not
agree: five of the twenty-four kinds start with a vowel, and "is a repo,
not a org" reads like nobody looked. aKind fixes that in the three places
it happened. And symbols now says the truth, which is that GitHub serves
no symbol list to a signed-out reader on any file tried, rather than
suggesting a retry that will not help.

doctor's help moved to doctor.go so that the one file allowed to name
GITHUB_TOKEN is the file whose job is to say it is ignored.

messages_test.go keeps this from drifting back. It parses every source
file in the module and fails a message that leads with a format verb, a
quote, or a capital, which are the three ways this gets written by
accident.

The one promise this tool makes that a reader cannot check by using it is
that nothing here authenticates. You can see that a command works without
logging in; you cannot see that no path would send a credential if one
happened to be in the environment.

So it is asserted. Every source file is parsed with comments dropped and
checked for Authorization, GITHUB_TOKEN, GH_TOKEN, and api.github.com.
Comments are dropped because this file and the doctor both talk about
tokens at length, and a plain grep would have to be switched off the
first time someone wrote the rule down.

The allow list holds two files and a reason for each, and a second test
fails if an entry stops existing, so the list cannot quietly become a
place to hide things.

GitHub dropped the metadata block from the blob route JSON. The route now
answers with codeViewBlobRoute alone, so asking it for a file's language
gets an empty string and asking it for the size and the line counts gets
nothing at all. The live test caught this, which is what it is for.

The page still carries every block, so the page is now the read. That is
one request where the old path was two, since the route was never enough
on its own and the page was already being fetched for the styled source.
blobRoute is gone.

Symbols were a third request. The list used to arrive on the first read
sometimes and only on the second other times, so an unavailable one was
asked for again before it was believed. It no longer arrives at all: the
page still says symbolsEnabled and still renders the button, and the
block behind it is null on every file of every repository tried, on both
surfaces, with or without a cache, signed out. That retry was spent to
learn nothing, so it is gone too.

A payload with no blocks in it is now a structure-changed error rather
than an empty file, and the routes table says what the blob read actually
does, so `github routes` does not lie about it.

-o url printed nothing for every record. The renderer takes the URL from
the field tagged table:"-,url", and Base had a plain table:"-", so there
was no field to take. Nothing failed; it just printed empty lines.

Actor gets a String method while the file is open. Without it the
renderer falls back to JSON for a struct field, and a cell holding a
whole actor pushes every other column off the screen. A login is what an
author column is for, and the full record is still there in every other
format.

A .diff or .patch suffix on something that is not a change gets answered
with the page rather than a 404. /golang/go/pull/1000 is an issue, so its
.diff is the issue page, 200 and all, and the command was writing a
quarter of a megabyte of markup to the terminal or, worse, to the file
someone redirected it into. The mistake surfaced later as a patch that
would not apply.

The binary is github and the package is ./cmd/github, so the Makefile
builds those. It also grows a live target, because the suite that talks
to github.com is the only one that can tell you the site changed, and it
should not need remembering.

Three naming decisions in the goreleaser config, each written down with
its reason. The image is ghcr.io/tamnd/github-cli, not the short name,
because a short name in a user namespace belongs to whichever repository
pushed it first and the workflow token can only write packages linked to
its own repository, which is how hf-cli's release died at the last step.
The cask is github-cli because homebrew-cask already has a github, and it
is GitHub Desktop. And the cask strips the quarantine attribute on
install, since Gatekeeper kills an ad-hoc signed binary, which is what
every cross-compiled Go binary is.

The old one described a tool that shelled out to something else. This one
opens with what the thing actually is, shows a real session as a gif
recorded from the tape in docs/demo, and then answers the two questions
people arrive with: what can it read, and why is there no token.

The gif is generated, so the tape is in the repository and re-recording
is one command rather than a screen capture nobody can reproduce.

Everything here described ghb, which was the old name and the old design.
Rewritten against a running binary: every command, flag, output format,
and error message on these pages was run before it was written down, and
three defects were found that way, in package URLs, in -o url, and in the
articles in the reference errors.

Six guides. Contents, the graph plane, linked data, the page plane,
resource URIs, and how to add a command. The URI guide's table has one
row per addressable kind with an example that round trips, because a URI
scheme nobody can check is a URI scheme nobody trusts.

Four reference pages. Commands grouped the way the help groups them,
configuration with the real defaults rather than plausible ones, output
with the template caveat that keys are the JSON keys, and troubleshooting
written from the errors the tool actually produces, exit codes included.

--profile is deliberately absent from configuration. It is accepted and
consumed by nothing, and documenting a knob that does nothing is worse
than leaving it out.