Command-line tool
Finds where a dead link went
Links in a collection stop working over time. Sites get reorganised, move to a new CMS or drop a domain. The page usually still exists somewhere, but nothing points to its new address. This tool narrows each dead link down to a short, ranked list that a person can check in a few seconds.
Pipeline
Reads the collection and keeps the entries that are dead, meaning working is false or the status code isn't 200. It sends a HEAD request to each domain's root to see whether the site itself still answers, and caches that, so fifty dead pages on one host cost one request.
writesdead_links.json
Searches for each dead link and ranks what comes back. It saves after every entry, so a stopped run picks up where it left off and only retries the entries that hit a search or model error.
writescandidates.json
A review loop in the terminal. It shows the original link, the ranked candidates with their scores and reasons, and the raw search results below. You pick a number, paste your own URL, skip or quit, and progress saves after every decision.
writesfinal_replacements.json
Stage 2
Whether the domain still answers decides the whole search. A dead page on a live site has usually just moved. A dead page on a dead domain needs a different approach.
Its URL, its stored title and the domain_alive flag from stage 1.
It searches site:domain title, which keeps results inside the organisation that published the original. If that finds nothing, it drops the site restriction and tries again.
It asks the Wayback Machine for the closest snapshot, reads the <title> from the archived page, and searches on that plus keywords taken from the dead URL's path.
The model returns its best three, each with a score from 1 to 10 and a one-line reason, plus a no_match flag when nothing looks convincing.
The candidates go to the review loop, where someone makes the final call.
Design choice
A model can tell you a page is on the same topic as a dead one. It can't tell you they're the same document, because the original is gone and there's nothing left to compare against. Two yearly reports from the same organisation will score well against each other while holding completely different files.
So the tool shortlists and a person decides. Scoring works better than filtering here too: a weak candidate with a stated reason is more useful to a reviewer than one that was silently dropped. The score is a suggested reading order, not a verdict.
Usage
sample/links.json holds ten fictional entries in the expected shape, so the pipeline runs on a fresh clone. Point stage one at your own collection to do real work.
$ python -m venv .venv $ source .venv/bin/activate $ pip install -r requirements.txt $ cp .env.example .env # add a search API key and an LLM API key $ python extract_dead_links.py sample/links.json -o dead_links.json $ python find_candidates.py -i dead_links.json -o candidates.json $ python review_candidates.py -i candidates.json -o final_replacements.json
Two flags are worth knowing: --timeout on stage one for slow hosts, and --concurrency on stage two, which defaults to 3 and is the one to turn down when search starts returning errors.
| Field | Type | Used for |
|---|---|---|
item_id | int | Identity across all three stages, and how resuming works |
link | string | The URL to check |
title | string | Search text, and what you read during review |
status_code | int | Liveness. Anything other than 200 counts as dead |
working | bool | Optional. False marks an entry dead whatever the status code says |
Limits
Three scripts, a sample collection and a README that walks through each stage.