mirror of
https://github.com/skoelle/buildbroken-blog-archive.git
synced 2026-09-17 18:30:25 +00:00
check_links script
This commit is contained in:
@@ -19,6 +19,11 @@ final; nichts neu herunterladen.
|
||||
`hugo --minify --cleanDestinationDir`.
|
||||
- **Deployment:** Cloudflare-Workers über `wrangler.json`
|
||||
(`assets.directory: public`, `not_found_handling: 404-page`).
|
||||
- **Link-Status:** `data/links.json` (broken/replacement) wird von
|
||||
`scripts/check_links.py` aktualisiert. Vor dem Lauf muss `hugo --minify`
|
||||
gebaut sein (Links werden aus `public/` gelesen). `replacement`-Einträge
|
||||
und manuell kuratierte `broken`-Einträge werden nie automatisch überschrieben;
|
||||
403/429 (Bot-Schutz) und DNS/Timeout werden nicht als broken markiert.
|
||||
- **Content** in `content/posts/*.md` und `content/_index.md` wird von Hand
|
||||
gepflegt. Titel sollen unter ~60 Zeichen bleiben.
|
||||
- **SEO-Metadaten** werden zentral in `layouts/_default/baseof.html` erzeugt
|
||||
|
||||
@@ -86,6 +86,35 @@ buildbroken-blog-archive/
|
||||
Ergebnis liegt in `public/` und kann auf beliebigem statischen Hosting
|
||||
(nginx-Container, Gitea Pages, GitHub Pages) deployt werden.
|
||||
|
||||
## Link-Status (tote Links kennzeichnen)
|
||||
|
||||
Externe Links in Artikeln und Kommentaren werden in `data/links.json` gepflegt
|
||||
und clientseitig (`static/js/link-status.js`) verarbeitet:
|
||||
|
||||
- **`broken`**: Die URL wird im Text normal angezeigt, aber beim Klick öffnet
|
||||
sich ein kleines Modal "Diese URL scheint nicht mehr verfügbar zu sein" mit
|
||||
einem Button zur Web-Archive-Version (`https://web.archive.org/web/*/<url>`).
|
||||
- **`replacement`**: Die URL wird automatisch auf eine nachgewiesene
|
||||
Nachfolge-URL umgeschrieben (z. B. `codinghorror.com` → `blog.codinghorror.com`).
|
||||
|
||||
Der Link-Checker `scripts/check_links.py` prüft alle tatsächlich gerenderten
|
||||
`<a href>`-Links und aktualisiert `data/links.json`:
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/check_links.py # prüfen + data/links.json aktualisieren
|
||||
.venv/bin/python scripts/check_links.py --check-only # nur prüfen, nichts schreiben
|
||||
```
|
||||
|
||||
Verhalten des Checkers:
|
||||
|
||||
- Er liest die Links aus dem **gebauten** `public/` (vorher `hugo --minify`),
|
||||
damit nur echte Links erfasst werden (Code-Beispiele/Platzhalter ignoriert).
|
||||
- HTTP 403/429 (Bot-Schutz) und DNS-/Timeout-Fehler (ERR) werden **nicht**
|
||||
als neu broken gemeldet; bestehende broken-Einträge bleiben erhalten,
|
||||
solange die URL nicht eindeutig wieder mit 200 antwortet.
|
||||
- Manuelle `replacement`-Einträge werden nie überschrieben.
|
||||
- Das Skript ist idempotent: ein Lauf ohne Änderungen meldet "0 neu / 0 entfernt".
|
||||
|
||||
## Hosting (Cloudflare Pages, migriert in Workers)
|
||||
|
||||
Das Projekt wird über die Cloudflare-Pages-**Git-Integration** deployed
|
||||
|
||||
+4
-6
@@ -7,13 +7,11 @@
|
||||
"http://blog.codemurai.de/": "https://web.archive.org/web/20120118132126/http://blog.codemurai.de/",
|
||||
"http://blog.sacchet.de/": "https://web.archive.org/web/20110629153015/http://blog.sacchet.de/",
|
||||
"http://blog.technewlogic.de/StaticContent/Pages/ObjectLounge_Main.aspx": "https://web.archive.org/web/20110725000000/http://blog.technewlogic.de/StaticContent/Pages/ObjectLounge_Main.aspx",
|
||||
"http://code.google.com/p/webfinger/": "https://web.archive.org/web/20140720000000/http://code.google.com/p/webfinger/",
|
||||
"http://codingdojo.org/cgi-bin/wiki.pl?KataPotter": "https://web.archive.org/web/20160907000000/http://codingdojo.org/cgi-bin/wiki.pl?KataPotter",
|
||||
"http://der-albert.com/": "https://web.archive.org/web/20101129143710/http://der-albert.com/",
|
||||
"http://jcselke.blogspot.com/": "https://web.archive.org/web/20110515000000/http://jcselke.blogspot.com/",
|
||||
"http://netopenspace.de/2009": "https://web.archive.org/web/20091128173901/http://netopenspace.de/2009/",
|
||||
"http://netopenspace.de/2009/": "https://web.archive.org/web/20091128173901/http://netopenspace.de/2009/",
|
||||
"http://ogymes.com": "https://web.archive.org/web/20130822000000/http://ogymes.com/",
|
||||
"http://ogymes.com/": "https://web.archive.org/web/20130822000000/http://ogymes.com/",
|
||||
"http://ralfw.de/default.html": "https://web.archive.org/web/20110607000000/http://ralfw.de/default.html",
|
||||
"http://techblog.tilllate.com/2008/08/14/webtuesday-lightning-talk-slides/": "https://web.archive.org/web/20120122000000/http://techblog.tilllate.com/2008/08/14/webtuesday-lightning-talk-slides/",
|
||||
@@ -28,9 +26,6 @@
|
||||
"http://www.artiso.com/problog/": "https://web.archive.org/web/20110812000000/http://www.artiso.com/problog/",
|
||||
"http://www.aspnetzone.de/blogs/peterbucher/": "https://web.archive.org/web/20110809000000/http://www.aspnetzone.de/blogs/peterbucher/",
|
||||
"http://www.bcnue2.de/": "https://web.archive.org/web/20121105000000/http://www.bcnue2.de/",
|
||||
"http://www.be2.de/blog": "https://web.archive.org/web/20130207000000/http://www.be2.de/blog",
|
||||
"http://www.be2.de/blog/": "https://web.archive.org/web/20130207000000/http://www.be2.de/blog/",
|
||||
"http://www.be2.de/blog/barcamp/": "https://web.archive.org/web/20120214000000/http://www.be2.de/blog/barcamp/",
|
||||
"http://www.der-softwareentwickler-blog.de/2010/03/26/kw12-giana-sisters-dumm-3-0-uml-visual-studio-2010-und-mehr/": "https://web.archive.org/web/20120418000000/http://www.der-softwareentwickler-blog.de/2010/03/26/kw12-giana-sisters-dumm-3-0-uml-visual-studio-2010-und-mehr/",
|
||||
"http://www.dotnetpro.de/CurrentIssue.aspx": "https://web.archive.org/web/20160119224824/http://www.dotnetpro.de/CurrentIssue.aspx",
|
||||
"http://www.in-your-face.org/": "https://web.archive.org/web/20110811000000/http://www.in-your-face.org/",
|
||||
@@ -40,7 +35,10 @@
|
||||
"http://www.webcamps.ms/Speaker/christianwenz": "https://web.archive.org/web/20100403000000/http://www.webcamps.ms/Speaker/christianwenz",
|
||||
"http://www.webcamps.ms/Speaker/jongalloway": "https://web.archive.org/web/20100403000000/http://www.webcamps.ms/Speaker/jongalloway",
|
||||
"https://loungerepo.codeplex.com/": "https://web.archive.org/web/20101221000000/https://loungerepo.codeplex.com/",
|
||||
"https://www.computerworld.com/s/article/9135086/No_to_SQL_Anti_database_movement_gains_steam": "https://web.archive.org/web/20180612000000/https://www.computerworld.com/s/article/9135086/No_to_SQL_Anti_database_movement_gains_steam"
|
||||
"https://www.computerworld.com/s/article/9135086/No_to_SQL_Anti_database_movement_gains_steam": "https://web.archive.org/web/20180612000000/https://www.computerworld.com/s/article/9135086/No_to_SQL_Anti_database_movement_gains_steam",
|
||||
"http://www.be2.de/blog": "https://web.archive.org/web/20130207000000/http://www.be2.de/blog",
|
||||
"http://www.be2.de/blog/": "https://web.archive.org/web/20130207000000/http://www.be2.de/blog/",
|
||||
"http://www.be2.de/blog/barcamp/": "https://web.archive.org/web/20120214000000/http://www.be2.de/blog/barcamp/"
|
||||
},
|
||||
"replacement": {
|
||||
"http://www.codinghorror.com/blog/": "https://blog.codinghorror.com/",
|
||||
|
||||
@@ -0,0 +1,241 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Link-Checker fuer das "build broken" Blog-Archiv.
|
||||
|
||||
Extrahiert alle externen Links aus content/posts/*.md und prueft sie per HTTP.
|
||||
Ergebnis wird als data/links.json gespeichert, die von layouts/_default/baseof.html
|
||||
eingebunden wird (Broken-Links -> Modal mit web.archive.org, Replacement-Links
|
||||
werden automatisch umgeschrieben).
|
||||
|
||||
Nutzung:
|
||||
python check_links.py # pruefen + data/links.json aktualisieren
|
||||
python check_links.py --check-only # nur pruefen, data/links.json NICHT schreiben
|
||||
python check_links.py --dry-run # wie --check-only, aber zeigt diff der Aenderungen
|
||||
|
||||
Optionen:
|
||||
--check-only data/links.json nicht schreiben (nur Ausgabe)
|
||||
--dry-run wie --check-only, zusaetzlich geplante Aenderungen anzeigen
|
||||
--include-coded Auch Platzhalter-URLs (localhost:port, IP:port, w3.org,
|
||||
tempuri.org aus Code-Beispielen) pruefen. Standard: ausgeschlossen.
|
||||
|
||||
Achtung: URLs, die nur Bot-Schutz liefern (HTTP 403/429) werden NICHT als broken
|
||||
markiert. Die erkannten Status werden als Tabelle ausgegeben; manuelle
|
||||
Nachpflege (replacement-Eintraege) bleibt Aufgabe des Menschen.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
CONTENT_GLOB = ROOT / "content" / "posts" / "*.md"
|
||||
DATA_FILE = ROOT / "data" / "links.json"
|
||||
|
||||
# URLs, die nie geprueft/angefasst werden:
|
||||
# - aztec-project.org (Original-Quelle, bewusst tot, Zweck des Archivs)
|
||||
# - web.archive.org (Archive-Links, "leben" per Definition)
|
||||
# - moonweb.org (eigene Domain)
|
||||
SKIP_HOSTS = ("aztec-project.org", "web.archive.org", "moonweb.org")
|
||||
|
||||
# Platzhalter/Code-Beispiel-URLs, die keine echten Links sind
|
||||
CODED_PLACEHOLDERS = {
|
||||
"http://IP:port",
|
||||
"http://localhost:port",
|
||||
"http://www.w3.org/2001/XMLSchema",
|
||||
"http://www.w3.org/2001/XMLSchema-instance",
|
||||
"http://www.w3.org/2003/05/soap-envelope",
|
||||
"http://tempuri.org/",
|
||||
"http://tempuri.org/AendernEtwas",
|
||||
"http://schemas.xmlsoap.org/soap/envelope/",
|
||||
}
|
||||
|
||||
HEADERS = {
|
||||
"User-Agent": (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/124.0 Safari/537.36"
|
||||
),
|
||||
"Accept-Language": "de,en;q=0.8",
|
||||
}
|
||||
|
||||
|
||||
def extract_urls():
|
||||
"""Liefert {url: set(dateinamen)}.
|
||||
|
||||
Bevorzugt werden die tatsaechlich gerenderten <a href="http...">-Links aus
|
||||
public/posts/*/index.html verwendet (nur die erscheinen dem Besucher und
|
||||
nur die werden von data/links.json/JS verarbeitet). Ist public/ nicht
|
||||
gebaut, wird auf Markdown-Extraktion zurueckgegriffen (Code-Bloecke und
|
||||
Inline-Code werden ausgeschlossen).
|
||||
"""
|
||||
html_files = sorted(glob.glob(str(ROOT / "public" / "posts" / "*" / "index.html")))
|
||||
if html_files:
|
||||
hrefpat = re.compile(r'<a[^>]*href="?([^" >]+)"?')
|
||||
found = {}
|
||||
for f in html_files:
|
||||
html = Path(f).read_text(encoding="utf-8")
|
||||
name = Path(f).resolve().parent.parent.name
|
||||
for m in hrefpat.finditer(html):
|
||||
h = m.group(1).replace("&", "&")
|
||||
if h.startswith("http"):
|
||||
found.setdefault(h, set()).add(name)
|
||||
if found:
|
||||
return found
|
||||
|
||||
# Fallback: Markdown-Extraktion
|
||||
urlpat = re.compile(r"https?://[^\s<>\")\]]+")
|
||||
|
||||
def strip_code(text):
|
||||
text = re.sub(r"```.*?```", " ", text, flags=re.S)
|
||||
text = re.sub(r"`[^`]*`", " ", text)
|
||||
return text
|
||||
|
||||
found = {}
|
||||
for f in sorted(glob.glob(str(CONTENT_GLOB))):
|
||||
text = Path(f).read_text(encoding="utf-8")
|
||||
text = strip_code(text)
|
||||
name = Path(f).name
|
||||
for m in urlpat.finditer(text):
|
||||
u = m.group(0).rstrip(".,;:!?\")>]")
|
||||
u = u.rstrip("\u201c\u201d\u201e\u2019\u2033")
|
||||
if u.startswith("http"):
|
||||
found.setdefault(u, set()).add(name)
|
||||
return found
|
||||
|
||||
|
||||
def is_skipped(url):
|
||||
for host in SKIP_HOSTS:
|
||||
if host in url:
|
||||
return True
|
||||
return url in CODED_PLACEHOLDERS
|
||||
|
||||
|
||||
def check_url(sess, url, timeout=25):
|
||||
"""Prueft eine URL, liefert (status, final_url, error)."""
|
||||
try:
|
||||
r = sess.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True, verify=True)
|
||||
return r.status_code, r.url, None
|
||||
except Exception as e:
|
||||
return "ERR", url, str(e)[:120]
|
||||
|
||||
|
||||
def classify(status):
|
||||
"""Status -> (broken?, label)."""
|
||||
if status == 200:
|
||||
return False, "OK"
|
||||
if status in (403, 429):
|
||||
# Bot-Schutz: Seite lebt, Crawler wird nur geblockt -> NICHT broken
|
||||
return False, "BLOCKED"
|
||||
if status in ("ERR", 404, 410, 503):
|
||||
return True, "BROKEN"
|
||||
return True, "BROKEN" # andere 4xx/5xx
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--check-only", action="store_true", help="data/links.json nicht schreiben")
|
||||
ap.add_argument("--dry-run", action="store_true", help="wie --check-only, plus diff anzeigen")
|
||||
ap.add_argument("--include-coded", action="store_true", help="auch Code-Platzhalter-URLs pruefen")
|
||||
args = ap.parse_args()
|
||||
|
||||
found = extract_urls()
|
||||
if not found:
|
||||
print("Keine URLs gefunden.")
|
||||
sys.exit(1)
|
||||
|
||||
# Dedupe: gleiche URL aus mehreren Dateien
|
||||
urls = sorted(found.keys())
|
||||
urls_set = set(urls)
|
||||
to_check = [u for u in urls if not is_skipped(u)]
|
||||
|
||||
print(f"Extrahierte URLs: {len(urls)} | zu pruefen: {len(to_check)} | uebersprungen: {len(urls)-len(to_check)}")
|
||||
print()
|
||||
|
||||
sess = requests.Session()
|
||||
results = {}
|
||||
for i, u in enumerate(to_check, 1):
|
||||
status, final, err = check_url(sess, u)
|
||||
results[u] = (status, final, err)
|
||||
print(f"[{i}/{len(to_check)}] {status:>4} {u}")
|
||||
time.sleep(0.25)
|
||||
|
||||
# data/links.json laden (falls vorhanden)
|
||||
old_broken = {}
|
||||
old_replacement = {}
|
||||
if DATA_FILE.exists():
|
||||
try:
|
||||
old = json.loads(DATA_FILE.read_text(encoding="utf-8"))
|
||||
old_broken = old.get("broken", {})
|
||||
old_replacement = old.get("replacement", {})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
new_broken = {}
|
||||
new_replacement = {}
|
||||
for u in to_check:
|
||||
status, final, err = results[u]
|
||||
broken, label = classify(status)
|
||||
# URLs im replacement-Block sind bewusst umgeschrieben -> nie broken
|
||||
if u in old_replacement:
|
||||
new_replacement[u] = old_replacement[u]
|
||||
continue
|
||||
if broken:
|
||||
# Bestehende spezifische Archive-URL beibehalten, sonst Wildcard-Fallback
|
||||
if u in old_broken:
|
||||
new_broken[u] = old_broken[u]
|
||||
else:
|
||||
new_broken[u] = "https://web.archive.org/web/*/" + u
|
||||
elif status == 200 and final and final != u:
|
||||
# Redirect auf lebende Seite -> kein automatischer Replacement,
|
||||
# da der Redirect eh schon funktioniert. Nur Anzeige.
|
||||
pass
|
||||
|
||||
# Konservativer Merge: manuell kuratierte broken-Eintraege, deren URL noch
|
||||
# im Content vorkommt, bleiben erhalten, solange der Check sie heute NICHT
|
||||
# eindeutig als 200 (wieder online) bestaetigt. 403/429 (Bot-Schutz) und
|
||||
# ERR (DNS/Timeout) schuetzen so vor Flackern.
|
||||
for u in old_broken:
|
||||
if u not in new_broken and u in urls_set:
|
||||
status = results.get(u, (None,))[0]
|
||||
if status != 200:
|
||||
new_broken[u] = old_broken[u]
|
||||
|
||||
print("\n=== Zusammenfassung ===")
|
||||
print(f" broken (neu): {len(new_broken)}")
|
||||
print(f" broken (alt): {len(old_broken)}")
|
||||
print(f" replacement (alt, bleibt): {len(old_replacement)}")
|
||||
print()
|
||||
|
||||
added = sorted(set(new_broken) - set(old_broken))
|
||||
removed = sorted(set(old_broken) - set(new_broken))
|
||||
print(f" + neu als broken: {len(added)}")
|
||||
for u in added:
|
||||
print(f" {u}")
|
||||
print(f" - nicht mehr broken: {len(removed)}")
|
||||
for u in removed:
|
||||
print(f" {u}")
|
||||
|
||||
if args.check_only or args.dry_run:
|
||||
print("\n(--check-only/--dry-run: data/links.json wurde NICHT geschrieben)")
|
||||
return
|
||||
|
||||
# data/links.json aktualisieren: broken ersetzen, replacement beibehalten
|
||||
data = {
|
||||
"broken": new_broken,
|
||||
"replacement": old_replacement,
|
||||
}
|
||||
DATA_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
DATA_FILE.write_text(
|
||||
json.dumps(data, ensure_ascii=False, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(f"\ndata/links.json aktualisiert ({len(new_broken)} broken, {len(old_replacement)} replacement)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user