Benign-candidate URL collector
Collects full URLs from sitemaps of a random sample of Tranco domains (all popularity levels), new Hacker News links, and real login pages found via the homepage "sign in" link of random global and Arab (.qa, .sa, .ae …) sites, every 6 hours. It reads robots.txt, sitemaps and, for login discovery, one homepage plus at most two linked pages per site; it never fills in or submits anything. Every URL is stored as unlabeled until it is re-checked against the phishing lists. URLs are never shown here. Updated 2026-10-13T19:47:57+00:00 UTC.
Unique URLs1,779
By sourcehn: 572 · login: 11 · manual: 158 · sitemap: 1,038
Domains tried / with URLs180 / 55
Login pages found / sites checked11 / 160100001-1000000: 1 · 10001-100000: 2 · 1001-10000: 1 · arab: 7
Sitemap URLs by popularity bucket (Tranco rank)
| Rank bucket | Domains | URLs |
|---|---|---|
| 1-1000 | 9 | 180 |
| 1001-10000 | 20 | 355 |
| 10001-100000 | 15 | 300 |
| 100001-1000000 | 11 | 203 |
Latest runs
| Run (UTC) | Source | OK | Listed | New | Error |
|---|---|---|---|---|---|
| 2026-10-13T19:46:32+00:00 | login | ✅ | 80 | 7 | |
| 2026-10-13T19:46:32+00:00 | sitemap | ✅ | 247 | 247 | |
| 2026-10-13T19:46:32+00:00 | hn | ✅ | 10 | 10 | |
| 2026-10-13T19:35:18+00:00 | login | ✅ | 80 | 4 | |
| 2026-10-13T19:35:18+00:00 | sitemap | ✅ | 417 | 417 | |
| 2026-10-13T19:35:18+00:00 | hn | ✅ | 51 | 51 | |
| 2026-10-13T18:51:15+00:00 | sitemap | ✅ | 374 | 374 | |
| 2026-10-13T18:51:15+00:00 | hn | ✅ | 511 | 511 |