A solo developer has quietly assembled the largest public collection of UK planning decisions by scraping 241 council portals across England, Scotland, and Wales. The dataset now contains more than 2.6 million decisions, exposing the chaotic state of councils’ online systems and revealing sharp disparities in how planning applications are handled from one ward to the next.
A patchwork of 400+ public portals with zero consistency
UK planning data is technically open, but accessing it in practice means navigating more than 400 council-run portals. Many still rely on legacy systems, with some using bespoke ASP.NET interfaces that resemble early-2000s software. Others operate behind AWS Web Application Firewalls (WAF) or enforce aggressive rate limits that can block non-browser requests within minutes.
To crack the problem, the developer built three distinct scraping tools. A standard requests-based scraper worked for straightforward sites, while a Playwright-based scraper was needed for councils that reject anything but genuine browser traffic. A third scraper, using curl_cffi, mimicked real browser TLS fingerprints to bypass fingerprinting defenses. Even then, some portals proved impenetrable. Liverpool’s planning system, shielded by AWS WAF and a JavaScript challenge, required a custom Playwright scraper that reused cookies but still hit a hard IP limit after about ten requests, effectively blocking further access for a full day.
Approval rates hide stark local variations
The national planning approval rate hovers around 88%, a figure often cited in public debates. Yet digging into the data reveals dramatic differences within individual councils. Approval odds can swing wildly from one ward to another, meaning residents in neighboring areas may face entirely different outcomes based solely on where their application lands.
Delayed decisions are getting worse, not better
The most striking finding emerged from the time-to-decision analysis. Between 2019 and 2025, the share of home extension applications exceeding England and Wales’ statutory eight-week deadline rose from 27.9% to 36.5% across 119 councils. In Guildford, the problem is acute: 66% of decisions overran the target, with applications taking an average of 13.3 weeks to process.
From raw data to public tools
The effort has evolved into a functional postcode lookup tool available for free, alongside premium PDF reports priced at £19 and £79. Despite the public interest in planning data, the service has not yet attracted paying users, a fact the developer describes as acceptable given the focus on data quality and coverage expansions.
The project’s public interface is hosted at planninglens.co.uk, where users can explore the dataset firsthand. For those curious about the technical challenges behind the scraper, the developer has offered an open invitation to ask questions, emphasizing the project’s most interesting problems lie in the scraping phase rather than the end product.
As planning systems evolve and councils gradually modernize their portals, the dataset remains a vital resource for researchers, journalists, and residents alike. It stands as a testament to what can be achieved when public data is wrested from fragmented, outdated infrastructure—one scraper run at a time.
AI summary
İngiltere, İskoçya ve Galler'deki 241 belediyenin planlama portalından 2,6 milyon karar toplandı. Farklı sistemler, gizli engeller ve değişken düzenlemeler nedeniyle projede neler yaşandı? Veriler ne gösteriyor?
