Why Crawl Budget Matters for Larger Sites
Too Long; Didn't Read
Optimising for Crawl Budget is an often-overlooked [technical SEO] hack that can have a big impact on larger websites. Google divides its attention among millions of websites every day. It assigns a ‘budget’ to your site that’s size, ‘health’ and backlink portfolio. It also takes into account the following two factors: Crawl limit or host load - how much crawling can a website handle and what are its owner's preferences? Crawl demand - which URLs are worth (re-crawling) based on its popularity and how often it's being updated.