Crawling rate depends on PageRank

You've done all your onsite SEO - page titles, urls, meta tags, content - all optimized. With excitement you submit your sitemap to search engines containing 4,000 urls, yet only 100 of them end up indexed. Why is that you ask? Google does not always crawl or index a site fully, and onsite search engine optimization is irrelevant on a page that isn't indexed, making it only part of the SEO equation.

Google desires to have pages that are "relevant and reputable." Matt Cutts recently gave a talk at WordCamp San Francisco in which he says that Google crawls in decreasing order of Page Rank, whereas Page Rank refers to the number and importance of links pointing to your site. "Importance" is the operative word in this sentence. Not all links to your are created equal, and having less links from relevant and reputable sites will go further than having more links from spammy sites.

What does this mean for you? Ensure that you update your site frequently to remain relevant, provide reputable content to establish yourself as an expert at the forefront of your industry, and seek links to such content from other reputable and relevant sites. The associated pagerank increase will help ensure your pages are crawled deeper and more often.