I just noticed that the Googlebot visited Gnorb.NET. It doesn’t quite have the “Santa’s coming to town” feel, but it does help me understand how the spiders crawl my site. Using the Googlebot tracking WordPress plugin, I’m able to see that the bots were here yesterday. By the looks of it, the bot’s in love with my front page, having visited it once every 5 1/2 hours or so yesterday.
I guess it likes all that content.
The trick is to get it to spider the site, at least more than the front page. To that end, I’ll probably start using the miniblog (look at the bottom of the sidebar) to create some daily moving content on each of the pages. Maybe on a category-by-category basis, or maybe just an overall thing. The second thing I’ll see about doing is setting up the categories as sub-domains (books.gnorb.net, business.gnorb.net, games-and-entertainment.gnorb.net, etc). That way, the site will be treated as multiple sites, instead of just one. (Better to have one egg in 100 baskets than 100 eggs in one basked, to paraphrase J. P. Getty.) For that, I’ll need my sys-admin Nate’s help. Finally, I’ll see about doing multi-page posts, though I’m not sure I like that too much; personally, I think I might get annoyed with that. Heck, I probably would if I saw it on another site, especially if there are like 15 ads and not very many words per page. (Just about every two-bit computer review site I’ve ever seen does this. Good one’s, like [H]ardOCP or Overclockers.com, don’t.)
At any rate, I’ll study these logs more and whenever I find anything out I’ll put it here.
Edit:Man, this thing is still going! Wow… I guess it IS going deep. The question is how often will it spider?