Some of these robots are very persistent (or just plain dumb). They try to access pages that haven't been around for a long time getting 404's or redirects elsewhere. I'm presuming that after some number N occurences of non-success responses, that these robots will get a clue and just crawl the links that are there... for some reason, new links that've shown up aren't crawled. Here's a list of crawler URL's:
Between the robots and viruses, I probably have as many software entities hitting my site as I do human readers.
( Mar 24 2004, 11:01:06 PM PST )