Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	Jeff Kaplan	Date:	December 08, 2010 07:56:22pm
Forum:	forums	Subject:	Re: Wayback Machine has not got my site since 2008

If your site is easily reachable chances are it is being crawled. Currently web pages are only indexed through 2008. After the next major reindexing it might then be on teh wayback machine.

Reply to this post

Reply [edit]

Poster:	Exodus Development, Inc.	Date:	December 26, 2010 06:01:30pm
Forum:	forums	Subject:	Re: Wayback Machine has not got my site since 2008

I have the same problem (nothing in the archive after 2007!) for http://exodusdev.com though the archive.org bot is hitting robots.txt (and only robots.txt) almost every day.


207.241.227.75 - - [18/Dec/2010:17:58:43 -0800] "GET /robots.txt HTTP/1.1" 200 1591 "-" "ia_archiver-web.archive.org"

If the archive.org bot isn't hitting anything but robots.txt, should I worry?

My robots.txt file doesn't have anything that excludes the ia_archiver bot.