Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	Jeff Kaplan	Date:	December 28, 2010 11:00:11am
Forum:	web	Subject:	Re: Nothing in index after 2007; ia_archiver hits robots.txt only, but we don't exclude ia_archiver bot.

While the Wayback Machine continues to crawl, nothing has been indexed that is more recent that mid-2008. We hope to have an updating of the index this year.

This post was modified by Jeff Kaplan on 2010-12-28 18:00:11

Reply to this post

Reply [edit]

Poster:	jackaddison	Date:	December 29, 2010 05:34:23pm
Forum:	web	Subject:	Re: Nothing in index after 2007; ia_archiver hits robots.txt only, but we don't exclude ia_archiver bot.

This is encouraging to hear. We thought it was totally dead.
[URL]

This post was modified by gojomo on 2010-12-30 00:34:23

Reply to this post

Reply [edit]

Poster:	SiPlus	Date:	April 11, 2011 01:15:37am
Forum:	web	Subject:	Re: Nothing in index after 2007; ia_archiver hits robots.txt only, but we don't exclude ia_archiver bot.

Did you exclude "*"?