Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	Jeff Kaplan	Date:	July 31, 2012 02:18:42pm
Forum:	web	Subject:	Re: Archive appears sometimes, other days it won't

there is a robots.txt in place
http://www.tarotjunkie.com/robots.txt

Reply to this post

Reply [edit]

Poster:	amnion	Date:	August 09, 2012 03:54:12pm
Forum:	web	Subject:	Re: Archive appears sometimes, other days it won't

Wouldn't that only apply to the current versions of that site? What about the versions of the site where there were no robots.txt in place? The website has already been crawled, when it was okay to crawl it and archive it. The robots.txt just says "do not crawl this website". That would mean it can't be archived right now, at this moment. It doesn't seem fair that a new domain owner can put out a decree stating that this website, and all versions of this website from the past (which I did not own) are mine to control and must not be archived.

I am only trying to find the versions that have already been archived, which I know exist, because I've seen them.

Reply to this post

Reply [edit]

Poster:	Jeff Kaplan	Date:	August 09, 2012 04:15:45pm
Forum:	web	Subject:	Re: Archive appears sometimes, other days it won't

robots.txt blocks all archived captures for that url.

Reply to this post

Reply [edit]

Poster:	amnion	Date:	August 09, 2012 05:30:11pm
Forum:	web	Subject:	Re: Archive appears sometimes, other days it won't

How do we change that? Who do I need to talk to?