Reply to this post | See parent post | Go Back

View Post [edit]

Poster:	Jeff Kaplan	Date:	March 14, 2012 10:44:09pm
Forum:	web	Subject:	Re: Page cannot be crawled due to robots.txt

the current owner of the domain has a robot.txt file disallowing crawling. the pages cannot be displayed unless it is removed.

http://www.usmlein.com/robots.txt

Reply to this post

Reply [edit]

Poster:	sids	Date:	March 15, 2012 08:49:42am
Forum:	web	Subject:	Re: Page cannot be crawled due to robots.txt

dont understand it was visible before last week why changed now really needed to see its contents tbh

anyway to get round this