Hello,
I have on my site wordfence and securi free both report the site is clean. I am also using the plugin wp robots txt do do my robot.txt file. When I look at it, I have the following in it.
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/plugins/
Disallow: /wp-includes/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /go/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /How ever when I look at google web tools it has this listed from the last down load 8 hours ago.
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-content/plugins/
Disallow: /wp-includes/
Disallow: /comments/feed/
Disallow: /feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /go/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /</title><style>.xhp0{position:absolute;clip:rect(452px,auto,auto,466px);}</style><div class=xhp0><a href="http://txmapaydayloans.com/" />payday loans LENDERS ONLINE</div>It has this listed under site health and blocked url
Blocked URLs
If your site has content you don't want Google or other search engines to access, use a robots.txt file to specify how search engines should crawl your site's content.
Check to see that your robots.txt is working as expected. (Any changes you make to the robots.txt content below will not be saved.)
robots.txt file Blocked URLs Downloaded Status
http://clanbert.com/robots.txt 1 8 hours ago 200 (Success)
How can I resolve this? Any help would be appreciated.