.Google.com's John Mueller addressed a concern concerning why Google.com indexes pages that are actually disallowed from crawling through robots.txt as well as why the it is actually risk-free to ignore the similar Explore Console documents concerning those creeps.Robot Web Traffic To Concern Specification URLs.The individual talking to the question recorded that robots were creating links to non-existent inquiry specification URLs (? q= xyz) to web pages along with noindex meta tags that are also blocked out in robots.txt. What prompted the inquiry is actually that Google is actually creeping the hyperlinks to those webpages, getting shut out through robots.txt (without noticing a noindex robots meta tag) then receiving reported in Google Search Console as "Indexed, though shut out by robots.txt.".The individual asked the following inquiry:." Yet here is actually the huge inquiry: why would certainly Google.com index web pages when they can not also view the web content? What's the perk because?".Google.com's John Mueller confirmed that if they can not creep the webpage they can not find the noindex meta tag. He also helps make an exciting acknowledgment of the site: search driver, encouraging to neglect the outcomes since the "typical" users will not view those end results.He composed:." Yes, you're appropriate: if we can not crawl the web page, our experts can't view the noindex. That pointed out, if we can not crawl the pages, at that point there is actually not a lot for us to mark. So while you may observe a few of those webpages along with a targeted internet site:- concern, the typical user won't see all of them, so I wouldn't fuss over it. Noindex is likewise great (without robots.txt disallow), it merely implies the URLs will definitely wind up being crawled (and also end up in the Explore Console record for crawled/not recorded-- neither of these standings induce concerns to the rest of the site). The fundamental part is that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the restrictions in operation the Site: hunt progressed hunt driver for diagnostic causes. One of those factors is because it's certainly not linked to the normal hunt index, it's a separate factor altogether.Google.com's John Mueller talked about the internet site hunt operator in 2021:." The brief solution is actually that a site: question is certainly not meant to become total, neither used for diagnostics reasons.A web site query is a details kind of hunt that restricts the outcomes to a particular site. It is actually primarily merely words internet site, a digestive tract, and then the web site's domain name.This concern limits the results to a details site. It is actually certainly not implied to be an extensive compilation of all the webpages coming from that web site.".2. Noindex tag without using a robots.txt is actually fine for these type of circumstances where a robot is actually connecting to non-existent web pages that are actually getting uncovered through Googlebot.3. Links with the noindex tag will generate a "crawled/not indexed" item in Search Console and that those won't have a negative result on the remainder of the site.Read through the question and also answer on LinkedIn:.Why would Google.com index pages when they can not even see the web content?Included Photo through Shutterstock/Krakenimages. com.