.Google's John Mueller responded to an inquiry regarding why Google.com indexes webpages that are refused from creeping through robots.txt and also why the it is actually safe to ignore the relevant Explore Console documents regarding those crawls.Crawler Website Traffic To Question Guideline URLs.The individual talking to the question chronicled that bots were developing web links to non-existent concern specification URLs (? q= xyz) to pages along with noindex meta tags that are actually likewise blocked in robots.txt. What triggered the question is actually that Google.com is crawling the web links to those web pages, obtaining blocked by robots.txt (without envisioning a noindex robotics meta tag) after that getting turned up in Google.com Look Console as "Indexed, though obstructed by robots.txt.".The person asked the following concern:." But here is actually the huge inquiry: why will Google.com mark pages when they can't also observe the information? What is actually the advantage in that?".Google's John Mueller validated that if they can't creep the webpage they can not view the noindex meta tag. He also makes an intriguing mention of the internet site: search driver, encouraging to overlook the end results considering that the "typical" users won't find those outcomes.He composed:." Yes, you're appropriate: if our team can't crawl the page, we can't see the noindex. That pointed out, if our company can't creep the webpages, after that there's certainly not a lot for our company to mark. Therefore while you could see a number of those pages along with a targeted internet site:- query, the common customer won't find all of them, so I would not fuss over it. Noindex is additionally great (without robots.txt disallow), it merely indicates the URLs will definitely end up being actually crept (and also wind up in the Look Console document for crawled/not catalogued-- neither of these statuses lead to concerns to the rest of the web site). The important part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the limits in using the Site: hunt advanced search operator for diagnostic factors. Some of those explanations is due to the fact that it's not hooked up to the regular hunt mark, it is actually a separate trait altogether.Google's John Mueller talked about the internet site search driver in 2021:." The quick response is that an internet site: query is certainly not indicated to become total, neither used for diagnostics reasons.A website query is a specific type of search that limits the outcomes to a particular internet site. It is actually essentially just words internet site, a bowel, and after that the internet site's domain.This question limits the end results to a particular website. It's certainly not meant to become an extensive collection of all the pages from that site.".2. Noindex tag without utilizing a robots.txt is great for these type of conditions where a bot is actually connecting to non-existent pages that are actually obtaining found through Googlebot.3. Links with the noindex tag will definitely produce a "crawled/not catalogued" entry in Search Console and that those won't have an adverse result on the remainder of the site.Go through the question and answer on LinkedIn:.Why would Google mark pages when they can not even see the material?Featured Image by Shutterstock/Krakenimages. com.