Item logo image for Robots Exclusion Checker

Robots Exclusion Checker

www.samgipson.com
Featured
4.8(

32 ratings

)
Item media 4 screenshot
Item media 5 screenshot
Item media 1 screenshot
Item media 2 screenshot
Item media 3 screenshot
Item media 4 screenshot
Item media 5 screenshot
Item media 1 screenshot
Item media 2 screenshot
Item media 1 screenshot
Item media 2 screenshot
Item media 3 screenshot
Item media 4 screenshot
Item media 5 screenshot

Overview

Checks robots.txt, meta robots, x-robots-tag with URL alerts. Canonical warnings, HTTP header info. An SEO extension, robots tester.

Robots Exclusion Checker is designed to visually indicate whether any robots exclusions are preventing your page from being crawled or indexed by Search Engines. ## The extension reports on 5 elements: 1. Robots.txt 2. Meta Robots tag 3. X-robots-tag 4. Rel=Canonical 5. UGC, Sponsored and Nofollow attribute values - Robots.txt If a URL you are visiting is being affected by an "Allow” or “Disallow” within robots.txt, the extension will show you the specific rule within the extension, making it easy to copy or visit the live robots.txt. You will also be shown the full robots.txt with the specific rule highlighted (if applicable). Cool eh! - Meta Robots Tag Any Robots Meta tags that direct robots to “index", “noindex", “follow" or “nofollow" will flag the appropriate Red, Amber or Green icons. Directives that won’t affect Search Engine indexation, such as “nosnippet” or “noodp” will be shown but won’t be factored into the alerts. The extension makes it easy to view all directives, along with showing you any HTML meta robots tags in full that appear in the source code. - X-robots-tag Spotting any robots directives in the HTTP header has been a bit of a pain in the past but no longer with this extension. Any specific exclusions will be made very visible, as well as the full HTTP Header - with the specific exclusions highlighted too! - Canonical Tags Although the canonical tag doesn’t directly impact indexation, it can still impact how your URLs behave within SERPS (Search Engine Results Pages). If the page you are viewing is Allowed to bots but a Canonical mismatch has been detected (the current URL is different to the Canonical URL) then the extension will flag an Amber icon. Canonical information is collected on every page from within the HTML <head> and HTTP header response. - UGC, Sponsored and Nofollow A new addition to the extension gives you the option to highlight any visible links that use a "nofollow", "ugc" or "sponsored" rel attribute value. You can control which links are highlighted and set your preferred colour for each. I’d you’d prefer this is disabled, you can switch off entirely. ## User-agents Within settings, you can choose one of the following user-agents to simulate what each Search Engine has access to: 1. Googlebot 2. Googlebot news 3. Bing 4. Yahoo ## Benefits This tool will be useful for anyone working in Search Engine Optimisation (SEO) or digital marketing, as it gives a clear visual indication if the page is being blocked by robots.txt (many existing extensions don’t flag this). Crawl or indexation issues have a direct bearing on how well your website performs in organic results, so this extension should be part of your SEO developer toolkit for Google Chrome. An alternative to some of the common robots.txt testers available online. This extension is useful for: - Faceted navigation review and optimisation (useful to see the robot control behind complex / stacked facets) - Detecting crawl or indexation issues - General SEO review and auditing within your browser ## Avoid the need for multiple SEO Extensions Within the realm of robots and indexation, there is no better extension available. In fact, by installing Robots Exclusion Checker you will avoid having to run multiple extensions within Chrome that will slow down its functionality. Similar plugins include: NoFollow https://chrome.google.com/webstore/detail/nofollow/dfogidghaigoomjdeacndafapdijmiid Seerobots https://chrome.google.com/webstore/detail/seerobots/hnljoiodjfgpnddiekagpbblnjedcnfp NoIndex,NoFollow Meta Tag Checker https://chrome.google.com/webstore/detail/noindexnofollow-meta-tag/aijcgkcgldkomeddnlpbhdelcpfamklm CHANGELOG: 1.0.2: Fixed a bug preventing meta robots from updating after a URL update. 1.0.3: Various bug fixes, including better handling of URLs with encoded characters. Robots.txt expansion feature to allow the viewing of extra-long rules. Now JavaScript history.pushState() compatible. 1.0.4: Various upgrades. Canonical tag detection added (HTML and HTTP Header) with Amber icon alerts. Robots.txt is now shown in full, with the appropriate rule highlighted. X-robots-tag now highlighted within full HTTP header information. Various UX improvements, such as "Copy to Clipboard” and “View Source” links. Social share icons added. 1.0.5: Forces a background HTTP header call when the extension detects a URL change but no new HTTP header info - mainly for sites heavily dependant on JavaScript. 1.0.6: Fixed an issue with the hash part of the URL when doing a canonical check. 1.0.7: Forces a background body response call in addition to HTTP headers, to ensure a non-cached view of the URL for JavaScript heavy sites. 1.0.8: Fixed an error that occurred when multiple references to the same user-agent were detected within robots.txt file. 1.0.9: Fixed an issue with the canonical mismatch alert. 1.1.0: Various UI updates, including a JavaScript alert when the extension detects a URL change with no new HTTP request. 1.1.1: Added additional logic Meta robots user-agent rule conflicts. 1.1.2: Added a German language UI. 1.1.3: Added UGC, Sponsored and Nofollow link highlighting. 1.1.4: Switched off nofollow link highlighting by default on new installs and fixed a bug related to HTTP header canonical mismatches. 1.1.5: Bug fixes to improve robots.txt parser. 1.1.6: Extension now flags 404 errors in Red. 1.1.7: Not sending cookies when making a background request to fetch a page that was navigated to with pushstate. 1.1.8: Improvements to the handling of relative vs absolute canonical URLs and unencoded URL messaging. 1.2.0.11: Updating to Google's new manifest V3 and fixing small bugs. Found a bug or want to make a suggestion? Please email extensions @ samgipson.com

4.8 out of 532 ratings

Google doesn't verify reviews. Learn more about results and reviews.

Review's profile picture

Konrad JurkSep 24, 2024

Does the job but interferes with caching, resulting in "Too many redirect" errors on some websites (like Gitlab).

3 out of 3 found this helpful
Review's profile picture

Salman MAug 26, 2024

Save this extension from extinction, Please

2 out of 2 found this helpful
Replier's profile picture

Sam GipsonDeveloperAug 28, 2024

Don't worry it will be saved. A new version is being released to align with Google's new manifest v3. Expect a few bugs while we roll it out it but rest assured, I'm on it! Thanks for your patience

Review's profile picture

Forrest BJun 9, 2024

this extension adds an annoying red dotted border box around certain links, when working on websites, this becomes an issue as i just spent the last 3 hours going through my 100+ extensions one at a time to narrow it down. I understand this extension helps with checking robots.txt exclusion, but there was nothing telling me that this was being caused by your extension, you need some type of info next to it, so people know your extension is doing this. Otherwise this extension is great *** Up... Show more

1 person found this review to be unhelpful

Details

  • Version
    1.2.0.11
  • Updated
    August 28, 2024
  • Size
    185KiB
  • Languages
    3 languages
  • Developer
    Website
    Email
    extensions@samgipson.com
  • Non-trader
    This developer has not identified itself as a trader. For consumers in the European Union, please note that consumer rights do not apply to contracts between you and this developer.

Privacy

The developer has disclosed that it will not collect or use your data. To learn more, see the developer’s privacy policy.

This developer declares that your data is

  • Not being sold to third parties, outside of the approved use cases
  • Not being used or transferred for purposes that are unrelated to the item's core functionality
  • Not being used or transferred to determine creditworthiness or for lending purposes

Support

Related

View Rendered Source

4.5(37)

View source is dead. See how the browser renders a page, not just what the server sends.

GSC Guardian

4.5(26)

Stay on top of your website's health. Check for incidents in your Google Search Console data with a simple graph.

Inspect Canonical

4.5(11)

Easily check the canonical tag of a page

Google Search Console Enhanced Analytics

4.8(21)

Google Search Console Enhanced Analytics is a simple, time-saving extension that enhances your analysis.

Link Redirect Trace

4.8(345)

The all-in-one redirect path analyzer. It also analyzes protocol headers, REL-canonicals, robots.txt, backlink power, and trust!

Core SERP Vitals

3.5(15)

Show the Core Web Vitals data from the Chrome User Experience Report in the Google Search Results.

Detailed SEO Extension

5.0(193)

Get SEO insights at the click of a button to any website you're currently on. Built by full-time SEO's for daily analysis.

SEO META in 1 CLICK

4.9(1.1K)

Displays all meta data and main SEO information for the best SEO

Schema Builder for Structured Data

3.7(29)

Easily build validated json-ld structured data markup for any webpage. Based on schema.org specification

Seerobots

4.0(30)

SeeRobots conveniently displays the meta robots information of the website you are viewing - For more info visit www.seerobots.com

Gandalf

4.1(23)

Gandalf is a powerful Chrome extension designed to transform your Google Search Console experience.

NoFollow

4.3(236)

Outline nofollow links, detect nofollow and noindex meta tags on webpages. Features website filtering and custom CSS outline styles.

View Rendered Source

4.5(37)

View source is dead. See how the browser renders a page, not just what the server sends.

GSC Guardian

4.5(26)

Stay on top of your website's health. Check for incidents in your Google Search Console data with a simple graph.

Inspect Canonical

4.5(11)

Easily check the canonical tag of a page

Google Search Console Enhanced Analytics

4.8(21)

Google Search Console Enhanced Analytics is a simple, time-saving extension that enhances your analysis.

Link Redirect Trace

4.8(345)

The all-in-one redirect path analyzer. It also analyzes protocol headers, REL-canonicals, robots.txt, backlink power, and trust!

Core SERP Vitals

3.5(15)

Show the Core Web Vitals data from the Chrome User Experience Report in the Google Search Results.

Detailed SEO Extension

5.0(193)

Get SEO insights at the click of a button to any website you're currently on. Built by full-time SEO's for daily analysis.

SEO META in 1 CLICK

4.9(1.1K)

Displays all meta data and main SEO information for the best SEO

Google apps