Free Spider Blocker Plugin

 
LAST UPDATE: September 2017

Due to issues with similar plugins, we decided to create our own spider/crawler blocker plugin, called Spider Blocker. It’s completely free, open-source and without footprints. You can easily add it to each new blog in the Easy Blog Networks Add Blog form or download it from WordPress.org and install it.

Spider Blocker in EBN Dashboard

Setup Spider Blocker

Plugin will block the most common crawlers like Majestic, Ahrefs and MOZ and hide your blogs from prying eyes of your competitors. It will do so by editing .htaccess file. It has no footprints and responds with standard 404 error when pinged.

Once installed Spider Blocker can be found under Tools menu. You can easily add new, remove existing or just allow/disallow existing spiders.

spider blocker

Troubleshooting

Some users receive “fatal error” when activating the plugin. This is most commonly caused by outdated PHP. Make sure your server is running PHP 5.4+ and Apache 2.4+.

23 thoughts on “Free Spider Blocker Plugin

  1. Simon

    This is a great addition. I don’t now if everyone else picked up on its main purpose, but the reason we use it is so your competitors cant bulk report all your PBN’s googles spam team.
    I would like the ability to add a redirection plugin from the admin panel? I know some people use the 404 redirect to home page but this is a known potential flag for a penalty for some time now. I like to recreate old pages that had links going to them and redirect the old URL to the new.
    So be it a simple custom redirect plug in you build or this one does not matter too much: https://wordpress.org/plugins/redirection/

    Reply
    1. Dejan Murko Post author

      Brad, can you provide more information – how would it look like, what it would do, etc.

      Reply
  2. Andrea

    Since PBNs should leave no footprint, then I don’t use Google analytics or webmaster tools. I have looked for plugins to replace these 2 but they have pieces of information so i would end up having at least 3 plugins to get all the info I want. It would be ideal to get an alternative to Google and that it gives you the ability to whitelist certain IPs so that your visits are not counted in the metrics.

    Another great plugin would be one that can automatically generate Privacy Policy and Terms and Conditions directly from WP. I know there are several websites but it would be nice to just click and have it all done and posted in the right place with no effort. A plus would be to have also the ABOUT US page randomly generated and posted. This way when we build a PBN we only worry about adding content.

    Reply
    1. Dejan Murko Post author

      Andrea, check out Piwik Analytics as an alternative to Google. You install it on your own server and use a WordPress plugin. However, I think it still leaves the Piwik ID in the source code so you can’t use it on all sites. I feel your pain, I’m using WassUp analytics for one of my sites and it’s bare basics.

      However creating an Analytics plugin would be quite a job.

      I’ll see what we can do with the privacy/terms plugin.

      Reply
        1. Kevin

          I agree Walt I have used this one to both create the pages plus as a guide to what a page should have on it but in my own words.

          Reply
  3. Mihai Cirstea

    There is a way to configure Piwik so that it doesnt insert the id in the website, but rather use a php file for tracking.

    Doing it this way doesn’t leave any footprints. Most plugins do it the other way you mentioned and does leave a footprint.

    A plugin that allows for this configuration and automatic upload of the php file would be great.

    Contact me if you need more details…

    Reply
    1. Dejan Murko Post author

      Yeah, I was aware of that feature but no plugin as of yet does not provide that.

      We can look into it if enough people use Piwik.

      Reply
  4. Raj

    Hi Dejan,

    When I need to block the bots from crawling my PBN pages which are the bare crawlers except google bot which I should allow in my PBN site? May be only Bing.com but I am not sure about these.

    Cheers
    Raj

    Reply
  5. Dennis O'Brien

    So what is the advantage of this plugin over using disallow in your htacess file? That has worked well for me. Now if it solves the problem of bots that hunt down WordPress installations that would be something. There may be something out there that does this but I haven’t been actively searching for some time.

    I am very interested in the elimination of the Piwik tracking ID in lieu of a PHP file as long as it can be sent to my CDN for caching. I used Piwik for a very long time and find it provides useful info but for clients wanting their local business at the top of local Google results I find Webmaster Tools to be the quickest path. I would be a little concerned in changing to Piwik for them as they paid me to get the results. Linking them to Google plus, YouTube and a Google business page with lots of circle sharing gave a kick start the the boost in their rankings.

    Regards Dennis

    Reply
    1. Dejan Murko Post author

      Hey Dennis,

      Spider Blocker edits .htaccess so the only difference is that you have a user interface to edit it.

      If you’re using Google tools for each client separately then I see no reason for changing that.

      Reply
  6. Brian Duvall

    I’ve tried installing it but get this error message:
    Fatal error: Call to undefined function Niteoweb\SpiderBlocker\apache_get_version() in /home/safetyst/public_html/wp-content/plugins/spider_blocker/index.php on line 170

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *