Latest discoveries about deindexations and Search Engine crawling

searchbotEvery time a blog gets deindexed on EBN, we save all the information we have about it so we can analyze it later. We then do batch analysis to see if there are any patterns or footprints that we can report to the community.

In the last few weeks, we found a few interesting things. We’re still discussing on how to implement these into Blog Health to improve deindexation prevention but they’re useful insights nonetheless.

Here’s what we found.

Search Engine Crawler is visiting old URLs all the time

If you buy an expired domain with history, old URLs get crawled often and for long periods of time. This can go on for months, even if the URLs report 404 error.

Unrelated content and/or language can cause deindexation

We found that rebuilding old domains and using unrelated content and/or different language can cause deindexation.

Comments increase Search Engine Crawler visits

Comment feed is checked daily. If there are no comments, the blog is crawled less often.

Blocking crawlers can cause deindexation

We did not find any issues with users of Spider Blocker, however a lot of users add more than one plugin and block additional crawlers. Do NOT do this. Use one blocker and block as little crawlers as possible.

Some domains are permanently penalized

Some penalized domains never get any crawler traffic and will therefore never get indexed. Unfortunately we don’t yet have the data on how long this penalty can persist or what is the root cause of it (email spam, malware, phishing etc.).

Search Engine Crawler still visits the blog after deindexation (!)

When a domain gets removed from the SERP (deindexed), the old URLs still get crawled regularly and that stops only after 5-7 days. This could mean there are still options to save your blog after it gets deindexed by rebuilding URLs with relevant content.

Since we’re using passive indexation check, this is the reason why our indexation status can be late for 7-14 days (while Blog Health is checked daily).

Summary

Here’s a quick recap:

  • Rebuild URLs with relevant content that would fit on the old domain. Use the same language.
  • Check domains in spam and malware databases before buying them.
  • Use only one spider blocker, we recommend our free Spider Blocker plugin and block only the most important crawlers.

While none of this is a complete surprise, it’s just something that we can now confirm with data, not just speculation.

In the future, we’re going to start collecting even more information about domains – from social metrics to backlinks and blacklist databases. Once we have that, our analysis and deindexation prevention will greatly improve.

Price adjustment for all plans and 500 IP plans coming in September

We have two major announcements to make in light of the enormous progress EBN has made as a platform in the last few months.

  • We’re adjusting prices on September 1st for all plans from Mini to Enterprise 300. The average price increase is 20%. This does not affect the plan you’re subscribed to.
  • From the same date, all plans will support up to 500 unique IPs.

Please continue reading for more information.

New Plans and Pricing

Plan # Blogs  # Unique IPs Price
Mini 10 10 $35
Blogger 15 15 $49
Blogger+ 25 25 $69
Webmaster 40 40 $99
Webmaster+ 70 70 $149
Agency 100 100 $199
Enterprise 150 150 150 $279
Enterprise 200 200 200 $359
Enterprise 250 250 250 $439
Enterprise 300 300 300 $529

The pricing change does not affect the plan you’re subscribed to.

Why are we increasing prices?

As we mention often, our IPs are only high quality IPv4 IPs received directly from hosting providers (and are not proxy, gaming or CDN IPs) and these IPs are *really* hard to get. This means we need to do a major upgrade to our infrastructure to be able to request more IPs, offer up to 500 unique IPs to users and keep each regular user below 5% of our total IP pool.

EBN is a lot better than when we started almost two years ago. We’re a safer and more secure platform and we see the lowest deindexation rates in the industry. The average daily deindexation rate is below 0.3% and we often see as low as 0.15% (and the large majority of those still do not follow our recommendations).

We also do A LOT more than any of our competitors. From automatic maintenance, updates, backups to security and performance and now automatic nameserver configuration. Plus, just a few weeks ago we’ve added Blog Health which is a feature that will keep improving and helping you avoid deindexation on your network.

But the expansion of EBN features has come with extra costs. We’ve tried to suck it up for as long as we can, but if we’re going to keep providing this high level of service and continue developing EBN, it’s become clear we need to raise the prices a bit.

Now, for some great news!

With that said, we also have some great news. With the upgrade of the infrastracture, we will be able to provide 500 unique IP sets. So instead of 50 that we offer now, you will be able to host blogs on up to 500 unique IPs. This will affect all plans that have more than 50 IPs: from Webmaster+ that will now include 70 unique IPs to Agency with 100 unique IPs and all Enterprise plans up to 500 unique IPs (largest ones will have sets of these 500 IPs).

This has been requested many, many times and has been a blocker for some of you who wanted to upgrade but really wanted more IPs. It’s a major upgrade and one it took us a long time to safely get to.

Lock-in the current prices by upgrading before September 1st

Let’s just repeat again so there are no misunderstandings: the prices will not change for the plan you’re subscribed to. This means you can take advantage of the current pricing table and lock-in your price for the future. Old plans will also receive a free upgrade to allow up to 500 unique IPs. This means if you’re subscribed to Enteprise 250 plan, you will be able to host your blogs on 250 unique IPs from September 1st for the same price.

Upgrade by going to the Settings page.

If you planned on upgrading in the coming weeks, we highly recommend you do it this month to take advantage of, and lock-in the current prices, before the increase.

We plan on keeping EBN strong and growing for months and years to come and this change is one that will help us do that.

If you have any questions, don’t hesitate to contact me directly on Facebook, email or just leave a comment below.

Not a user yet?

If you’re not yet a user of EBN, sign up for our waiting list on the homepage.

Easy Blog Networks Update #19

In the last two months we’ve been working on adding more hosts, improving the stability and security of the platform and scaling to accommodate for more servers and IPs.

Here’s a quick summary:

  • Automatic nameserver setup.
  • New hosting and DNS providers: AccuWebHosting, SupremeServers, Route53.
  • Added more servers and IPs.
  • Scaling the platform.
  • Whois tab in DNS info section of the Blog Info page.
  • Adding API Key field instead of password for more secure third-party use of EBN.

What are some of the features you’d like to see next? Vote on our feedback page!

Taking another step in automating setup – Automatic Nameservers

mngregcredWe took another big step in automating setup of your blogs. Instead of going to your registrar for each domain to set your nameservers, you can now setup the credentials once and EBN will use their API to set them up automatically.

All you need to do is add the credentials in your account and everything else is done automatically. If the registrar is not supported or you don’t have credentials set, then you will get the nameserver addresses reported in the Activity Log, the same way as it is now.

Please keep in mind the domains have to be registered and in your account.

Supported Registrars

  • GoDaddy works out-of-box.
  • Dynadot supports the API only for bulk customers (above $500/year spend).
  • NameCheap and NameSilo need whitelisting of our IP. See our help article for more information.

Which other registrars are you using?

Let us know which other registrars you’re using and we’ll make sure to add them if they support nameserver setup through an API.

IP Ownership Analysis Of Cheap Hosting Providers: Only 20% With Own Servers

When people new to PBNs ask where to host their blogs, a lot of experts recommend using different cheap hosting providers to get a diversity of IPs. We wanted to put that claim to the test for some time now.

Analyzing 80 Cheap Hosting Providers

We collected a list of 80 popular cheap hosts from different forums and FB groups. Then we checked IP ownership of the homepage website with the method we discussed in our blog post about ASN in the footprints series.

The result?

The 80 hosting providers’ IPs are owned by only 17 companies. More than a third of those IPs are owned by just one company (EIG). Only 20% looks like they have their own servers.

In comparison – Easy Blog Networks IPs are spread over more than 30 companies – and you only need one login, not 80.

Screen Shot 2016-05-23 at 19.02.44

By far the largest owner is The Endurance International Group (36%) that is known for their acquisitions in the hosting industry. They own BlueHost, Hostgator, HostMonster, HostNine, SEOWebHosting, Site5 and tens of other hosting brands.

Other known brands are OVH, SoftLayer and HostDime.

So, what’s happening?

What people don’t know is that cheap hosting providers don’t usually have their own servers, IPs nor data centers. They buy servers or reseller plans from big brand hosting providers and just host all websites there.

Now, let’s be clear – there is nothing wrong with this and it will not get your PBN deindexed. However, if you plan on subscribing to 80 hosts and manage 80 logins, I am sure that you expect a lot more IP diversification. Therefore before singing up with individual hosting providers, make sure to analyze their homepage’s IP and ASN to get as much diversification as possible.

And of course with Easy Blog Networks you get access to almost twice the ASNs as in the above 80 hosts with A LOT less hassle.