Bots are giving inaccurate thread view numbers

[commando]

Bots seem to be the only ones scouring the forums and adding unrealistic thread count numbers, thousands and thousands more.
EG. Bing [Bot], Google [Bot], J M, Majestic-12 [Bot]
Privacy among members is compromised

[CasualNerd]

If the forum software is picking up the bots as it should, I don't see why it would be adding those numbers to thread counts. The thread counts all look pretty reasonable too.

What's the privacy concern ? It's a publicly available website.

[FatCanyoner]

As someone who runs a different, smaller forum, I can tell you that all those bots are search engine crawlers. You don't see them do their work on most websites, but in forum software they are generally given specific privileges. That allows the admins to control what parts of the forum are added to search engines, and what areas are not captured. It also means their work is transparent. Search engine bots don't impact thread counts. They are important though, because without them people wouldn't stumble on this forum based on web searches. There is absolutely no privacy issue, because all they trawl is public posts.

There are many nasty bots out there, but they won't show up because the nefarious ones don't get access to those special privileges. They generally just try to infiltrate the site as standard members then post spam. I guarantee the admins are kept busy keeping them out. They could potentially skew the thread counts, but they mostly just try to post to forums.

[wildwalks]

Thanks guys.
Yes I think I agree with FatCayoner and CasualNerd.

This is a public website, so people are aware (and it is a good thing) for our conversation to be found and shared.

Search engine Bots don't keep smashing a page, and we do detect common bots and ignore them in counts.

There are occasional bots that are just roaming the web looking to cause trouble that looks more like regular users -- but if this was the case then the count would be more evenly spread across forums.
Instead, I think what we see are actual numbers -- the do correlate fairly well with Google Analytics (which is very good at filtering out non-human users).
In the last 12 months, we have served about 1.5m pages. Most of our users are stalkers, not members

With really good and a very long history of information (and with many forums and FB groups now hidden away from search engines) the information in this forum actually rates very well. I see spikes in usage correlating with something in the media or sometimes changes in search algorithms meaning pages get promoted.

So I recon we are doing something right. We are a helpful community making sure that we responsibly share our ideas and experiences broadly.

Thanks
Matt