Robot concerns

[garuda]

Tim, should there be concern with these numerous, repeated robot queries? Do we know what they are extracting? See pic below:

'

 

[Jeff A]

I for one, have never seen what you have shown above.

 

[garuda]

I for one, have never seen what you have shown above.

I was browsing about the forum to learn all the features and just noticed this. Seems those little guys are quite busy. I thought Tim should know about it, if he doesn't already. It may not be a problem, since nowadays bots are everywhere. Depends on what they are extracting, I guess.

 

[Tim Mayo]

These are just your typical Google/Bing Search Robots. They crawl the site looking for new content to include in their Search Results Pages.

Googlebot - Wikipedia

en.wikipedia.org

They are able to view everything that none registered members can see, which is most things, although they can't view profiles. Some images might also be indexed if they are included in threads that have been indexed.

I try to block a lot of the gallery images from being indexed by search engines as there is no need for them to be really, I also only allow useful threads to be indexed at this time.

 

[Ivanturas]

XenForo (which I believe is the engine underneath these forums) is pretty good at handling search engine crawlers. I wouldn't worry much; as Tim said, they will only index what's public.

 

[garuda]

...... they will only index what's public.

.
Q#1: When you say they index what is public, describe how indexing works, and what parts of the indices are available directly to public surfing internet? Except for profile data, isn't everything on all forums public and available to surfers?

Q#2: While surfing internet, I've noticed references to forum posts on various forums. When you click them, you go to the post to read. So if a photo is attached or embedded in that post, is the photo also available to surfers?

Q#3: Can Google extract photos and place them in their "Goggle Photos" library? Just curious.

 

[Tim Mayo]

Q#1: When you say they index what is public, describe how indexing works, and what parts of the indices are available directly to public surfing internet? Except for profile data, isn't everything on all forums public and available to surfers?

Search engines can index anything that is viewable without logging into the site, which is most content here apart from user profiles. I can add add a tag rel="noindex" to pages/threads/posts that tells search engines that you would rather that particular page isn't included in the search results, and they may or may not adhere to that advice. Probably around 95% of content here has the rel="noindex" tag, as if you allow Google to index everything from a forum you can run into problems with thin content penalties (like a single short thread with no replies), then the site doesn't rank well in Google and the traffic/members drop.

Q#2: While surfing internet, I've noticed references to forum posts on various forums. When you click them, you go to the post to read. So if a photo is attached or embedded in that post, is the photo also available to surfers?

Yes the photo will also be available to surfers which will be guests on the forum until they register, unless it's in a personal album that you have restricted to members only.

Q#3: Can Google extract photos and place them in their "Goggle Photos" library? Just curious.

Yes it's possible for Google to include images on the forum in their image search. I'm not sure what the latest dimensions are but typically if they do this then it's a low resolution image with a link back to the forum.

 

[Ivanturas]

.
Q#1: When you say they index what is public, describe how indexing works, and what parts of the indices are available directly to public surfing internet? Except for profile data, isn't everything on all forums public and available to surfers?

Q#2: While surfing internet, I've noticed references to forum posts on various forums. When you click them, you go to the post to read. So if a photo is attached or embedded in that post, is the photo also available to surfers?

Q#3: Can Google extract photos and place them in their "Goggle Photos" library? Just curious.

I think Tim replied great to all of these. I'd just add:

Not necessarily! Forums allow admins to define both public and private content, including categories. As an example, I used to participate on a gearhead forum, where the whole "show us your project car" category was private for security reasons, as some people had a load of cash thrown in their cars.

If by "Google Photos" you mean "Google Image Search", then as Tim said, yeah, they are indexed. That doesn't give Google any rights over the images, but it is possible for people to download a low res version of the pic, and depending on how tech savvy they are, download the same pic being displayed here, without ever entering the forum.

Now, considering "how the internet works", regardless of how search engines work, or the forum's privacy settings, if what you want is for no one to download any of your photos without you knowing, then the only way is to not upload them at all, unless it's something like a private cloud drive or your own server (like the laptop that's in my home office's closet running Ubuntu server)... and even then, your stuff is still "connected" to the world, just behind some walls, but that's a more technical thing not relevant to what we're talking right now.

Perhaps it could be a good idea to make all showcase categories private, while keeping "conversation" categories public? But then again, there's a lot of really useful information in the showcase threads that would be excluded, and we should actually want for more people to find it, read more and more until they sign up and hopefully become good contributors in the community. I believe I actually got here while searching for info after getting my Alpha! (switched from Nikon about... 4 months ago?).

 

[Tim Mayo]

Perhaps it could be a good idea to make all showcase categories private, while keeping "conversation" categories public? But then again, there's a lot of really useful information in the showcase threads that would be excluded, and we should actually want for more people to find it, read more and more until they sign up and hopefully become good contributors in the community. I believe I actually got here while searching for info after getting my Alpha! (switched from Nikon about... 4 months ago?).

I might create a private pro member forum that is restricted to pro members only. But obviously making all of the other categories restricted means that not many people would find the site, since many are discovering it via search engines like yourself. If that was the case it might not even be worth running it as very few people would join up and the community here would be very quiet.

 

[Ivanturas]

I might create a private pro member forum that is restricted to pro members only. But obviously making all of the other categories restricted means that not many people would find the site, since many are discovering it via search engines like yourself. If that was the case it might not even be worth running it as very few people would join up and the community here would be very quiet.

Absolutely agree! The private Pro forum does sound like a really good idea. Hadn't thought about that!