Support community for TTG plugins and products.
NOTICE
The Turning Gate's Community has moved to a new home, at https://discourse.theturninggate.net.
This forum is now closed, and exists here as a read-only archive.
You are not logged in.
Pages: 1
you access your .htaccess file from the root of your site using your FTP program or your host's admin panel. There's nowhere in TTG plugins to store that.
Rod
Just a user with way too much time on his hands.
www.rodbarbee.com
ttg-tips.com, Backlight 2/3 test site
Offline
Rod
Just a user with way too much time on his hands.
www.rodbarbee.com
ttg-tips.com, Backlight 2/3 test site
Offline
Thank you Rod. I will be more careful in the future to do my own work. But you continue to be a good source.I want to put this in this post in case other people want the same thing
It's good programming policy. Pros have a robots.txt. Amateurs don't. What group do you want your site to be in? This is more of an ego/image thing than a "real" reason but in competitive areas or when applying for a job can make a difference. Some employers may consider not hiring a webmaster who didn't know how to use one, on the assumption that they may not to know other, more critical things, as well. Many feel it's sloppy and unprofessional not to use one.
The creation of the robots.txt file can be found http://www.mcanerin.com/EN/search-engine/robots-txt.asp
or here: http://searchengineland.com/google-offe … ator-13653
Here is the contents of the robots.txt file I created
# Disallows all robots
User-agent: *
Disallow: /
Last edited by gwlco (2013-03-13 00:29:05)
Offline
Good info. Thanks for posting it.
Rod
Just a user with way too much time on his hands.
www.rodbarbee.com
ttg-tips.com, Backlight 2/3 test site
Offline
robots.txt is the standard way.
Unfortunately, it is entirely voluntary on the part of the web crawler whether to honour it or not.
Offline
I run a number of different domains - one of which is a genelaogy site - I use exclusions in robots.txt files and, so far, (touch wood here!) none of this site can be found using any of the search engines I have used or experimented with.
Whilt it is true I see some activity from search engines, it is usually only in the order of around 30-100k of data a month from each one (as a comparison, one other ite I don't restrict, currently shows Google accessing around 10Mb of data a month - it all starts to build up).
Other sites to consider not allowing are archive.org, sitedossier.com amongst others - thee ite archive (without permission) your site and it becomes increasingly frustrating when people find older versions of your site when you don't want them found!
Offline
Could I ask a real dumb newbie question?
Why would I want to block crawlers from Indexing my site?
--Lofty
Offline
Not all crawlers are benign.
Not all galleries (e.g. boudoir, wedding etc.) are public interest, even though held on a www server.
Offline
OK Kris,
What are the consequences of stopping the crawlers, presumably no one would know about your site unless you told them?
--Lofty
Offline
Right. The intent is to prevent your site -- or sections thereof -- from being found via search engine. The opposite of SEO.
Offline
OK Thanks Guys, asked and answered..
Offline
It's not to just stop your site from being 'searched', it's also to prevent these crawlers from using massive amounts of bandwidth which they can, and often, do!
Offline
Pages: 1