robots. txt / googlebot questions

Involved Joined: Feb 08, 2005 Posts: 290 Location: USA

I have the same robot.txt in the /public_html/ directory and I do my /nuke/ directory. However, these directories do not contain the same folders and files obviously ... nuke is a directory (folder) under the public_html directory. Am I to assume that each needs to list their own folders/files I want to disallow?

One site I saw said to do a robot.txt file disallowing the following (which obviously is for the site's /public_html/ folder:
/_private/
/_vti_bin/
/_vti_cnf/
/_vti_log/
/_vti_pvt/
/_vti_txt/
/cgi-bin/
Being private folders and not being in my nuke folder I assume I need a whole new robot.txt for the public_html folder listing those folders above I don't want accessed? Do I also need one for the www folder?

Should googlebot be able to view ... ?
--> Your_Account
--> Private_Messages
My IP_Tracker says some of these are being accessed by googlebot even tho the /Modules/ and /Admin/ folders are listed as Disallow for the robots.txt

Should I do these two steps as indicated by a site I was reading for the robots.txt?
Ban Alta-Vista *Scooter* bot altogether
--> User-Agent: Scooter
--> Disallow: /

Ban *Googlebot-Image* bot altogether
--> User-Agent: Googlebot-Image
--> Disallow: /

Finally, should I allow or disallow these two folders in the nuke robots.txt?
--> /Experimental/
--> /import/

IP_Tracker is also showing these, who are they???
164.82.146.3 - gw1.dc.gov
162.58.0.224 - nat.jccbi.gov
I know they're US Gov't sites ... I'm just not sure why they'd be visiting my lil space on the web LOL Are they ok or should I ban em?

Posted: Fri Sep 22, 2006 5:42 pm

Bluezzz wrote:

I have the same robot.txt in the /public_html/ directory and I do my /nuke/ directory. However, these directories do not contain the same folders and files obviously ... nuke is a directory (folder) under the public_html directory. Am I to assume that each needs to list their own folders/files I want to disallow?

One site I saw said to do a robot.txt file disallowing the following (which obviously is for the site's /public_html/ folder:
/_private/
/_vti_bin/
/_vti_cnf/
/_vti_log/
/_vti_pvt/
/_vti_txt/
/cgi-bin/
Being private folders and not being in my nuke folder I assume I need a whole new robot.txt for the public_html folder listing those folders above I don't want accessed?

You would only need those extra directories listed if they exist. You would only normally find those on a Windows server or a server which has had the Windows Frontpage extensions added.

Quote:

Should googlebot be able to view ... ?
--> Your_Account
--> Private_Messages
My IP_Tracker says some of these are being accessed by googlebot even tho the /Modules/ and /Admin/ folders are listed as Disallow for the robots.txt

There is nothing be gained by letteing google or other bots access those and it will just waste bandwidth.

Quote:

Should I do these two steps as indicated by a site I was reading for the robots.txt?
Ban Alta-Vista *Scooter* bot altogether
--> User-Agent: Scooter
--> Disallow: /

Ban *Googlebot-Image* bot altogether
--> User-Agent: Googlebot-Image
--> Disallow: /

That is entirely a personal preference thing - Sentinel wil block a lot of bad bots by default any way.

Quote:

Finally, should I allow or disallow these two folders in the nuke robots.txt?
--> /Experimental/
--> /import/

They shouldnt be left on the server so, no.

Posted: Fri Sep 22, 2006 6:40 pm

The server I'm on is linux.

I don't want google picking up my images so I will add that one to both public and nuke directories ... altavista one I'm not sure why it would matter so I'll leave it out.

I've removed these two folders ... thank you for telling me : o}

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
More questions ...

1) So then make a new robots.txt file to keep googlebot (and others) out of the main public_html directory blocking access to the _private folders?

2) Why is googlebot getting access to Your_Account and Private_Messages when both are covered under /modules/ for Disallow? I noticed that someone used /modules (no last backslash) ... which is correct in robots.txt ... ?
/modules/
or
/modules

3) What about those two gov't sites that IP_Tracker is showing? Are these ok to leave on or should they be banned or what? Normal or not?

Posted: Fri Sep 22, 2006 11:54 pm

You dont need to make a new robots.txt, you simply need to add those folder paths to it if you don not want bots to crawl theose folders.

I need to explain a bit more fully for others whilst replying to your point 2 above.
the normal robots.txt directive for /modules is to stop well behaved bots from crawling inside the actual folder itself. As all nuke sites have links to SOME of the files inside that folder such as all the index.php files in all of the different modules, those will quite naturally and legititimately get crawled and indexed.
So to put it in really simple terms, it tells the bots to not go 'snooping' in that folder but it is allowed to follow other links even if they go to files within that folder.

As for the gov't sites, I'm going to leave you to research those but you would need to find out the IP for those URL's then also cross check to see if the IP's give you back the web address. If not why not?

Posted: Sat Sep 23, 2006 1:36 am

Thanks for the clarification. I'll leave that bot to wander for now and if I find out it's cataloging those particular folders I'll add them in the Disallow. I did go ahead and add the _ Private ones also tho.

As for the two gov't sites they are:

164.82.146.3 gw1.dc.gov
162.58.0.224 nat.jccbi.gov

Seems they show up for others' tracking also as I did a google search on them. The 1st one is nothing with gw1 but without that it's Washington, DC gov't page ... the second one (jccbi.gov) returned failed pages for url with and without nat. I dunno why they'd be interested in my site LOL ... it's just a lil ole thang and very new yet! Man they're quick!

0.o @ Big Brother LOL