Eve For Enterprise    Eve Support Community    Forums  Hop To Forum Categories  Resources  Hop To Forums  Community Management    Do search engines crawl the ubb boards???
Go
New
Find
Notify
Tools
Reply
  
-star Rating Rate It!  Login/Join 
wilty44
Groupee Newbie
Posted
I have read many of the topics in this and other ubb boards but I have not recieved the answer I am looking for.

I know search engine robots dont crawl cgi files, so what is the BEST way to make sure your pages are crawled?

Any help will be truly appreciated!!

Thanks,

Mark
 
Posts: 6 | Registered: July 05, 2000Reply With QuoteEdit or Delete MessageReport This Post
certify
Groupee Member
Posted Hide Post
I don't think search engines can craw UBB boards since the extentions end with .cgi

Just a thought.

------------------
Your Free Certification Resource
I.T Discussion Board
UBB TopSites 100
 
Posts: 210 | Registered: September 01, 2000Reply With QuoteEdit or Delete MessageReport This Post
Dan Hall
Groupee Member
Posted Hide Post
I went to Yahoo and typed in Ultimate.cgi as a keyword. It pulled up 26,200 web page matches, almost all of them for UBB boards. So, I would say that the search engines do crawl the UBB, even though they end in .cgi. Maybe not ... I read on here somewhere about a hack that will allow search engines to crawl through UBB sites, but I can't find it now ... Anyone have any suggestions?

------------------
EndlessPath Forums - Join the Revolution
Click Here
 
Posts: 335 | Location: Columbus, GA, USA | Registered: July 01, 2000Reply With QuoteEdit or Delete MessageReport This Post
SWFan

Posted Hide Post
Remember, that the thread .cgi files actually create a whole bunch of .html files which search engines might crawl. Also, you could put in meta tags in the header insert field which would make the board and threads more open to search engine crawls.

------------------
Doug
darksaber.hispeed.com
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
--J.R.R. Tolkien, The Lord of the Rings
 
Posts: 5917 | Location: Kansas City | May The Force Be With You | Registered: February 26, 2000Reply With QuoteEdit or Delete MessageReport This Post
Slurpee


Posted Hide Post
Google which Yahoo! uses to match web pages will show your Ultimate.cgi page in their search results.

------------------
Brett (AKA Slurpee)
Van Halen Discussion Forum at VHLinks.com
http://www.vhlinks.com/cgi-bin/vhlforum/Ultimate.cgi
Moderator - UBB Marketing, Web Hosts & ISP'S
 
Posts: 2883 | Location: Sherman Oaks, CA | Registered: October 11, 2001Reply With QuoteEdit or Delete MessageReport This Post
METRO25
Beta Customer
Posted Hide Post
Yes, search engines actually do crawl through UBB HTML pages!

I had my CGI directory secured with .htaccess however left the Non-Cgi directory un-secured. I dont know how it happened, but somehow the search engines got my entire UBB in their system frown

Doesnt matter now since Im using OpenTopic, but Its amazing that I still get hits to my old UBB when its been inactive for the past 5 months!
 
Posts: 424 | Location: Detroit, Mi | Registered: March 16, 2000Reply With QuoteEdit or Delete MessageReport This Post
certify
Groupee Member
Posted Hide Post
Cool! smile

So we know Yahoo do craw, what about other search engines?

------------------
Your Free I.T Certification Resource
Your Free Certification Resource
I.T Discussion Board
Looking for new Jobs? More than 500,000 positions available now!
 
Posts: 210 | Registered: September 01, 2000Reply With QuoteEdit or Delete MessageReport This Post
mbuna
Groupee Member
Posted Hide Post
Every search engine is different. Some crawl cgi's others don't.

Take a look at http://searchengines.com
to find out info on the different engines.
 
Posts: 101 | Registered: September 06, 2000Reply With QuoteEdit or Delete MessageReport This Post
PepeThomas
Groupee Newbie
Posted Hide Post
quote:
Originally posted by SWFan:
Remember, that the thread .cgi files actually create a whole bunch of .html files which search engines might crawl. Also, you could put in meta tags in the header insert field which would make the board and threads more open to search engine crawls.

But how does the search engines locate the HTML pages that are automatically created?

Where can we find them?

Is there a single URL with links to HTML versions of all topics? Such a link could be submitted to the search engines.
 
Posts: 3 | Registered: January 10, 2001Reply With QuoteEdit or Delete MessageReport This Post
Dan Hall
Groupee Member
Posted Hide Post
Yes, Pepe. For instance, the direct link to the page that you are looking at right now is http://www.scriptkeeper.com/ubb/Forum13/HTML/000218.html

smile
 
Posts: 335 | Location: Columbus, GA, USA | Registered: July 01, 2000Reply With QuoteEdit or Delete MessageReport This Post
 Previous Topic | Next Topic powered by eve community  
 

Eve For Enterprise    Eve Support Community    Forums  Hop To Forum Categories  Resources  Hop To Forums  Community Management    Do search engines crawl the ubb boards???