MINING THE WEB
N.Subramanian

Mining the web

Search Engines

Meta Search Engines

Directories

The invisible web

Discussion Groups


Everyone knows that the Web is a balloon of information. Everybody also knows that this balloon is big, very, very big. No matter what you need to know, chances are that it’s already on the Web waiting for you to tap it. Logically therefore, whenever you need any information on say “the mining industry”, all you need to do is pick it from the Web. But it isn’t that easy always, is it? While the Web is accessible, getting it to supply the information you require is a tough ask if you don’t know how. This article looks at various tools available today that help query the Web effectively and zero down on the information we want. Of course, the emphasis is on mining industry-related information.

Start the search:

Information on the Web is accessed primarily through four avenues:
    Search engines
    Meta search engines
    Directories
    Invisible Web
    Discussion groups
As with everything in life, understanding the specifications of your requirement is most important for deciding the right search tool. Lets look at each of the above in some detail highlighting when each is most useful.

Search Engines:

The most used search tool on the Web, the typical search engine is a three-part tool consisting of a spider, an index and the search engine software. The spider (or crawler) searches the web and builds a net of web pages. These are then filed into a database containing copies of web pages gathered by the spider called the index. Finally the search engine software enables users to query the index and returns the results in the order it considers most relevant.
So why is all this important? The simple reason is if you understand what portion of the Web is crawled by a spider and how the search engine software interprets your queries, you can very well estimate your chances of obtaining the right answers to your query.
In terms of coverage, Google is the clear leader with a crawled base of about 1,060,000,000 web pages. It is also unique in that it computes page-ranks for each page, which is calculated depending on the number of links from authoritative pages and documents with higher page ranks corresponding to your queries are returned as primary results. Google is a prime candidate to begin your search with.

Search engines are generally either keyword based in which case they try to find if your queried words appear in the documents in its indexed database or concept based (meaning they try to infer what you are actually looking for from the combination and context of your keywords).
You may also try using Alta vista, Excite or Ask Jeeves as these not only search for the queried documents in their indexed database but also words with similar meaning.
Specialized mining search engines are probably the best bet though. These are specific to the mining industry and have sufficiently large databases to answer most of your queries. Two of the most popular in use today are minelinks and geoindex. Searchenginewatch.com provides a very handy chart to help you decide which search engine to use.

Meta search engines:

Meta search engines are the grand daddies of search engines, being entities that search the search engines. Some of them simply return the results from the individual search engines, leaving you shouting for mercy by the sheer volume of documents returned. Others however adopt a method of clustering where they remove redundant documents leaving you with a more relevant, smaller set of documents.
WebCrawler, Metaspider and Dogpile are all examples of meta search engines. For focused search on mining topics they may turn out to be too cumbersome for use.

Directories:

Directories are links categorized by topic. Experts in a particular subject with a basic knowledge of taxonomy are preferred for compilation of directories. Since experts usually compile directories, the Web pages available are of higher relevance than in an indexed search engine database. The coverage however is usually only a fraction of the search engine crawled area. One of the best examples of a directory is your own About. COM! Categorized by people who have worked in the industry for quite a while, you are very unlikely to leave an About site without finding what you came for. You may also want to try the Dmoz and minelinks directories for mining-related information.
This list of major Web directories will come in handy.

Invisible Web:

The World Wide Web (WWW) as we know it today forms only a fraction of the Internet. There is a lot of information ‘out there” which is beyond most of the traditional search engine spiders’ reach. This portion of the Internet, which is invisible to the spiders, has come to be known as the Invisible Web. Getting to the Invisible Web requires us to dig a little more into the Internet. This is a little difficult to do on your own, but tools like and engines like Northernlight come to our rescue by making available significant parts of the Invisible Web for search.
For a detailed explanation on the nuances of the Invisible Web, please refer to About. COM.

Discussion groups:

Discussion groups are just that – groups on the net which hold discussions on a particular topic. The most popular group on the Web today is Google Groups (formerly Deja news). The groups do not place any restriction on entry or exit and allow anyone to post a question/ observation as well as refer previous postings. A look at the geology and mining groups should provide a good idea as to the utility of this resource. The entry of the mining browser, a desktop software application, has brought a world of mining company information to your fingertips. Searching for global company-specific material has never been so easy, the only jarring note being that this tool comes with a monthly subscription of about US$ 20 a month. But then, into each life, some rain must fall!
Thus, there are many tools to help us get to the information we require on the Web. Each tool has its own purpose and speaks its own language. For making best use of the Web, it is necessary that we spend some time honing our searching skills. After all even looking for that needle in the proverbial haystack is not all that difficult if we have the right metal detector to guide us!