Newsletter #87 The Deep Web

Created on 29/04/04

A special hello to all new subscribers. This article is on the Web at this address.

The last newsletter (25th March, 2004) was "Manage Your Own Site?"

You probably love using Google. But you are also probably unaware how little the most comprehensive search engines actually index. Google recently (February this year) announced that it indexed nearly 4.3 billion Webpages but the "Deep Web" is estimated to be at least 50 billion pages.

So what is the Deep Web? All the major search engines survey the Web by dispatching simple programs known as spiders, crawlers, or robots to trace their way through the millions of chains of hyperlinks that tie Webpages and therefore Websites together. Areas like library catalogs, classified ads, patent filings, flight schedules, product catalogs and genetic research data are often protected by registration forms, the use of cookies and the fact that much of the data is dynamically generated. Most of the information within these databases is therefore "hidden" or "invisible" to the search engines. This content is known as the Deep Web.

Last month Search Engine Yahoo! announced what they called a Content Acquisition Program to get commercial firms to provide their "hidden" data. The race is on to unleash the power of this vast sprawl of information with Yahoo!, Google, Microsoft and even IBM the major players. The consequences will be huge, perhaps even inconceivable - Google in its current form has already changed the way we obtain information - what will the future hold?

----------------------------------------------------------------------------------------------------------------

DOES YOUR WEBSITE LOOK UNPROFESSIONAL? Netinsites are experts at revamping sites that look dated or old-fashioned. We have the skills to overhaul your site so it looks professional, all for a very cost-effective price. Just reply to this email or go to this page.

--------------------------------------------------------------------------------------------------------------

Some of the things that may happen are:

• Information will be cheaper, far cheaper. In the United States scientific, technology and medical publishing is a $US 11 billion a year industry, according to Salon. It's not hard to imagine that industry being severely restructured previously restricted articles find their way on to the Web.
  
  
• The barriers between internal company data and public data will be far more porous. IT departments will not need to bother with internal search mechanisms - employees will just use the Web.
  
  
• Because the search robots will become more intelligent they will index data that companies and individual don't want in the private domain. Hackers may make a company's data available to the search robots as a way of sabotaging their operations. Privacy concerns will abound, and lawsuits will multiply.
  
  
• Because databased data is likely to be better structured and uniform, deep Web search engines will present more useful views of information that can be manipulated into endless combinations. The opportunities for data mining and analysis will be immense, throwing up numerous business possibilities.
  
  
• Search engine results will be less one-dimensional. There are already several initiatives that present data in a more intuitive three-dimensional manner. See Vivisimo at http://vivisimo.com/ and Grokker at http://www.groxis.com/service/grok for examples. These new ways of showing search results will be even more important when the deep Web becomes more accessible.

There is no doubt that the indexing of the deep Web will trigger a series of massive disruptions in the information economy. If you can see it coming, you can take advantage of the forces that are unleashed.

Newbies

Power Users

Windows XP makes it easy to locate your cursor by pushing the CTRL key. To enable the feature, go to 'Start', 'Control Panel' and click the Mouse icon. Select the Mouse Properties and Pointer Options tabs and check "Show location of pointer when I press the CTRL key".

For previous tips visit the Power Hot Tips page.

Are you sick of posting URLs in emails only to have it break when sent causing the recipient to have to cut and paste it back together? TinyURL.com is a free site that converts huge strings of characters into more manageable snippets. We converted the link to a Wired Article "A Web of Electronic Denial" from this: http://www.wired.com/news/business/0,1367,63240,00.html?tw=wn_tophead_1 to this:
http://tinyurl.com/2kvgw

If you're looking for some great images or icons check out Pixelgirl Presents. A site owned by Morfia Design is lovely to look at and has some great content. http://www.pixelgirlpresents.com/index.html

No new articles this month.

To search through all the articles in our online database use the search function on any page of our Website.

"The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' (I found it!) but 'That's funny...'" Isaac Asimov

You can see our other Quick Quotes on the front page of www.netinsites.com; just refresh the page to see another one randomly selected from our database. Great for presentations or times when you want to appear to be a techno-dude(ss)!

To access previous newsletters visit our newsletter archive.

If you want to subscribe to this newsletter just visit www.netinsites.com and use the box at the top right.

Best wishes

Alex Garden

Internet Strategy | Website Design | Website Promotion | Web Text Messaging | Email Newsletters | Online Sports Games | Content Management