The Invisible Web

By Deane Barker on October 3, 2005

Seek and Ye Shall Find: This is an interesting article about “the invisible Web” — large repositories of data that search engines don’t pick up on.

For instance, Web-enabled databases that use POST forms to search. Spiders don’t do POST requests — if there are no direct URLs to the content, they skim right by it. Here’s a good trick:

[…] just put the word “database” in your query and more often than not you’ll come back lucky. This Penn State database has more than enough searchable information regarding warthogs than I’ll ever need, plus, it’s an academic, accredited, footnote-able institution.

What this will do is get you to the front door of the database, from which can walk through and look around, whereas the search engine spiders can’t.

I don’t have a current problem on which I can test this theory, but it sounds reasonable.