Web Spidering For You And Me: How To Find Those Pages That Others Can’t Or Won’t Find

This java application that I am going to start using is called ItSucks and is located here. I am going to be testing this little java app against my custom written in “C” spiders on my Linux machine. I will be using these spiders to find specialized Web pages that I have a hard time finding using the standard search engines like Google, Google Scholar, and Yahoo! I often search through university on line archives to find special articles that are often overlooked or orphaned. These articles are often missed because there are not a lot of connections to them, But the information included in these on-line papers can be invaluable in my work.

The results are in and I guess I am just not cut out to use complex regular expressions that are required to be used by this because I could not get the same results from my Linux C++ custom coded spiders, and this application.  So I am going to leave this app alone for right now until I can figure out how to the regular expression.  This does not mean that program does not work, it only means that I have some kind of mental block in using this application.
[tags]MicroFace, Open Source, Web Spider[/tags]