Designing Websites For Humans In A World Of Robots!!

Share |

In this day and age, it can be easy to forget the basics of why your website is online. Crawlers/Robots, they come, they go, but they never pay. Thats where your visitors come in.With the ever increasing number of web pages & documents available on the internet, it has become difficult to find information fast and without having hundreds of advertisements thrown into our faces (most of which will have no relevancy to the information we are seeking). There is quite simply no realistic method to finding material on the internet other than using search engines. At this point, everyone should realize that search engines use "robots" in order to "crawl" through the internet and collect web pages & other documents. The search engine will then use these documents to make up the engines "index" or "database". This in itself is not a problem, but to every action there is an equal & opposite reaction (Newton's 3rd law of motion!).
With this expansion of information on the web, which has driven more people to use search engines on a daily basis, it has become a requirement for the search engines to become more active in order to keep their database up to date. This means crawling more web pages at a greater frequency. Website owners have indeed noticed this increase in activity, and they have not stared at it blankly in the face, they have reacted. They now realize that these search engines are producing significant percentages of their traffic (up to 90% in some cases). So what to do'
Again, with the expansion of the web, there has also come more competition in essentially every industry, from computers, travel, food, right down to buying pets online. This competition is healthy in that it has pushed prices lower, but this very same competition has indirectly lowered the overall satisfaction level of website visitors