Cloaking

Cloaking is a black hat search engine optimization (SEO) technique in which the content presented to the search engine spider is different to that presented to the users' browser. This is done by delivering content based on the IP addresses or the User-Agent [] header of the user requesting the page. When a user is identified as a search engine spider, a server-side script delivers a different version of the web page, one that contains content not present on the visible page. The purpose of cloaking is to deceive search engines so they display the page when it would not otherwise be displayed.

As of 2006, better methods of accessibility, including progressive enhancement are available, so cloaking is not considered necessary. Cloaking is often used as a spamdexing technique, to try to trick search engines into giving the relevant site a higher ranking; it can also be used to trick search engine users into visiting a site based on the search engine description which site turns out to have substantially different, or even pornographic content. For this reason, major search engines consider cloaking for deception to be a violation of their guidelines, and therefore, they delist sites when deceptive cloaking is reported.

Cloaking is a form of the doorway page technique.

A similar technique is also used on the Open Directory Project web directory. It differs in several ways from search engine cloaking:

• It is intended to fool human editors, rather than computer search engine spiders.
• The decision to cloak or not is often based upon the HTTP referrer, the user agent or the visitor's IP; but more advanced techniques can be also based upon the client's behaviour analysis after a few page requests: the raw quantity, the sorting of, and latency between subsequent HTTP requests sent to a website's pages, plus the presence of a check for robots.txt file, are some of the parameters in which search engines spiders differ heavily from a natural user behaviour. The referrer tells the URL of the page on which a user clicked a link to get to the page. Some cloakers will give the fake page to anyone who comes from a web directory website, since directory editors will usually examine sites by clicking on links that appear on a directory web page. Other cloakers give the fake page to everyone except those coming from a major search engine; this makes it harder to detect cloaking, while not costing them many visitors, since most people find websites by using a search engine.

The Black Hat Perspective

Increasingly, for a page without natural popularity due to compelling or rewarding content to rank well in the search engines, webmasters may be tempted to design pages solely for the search engines. This results in pages with too many keywords and other factors that might be search engine "friendly", but make the pages difficult for actual visitors to consume. As such, black hat SEO practitioners consider cloaking to be an important technique to allow webmasters to split their efforts and separately target the search engine spiders and human visitors.

In September 2007, Ralph Tegtmeier & Ed Purkiss coined the term "mosaic cloaking" whereby dynamic pages are constructed as tiles of content and only portions of the pages, javascript and CSS are changed, simultaneously decreasing the contrast between the cloaked page and the "friendly" page while increasing the capability for targeted delivery of content to various spiders and human visitors.

Cloaking versus IP Delivery

IP delivery can be considered a more benign variation of cloaking, where different content is served based upon the requester's IP address. With cloaking, search engines and people never see the other's pages, whereas, with other uses of IP delivery, both search engines and people can see the same pages. This technique is sometimes used by graphics-heavy sites that have little textual content for spiders to analyze.

One use of IP delivery is to determine the requestor's location, and deliver content specifically written for that country. This use isn't necessarily cloaking. For instance, Google uses IP delivery for AdWords and AdSense advertising programs in order to target users in different geographic locations.

As a mean of determining the language(s) in which to provide content, IP delivery is a crude and unreliable method; many countries and regions are multi-lingual, or the requestor may be a foreign national. A better method of content negotiation is to examine the client's Accept-Language HTTP header.

As of 2006, many well-known and well respected sites have taken up IP delivery to personalise content for their regular customers. In fact, many of the top 1000 sites, including household names like Amazon (amazon.com), actively use IP delivery. None of these have been banned from search engines because their intention is not deceptive.

In 2007 Yahoo! was outed for cloaking content on its Yahoo! Autos page

See also

• Search Engine Spam: Doorway page, Keyword stuffing, Link farms, Spamdexing, URL redirection
• Technology: Content negotiation, Geo targeting

References

• A Walkthrough of some Basic Cloaking Techniques - A blackhat's walkthrough of how to detect various search engine spiders.
• Search Engine Cloaking: The Controversy Continues, a response to In Defense of Search Engine Cloaking
• Baoning Wu and Brian D. Davison: " Cloaking and Redirection: A Preliminary Study". Workshop on Adversarial Information Retrieval on the Web, Chiba, Japan, 2005.
• Cloaking Implementation in PHP Cloaking implementation written in the PHP programming language.
• Cloaking Detector Tool that is capable to see whether a web page is doing user-agent cloaking or not.

Mosaic Cloaking :: Advanced Cloaking Technique