Google begrudgingly adjusts news crawling for paid publishers

With a group of blog posts this morning, one of which literally said paid content was probably not a good idea anyway, Google announced it was making an adjustment to its First Click Free aggregation scheme for news publishers. The adjustment to the service, first rolled out in 2007, will now enable readers to sample more content before subscribing through the publisher's own portal.

"While we're happy to see that a number of publishers are already using First Click Free, we've found that some who might try it are worried about people abusing the spirit of First Click Free to access almost all of their content," reads this morning's post to Google's Webmaster Central blog. "As most users are generally happy to be able to access just a few pages from these premium content providers, we've decided to allow publishers to limit the number of accesses under the First Click Free policy to five free accesses per user each day."

Google charges nothing for its search, as everyone knows; and the whole point of search is to let users sample content throughout the Web. Some of that content is for publishers' paid subscribers. But if all of that content were put behind a pay wall, as publishers are free to do, then Google can't index it, and users can't find it. So in 2007, the search provider established First Click Free as a way to enable publishers to let readers see a sample in Google's index, and click through to read the page associated with that sample, for free. Should the reader click elsewhere in the site, he's directed to that site's signup page.

That sounded nice on paper, at least at first. The problem was that readers found a simple way around being directed to the sign-up page: simply hitting the Back button, returning to Google, and finding another First Click Free page from the same index. Publishers got around that problem by instituting a method that Google calls cloaking. Since a site "knows" when it's being crawled by Google or another index, it has the power of treating the crawler like a pre-paid subscriber, giving it free reign to index everything. When the user clicks on an indexed page, the reader goes directly to the subscriber sign-up sheet.

That sounded nice on paper as well. The problem there, as Google put it this morning, was a security issue: If a Web site can make nice with Google while directing the user somewhere else, that other location could be a malicious misdirection.

The changes to First Click Free today don't actually solve that problem, although this morning, Senior Business Product Manager Josh Cohen cited the need to avoid deceiving the reader as the principal reason for today's adjustments. He did not cite rumors of News Corp. Chairman Rupert Murdoch making an exclusive indexing deal with Microsoft's Bing, although Murdoch's appearance yesterday at an FTC workshop certainly brought the issue to a head.

"If you're a Google user, this [adjustment] means that you may start to see a registration page after you've clicked through to more than five articles on the website of a publisher using First Click Free in a day. We think this approach still protects the typical user from cloaking, while allowing publishers to focus on potential subscribers who are accessing a lot of their content on a regular basis."

Google will also be instituting a system, Cohen said, capable of crawling the first few paragraphs of a story rather than the entire page, as long as the publisher makes those few paragraphs available in a preview page to regular users as well. This doesn't actually differentiate itself very much from what Google already does for non-cloaking publishers that produce preview pages, according to Cohen's description, except that now, preview page content will be labeled in Google's indexes.

"Because the preview page is identical for both users and the crawlers, it's not cloaking," Cohen wrote. "We will then label such stories as 'subscription' in Google News."

But Cohen then closed with a parting kick in the teeth for any publisher that insists on charging for content, and bothering to use the First Click Free program: "Paid content may not do as well as free options, but that is not a decision we make based on whether or not it's free. It's simply based on the popularity of the content with users and other sites that link to it."