How did Google become the internet’s censor and master manipulator, blocking access to millions of websites?

June 22, 2016

The New Censorship

June 22, 2016, at 9:00 a.m.

Google, Inc., isn't just the world's biggest purveyor of information; it is also the world's biggest censor.

The company maintains at least nine different blacklists that impact our lives, generally without input or authority from any outside advisory group, industry association or government agency. Google is not the only company suppressing content on the internet. Reddit has frequently been accused of banning postings on specific topics, and a recent report suggests that Facebook has been deleting conservative news stories from its newsfeed, a practice that might have a significant effect on public opinion – even on voting. Google, though, is currently the biggest bully on the block.

When Google's employees or algorithms decide to block our access to information about a news item, political candidate or business, opinions and votes can shift, reputations can be ruined and businesses can crash and burn. Because online censorship is entirely unregulated at the moment, victims have little or no recourse when they have been harmed.

Eventually, authorities will almost certainly have to step in, just as they did when credit bureaus were regulated in 1970. The alternative would be to allow a large corporation to wield an especially destructive kind of power that should be exercised with great restraint and should belong only to the public: the power to shame or exclude.

If Google were just another mom-and-pop shop with a sign saying "we reserve the right to refuse service to anyone," that would be one thing. But as the golden gateway to all knowledge, Google has rapidly become an essential in people's lives – nearly as essential as air or water. We don't let public utilities make arbitrary and secretive decisions about denying people services; we shouldn't let Google do so either.

Big social media companies like Facebook and Google have too much power to manipulate elections.

Let's start with the most trivial blacklist and work our way up. I'll save the biggest and baddest – one the public knows virtually nothing about but that gives Google an almost obscene amount of power over our economic well-being – until last.

1. The autocomplete blacklist. This is a list of words and phrases that are excluded from the autocomplete feature in Google's search bar. The search bar instantly suggests multiple search options when you type words such as "democracy" or "watermelon," but it freezes when you type profanities, and, at times, it has frozen when people typed words like "torrent," "bisexual" and "penis." At this writing, it's freezing when I type "clitoris." The autocomplete blacklist can also be used to protect or discredit political candidates. As recently reported, at the moment autocomplete shows you "Ted" (for former GOP presidential candidate Ted Cruz) when you type "lying," but it will not show you "Hillary" when you type "crooked" – not even, on my computer, anyway, when you type "crooked hill." (The nicknames for Clinton and Cruz coined by Donald Trump, of course.) If you add the "a," so you've got "crooked hilla," you get the very odd suggestion "crooked Hillary Bernie." When you type "crooked" on Bing, "crooked Hillary" pops up instantly. Google's list of forbidden terms varies by region and individual, so "clitoris" might work for you. (Can you resist checking?)

2. The Google Maps blacklist. This list is a little more creepy, and if you are concerned about your privacy, it might be a good list to be on. The cameras of Google Earth and Google Maps have photographed your home for all to see. If you don't like that, "just move," Google's former CEO Eric Schmidt said. Google also maintains a list of properties it either blacks out or blurs out in its images. Some are probably military installations, some the residences of wealthy people, and some – well, who knows? Martian pre-invasion enclaves? Google doesn't say.

3. The YouTube blacklist. YouTube, which is owned by Google, allows users to flag inappropriate videos, at which point Google censors weigh in and sometimes remove them, but not, according to a recent report by Gizmodo, with any great consistency – except perhaps when it comes to politics. Consistent with the company's strong and open support for liberal political candidates, Google employees seem far more apt to ban politically conservative videos than liberal ones. In December 2015, singer Susan Bartholomew sued YouTube for removing her openly pro-life music video, but I can find no instances of pro-choice music being removed. YouTube also sometimes acquiesces to the censorship demands of foreign governments. Most recently, in return for overturning a three-year ban on YouTube in Pakistan, it agreed to allow Pakistan's government to determine which videos it can and cannot post.

4. The Google account blacklist. A couple of years ago, Google consolidated a number of its products – Gmail, Google Docs, Google+, YouTube, Google Wallet and others – so you can access all of them through your one Google account. If you somehow violate Google's vague and intimidating terms of service agreement, you will join the ever-growing list of people who are shut out of their accounts, which means you'll lose access to all of these interconnected products. Because virtually no one has ever read this lengthy, legalistic agreement, however, people are shocked when they're shut out, in part because Google reserves the right to "stop providing Services to you … at any time." And because Google, one of the largest and richest companies in the world, has no customer service department, getting reinstated can be difficult. (Given, however, that all of these services gather personal information about you to sell to advertisers, losing one's Google account has been judged by some to be a blessing in disguise.)

5. The Google News blacklist. If a librarian were caught trashing all the liberal newspapers before people could read them, he or she might get in a heap o' trouble. What happens when most of the librarians in the world have been replaced by a single company? Google is now the largest news aggregator in the world, tracking tens of thousands of news sources in more than thirty languages and recently adding thousands of small, local news sources to its inventory. It also selectively bans news sources as it pleases. In 2006, Google was accused of excluding conservative news sources that generated stories critical of Islam, and the company has also been accused of banning individual columnists and competing companies from its news feed. In December 2014, facing a new law in Spain that would have charged Google for scraping content from Spanish news sources (which, after all, have to pay to prepare their news), Google suddenly withdrew its news service from Spain, which led to an immediate drop in traffic to Spanish new stories. That drop in traffic is the problem: When a large aggregator bans you from its service, fewer people find your news stories, which means opinions will shift away from those you support. Selective blacklisting of news sources is a powerful way of promoting a political, religious or moral agenda, with no one the wiser.

6. The Google AdWords blacklist. Now things get creepier. More than 70 percent of Google's $80 billion in annual revenue comes from its AdWords advertising service, which it implemented in 2000 by infringing on a similar system already patented by Overture Services. The way it works is simple: Businesses worldwide bid on the right to use certain keywords in short text ads that link to their websites (those text ads are the AdWords); when people click on the links, those businesses pay Google. These ads appear on Google.com and other Google websites and are also interwoven into the content of more than a million non-Google websites – Google's "Display Network." The problem here is that if a Google executive decides your business or industry doesn't meet its moral standards, it bans you from AdWords; these days, with Google's reach so large, that can quickly put you out of business. In 2011, Google blacklisted an Irish political group that defended sex workers but which did not provide them; after a protest, the company eventually backed down.

In May 2016, Google blacklisted an entire industry – companies providing high-interest "payday" loans. As always, the company billed this dramatic move as an exercise in social responsibility, failing to note that it is a major investor in LendUp.com, which is in the same industry; if Google fails to blacklist LendUp (it's too early to tell), the industry ban might turn out to have been more of an anticompetitive move than one of conscience. That kind of hypocrisy has turned up before in AdWords activities. Whereas Google takes a moral stand, for example, in banning ads from companies promising quick weight loss, in 2011, Google forfeited a whopping $500 million to the U.S. Justice Department for having knowingly allowed Canadian drug companies to sell drugs illegally in the U.S. for years through the AdWords system, and several state attorneys general believe that Google has continued to engage in similar practices since 2011; investigations are ongoing.

7. The Google AdSense blacklist. If your website has been approved by AdWords, you are eligible to sign up for Google AdSense, a system in which Google places ads for various products and services on your website. When people click on those ads, Google pays you. If you are good at driving traffic to your website, you can make millions of dollars a year running AdSense ads – all without having any products or services of your own. Meanwhile, Google makes a net profit by charging the companies behind the ads for bringing them customers; this accounts for about 18 percent of Google's income. Here, too, there is scandal: In April 2014, in two posts on PasteBin.com, someone claiming to be a former Google employee working in their AdSense department alleged the department engaged in a regular practice of dumping AdSense customers just before Google was scheduled to pay them. To this day, no one knows whether the person behind the posts was legit, but one thing is clear: Since that time, real lawsuits filed by real companies have, according to WebProNews, been "piling up" against Google, alleging the companies were unaccountably dumped at the last minute by AdSense just before large payments were due, in some cases payments as high as $500,000.

8. The search engine blacklist. Google's ubiquitous search engine has indeed become the gateway to virtually all information, handling 90 percent of search in most countries. It dominates search because its index is so large: Google indexes more than 45 billion web pages; its next-biggest competitor, Microsoft's Bing, indexes a mere 14 billion, which helps to explain the poor quality of Bing's search results.

Google's dominance in search is why businesses large and small live in constant "fear of Google," as Mathias Dopfner, CEO of Axel Springer, the largest publishing conglomerate in Europe, put it in an open letter to Eric Schmidt in 2014. According to Dopfner, when Google made one of its frequent adjustments to its search algorithm, one of his company's subsidiaries dropped dramatically in the search rankings and lost 70 percent of its traffic within a few days. Even worse than the vagaries of the adjustments, however, are the dire consequences that follow when Google employees somehow conclude you have violated their "guidelines": You either get banished to the rarely visited Netherlands of search pages beyond the first page (90 percent of all clicks go to links on that first page) or completely removed from the index. In 2011, Google took a "manual action" of a "corrective" nature against retailer J.C. Penney – punishment for Penney's alleged use of a legal SEO technique called "link building" that many companies employ to try to boost their rankings in Google's search results. Penney was demoted 60 positions or more in the rankings.

Search ranking manipulations of this sort don't just ruin businesses; they also affect people's opinions, attitudes, beliefs and behavior, as my research on the Search Engine Manipulation Effect has demonstrated. Fortunately, definitive information about Google's punishment programs is likely to turn up over the next year or two thanks to legal challenges the company is facing. In 2014, a Florida company called e-Ventures Worldwide filed a lawsuit against Google for "completely removing almost every website" associated with the company from its search rankings. When the company's lawyers tried to get internal documents relevant to Google's actions though typical litigation discovery procedures, Google refused to comply. In July 2015, a judge ruled that Google had to honor e-Ventures' discovery requests, and that case is now moving forward. More significantly, in April 2016, the Fifth Circuit Court of Appeals ruled that the attorney general of Mississippi – supported in his efforts by the attorneys general of 40 other states – has the right to proceed with broad discovery requests in his own investigations into Google's secretive and often arbitrary practices.

This brings me, at last, to the biggest and potentially most dangerous of Google's blacklists – which Google's calls its "quarantine" list.

9. The quarantine list. To get a sense of the scale of this list, I find it helpful to think about an old movie – the classic 1951 film "The Day the Earth Stood Still," which starred a huge metal robot named Gort. He had laser-weapon eyes, zapped terrified humans into oblivion and had the power to destroy the world. Klaatu, Gort's alien master, was trying to deliver an important message to earthlings, but they kept shooting him before he could. Finally, to get the world's attention, Klaatu demonstrated the enormous power of the alien races he represented by shutting down – at noon New York time – all of the electricity on earth for exactly 30 minutes. The earth stood still.

Substitute "ogle" for "rt," and you get "Google," which is every bit as powerful as Gort but with a much better public relations department – so good, in fact, that you are probably unaware that on Jan. 31, 2009, Google blocked access to virtually the entire internet. And, as if not to be outdone by a 1951 science fiction move, it did so for 40 minutes. Impossible, you say. Why would do-no-evil Google do such an apocalyptic thing, and, for that matter, how, technically, could a single company block access to more than 100 million websites?The answer has to do with the dark and murky world of website blacklists – ever-changing lists of websites that contain malicious software that might infect or damage people's computers. There are many such lists – even tools, such as blacklistalert.org, that scan multiple blacklists to see if your IP address is on any of them. Some lists are kind of mickey-mouse – repositories where people submit the names or IP addresses of suspect sites. Others, usually maintained by security companies that help protect other companies, are more high-tech, relying on "crawlers" – computer programs that continuously comb the internet.

But the best and longest list of suspect websites is Google's, launched in May 2007. Because Google is crawling the web more extensively than anyone else, it is also in the best position to find malicious websites. In 2012, Google acknowledged that each and every day it adds about 9,500 new websites to its quarantine list and displays malware warnings on the answers it gives to between 12 and 14 million search queries. It won't reveal the exact number of websites on the list, but it is certainly in the millions on any given day.

In 2011, Google blocked an entire subdomain, co.cc, which alone contained 11 million websites, justifying its action by claiming that most of the websites in that domain appeared to be "spammy." According to Matt Cutts, still the leader of Google's web spam team, the company "reserves the right" to take such action when it deems it necessary. (The right? Who gave Google that right?)

And that's nothing: According to The Guardian, on Saturday, Jan. 31, 2009, at 2:40 pm GMT, Google blocked the entire internet for those impressive 40 minutes, supposedly, said the company, because of "human error" by its employees. It would have been 6:40 am in Mountain View, California, where Google is headquartered. Was this time chosen because it is one of the few hours of the week when all of the world's stock markets are closed? Could this have been another of the many pranks for which Google employees are so famous? In 2008, Google invited the public to submit applications to join the "first permanent human colony on Mars." Sorry, Marsophiles; it was just a prank.

When Google's search engine shows you a search result for a site it has quarantined, you see warnings such as, "The site ahead contains malware" or "This site may harm your computer" on the search result. That's useful information if that website actually contains malware, either because the website was set up by bad guys or because a legitimate site was infected with malware by hackers. But Google's crawlers often make mistakes, blacklisting websites that have merely been "hijacked," which means the website itself isn't dangerous but merely that accessing it through the search engine will forward you to a malicious site. My own website, http://drrobertepstein.com, was hijacked in this way in early 2012. Accessing the website directly wasn't dangerous, but trying to access it through the Google search engine forwarded users to a malicious website in Nigeria. When this happens, Google not only warns you about the infected website on its search engine (which makes sense), it also blocks you from accessing the website directly through multiple browsers – even non-Google browsers. (Hmm. Now that's odd. I'll get back to that point shortly.)

The mistakes are just one problem. The bigger problem is that even though it takes only a fraction of a second for a crawler to list you, after your site has been cleaned up Google's crawlers sometimes take days or even weeks to delist you – long enough to threaten the existence of some businesses. This is quite bizarre considering how rapidly automated online systems operate these days. Within seconds after you pay for a plane ticket online, your seat is booked, your credit card is charged, your receipt is displayed and a confirmation email shows up in your inbox – a complex series of events involving multiple computers controlled by at least three or four separate companies. But when you inform Google's automated blacklist system that your website is now clean, you are simply advised to check back occasionally to see if any action has been taken. To get delisted after your website has been repaired, you either have to struggle with the company's online Webmaster tools, which are far from friendly, or you have to hire a security expert to do so – typically for a fee ranging between $1,000 and $10,000. No expert, however, can speed up the mysterious delisting process; the best he or she can do is set it in motion. So far, all I've told you is that Google's crawlers scan the internet, sometimes find what appear to be suspect websites and put those websites on a quarantine list. That information is then conveyed to users through the search engine. So far so good, except of course for the mistakes and the delisting problem; one might even say that Google is performing a public service, which is how some people who are familiar with the quarantine list defend it. But I also mentioned that Google somehow blocks people from accessing websites directly through multiple browsers. How on earth could it do that? How could Google block you when you are trying to access a website using Safari, an Apple product, or Firefox, a browser maintained by Mozilla, the self-proclaimed "nonprofit defender of the free and open internet"?

The key here is browsers. No browser maker wants to send you to a malicious website, and because Google has the best blacklist, major browsers such as Safari and Firefox – and Chrome, of course, Google's own browser, as well as browsers that load through Android, Google's mobile operating system – check Google's quarantine list before they send you to a website. (In November 2014, Mozilla announced it will no longer list Google as its default search engine, but it also disclosed that it will continue to rely on Google's quarantine list to screen users' search requests.)

If the site has been quarantined by Google, you see one of those big, scary images that say things like "Get me out of here!" or "Reported attack site!" At this point, given the default security settings on most browsers, most people will find it impossible to visit the site – but who would want to? If the site is not on Google's quarantine list, you are sent on your way.

OK, that explains how Google blocks you even when you're using a non-Google browser, but why do they block you? In other words, how does blocking you feed the ravenous advertising machine – the sine qua non of Google's existence?

Have you figured it out yet? The scam is as simple as it is brilliant: When a browser queries Google's quarantine list, it has just shared information with Google. With Chrome and Android, you are always giving up information to Google, but you are also doing so even if you are using non-Google browsers. That is where the money is – more information about search activity kindly provided by competing browser companies. How much information is shared will depend on the particular deal the browser company has with Google. In a maximum information deal, Google will learn the identity of the user; in a minimum information deal, Google will still learn which websites people want to visit – valuable data when one is in the business of ranking websites. Google can also charge fees for access to its quarantine list, of course, but that's not where the real gold is.

Chrome, Android, Firefox and Safari currently carry about 92 percent of all browser traffic in the U.S. – 74 percent worldwide – and these numbers are increasing. As of this writing, that means Google is regularly collecting information through its quarantine list from more than 2.5 billion people. Given the recent pact between Microsoft and Google, in coming months we might learn that Microsoft – both to save money and to improve its services – has also started using Google's quarantine list in place of its own much smaller list; this would further increase the volume of information Google is receiving.

To put this another way, Google has grown, and is still growing, on the backs of some of its competitors, with end users oblivious to Google's antics – as usual. It is yet another example of what I have called "Google's Dance" – the remarkable way in which Google puts a false and friendly public face on activities that serve only one purpose for the company: increasing profit. On the surface, Google's quarantine list is yet another way Google helps us, free of charge, breeze through our day safe and well-informed. Beneath the surface, that list is yet another way Google accumulates more information about us to sell to advertisers.

You may disagree, but in my view Google's blacklisting practices put the company into the role of thuggish internet cop – a role that was never authorized by any government, nonprofit organization or industry association. It is as if the biggest bully in town suddenly put on a badge and started patrolling, shuttering businesses as it pleased, while also secretly peeping into windows, taking photos and selling them to the highest bidder.

Consider: Heading into the holiday season in late 2013, an online handbag business suffered a 50 percent drop in business because of blacklisting. In 2009, it took an eco-friendly pest control company 60 days to leap the hurdles required to remove Google's warnings, long enough to nearly go broke. And sometimes the blacklisting process appears to be personal: In May 2013, the highly opinionated PC Magazine columnist John Dvorak wondered "When Did Google Become the Internet Police?" after both his website and podcast site were blacklisted. He also ran into the delisting problem: "It's funny," he wrote, "how the site can be blacklisted in a millisecond by an analysis but I have to wait forever to get cleared by the same analysis doing the same scan. Why is that?"

Could Google really be arrogant enough to mess with a prominent journalist? According to CNN, in 2005 Google "blacklisted all CNET reporters for a year after the popular technology news website published personal information about one of Google's founders" – Eric Schmidt – "in a story about growing privacy concerns." The company declined to comment on CNN's story.