I was interviewed (in my home office) and featured in a news segment on KTVU-2 news last night; you can view the segment here (but you’ll need to wait through a 15-second commercial before seeing the 2-minute, 15 second segment).

I was also interviewed today (June 30) by KPIX-5 for broadcast on tonight’s news.

Google always strives to deliver relevant and useful links in its search results.  All search engines must deal with lots of unethical people who seek an unfair advantage. One specific strategy that has exploded in recent years is called “content farming,” which means creating web pages that include text content which superficially appears relevant and useful for specific search terms — but which isn’t useful to anyone.

We’ve all had the experience of searching for a specific phrase on Google, and clicking on a link that appears to provide the exact information we wanted — only to end up at a web page that doesn’t provide any information, but instead urges us to click on paid links and advertisements.  Many “content farmers” simply merge the relevant keywords together with snippets of text copied from other web sites which rank well for that search phrase.

Several years ago, Google’s algorithms added penalties for “duplicate content,” so that “copied” web sites would be downgraded or excluded from search results. Many “content farmers” responded by using a variety of automated “translation” strategies so that the text wasn’t identical to the source which was being copied (most simply, by substituting synonyms for randomly-selected words in an article, or by automatically translating words, phrases, or entire articles into another language and then translating them back).  When Google tweaked its algorithms to counter this strategy, spammers began hiring writers in third-world countries to write “original articles” that would appear relevant to Google’s algorithms (these articles are often incomprehensible to humans, and never useful to anyone).

It appears that for the Panda (Farmer) update, Google created a list of factors or elements which were commonly found on unsatisfactory pages in search results (and not on useful pages), but which weren’t already recognized by its algorithms.  These are measurable, quantitative factors that correspond to users’ perceptions of quality.  Google also manually downgraded some specific “content farmers” whose web sites provided little or no value for consumers.  After the upgrade, Google also made adjustments to restore the rankings of a few publishers who had been wrongly downgraded by the new algorithms).

The new algorithm doesn’t seem to have affected my sites; traffic from Google to my newest site has continued to grow by about 30% per week.  (Any increased traffic after the upgrade wouldn’t mean that Google “likes” my web pages more than last week.  In amusement-park terms, nobody gets to skip ahead in the line, but instead some naughty people were removed from the line and ejected from the park, though we expect that they’ll soon sneak back in.)

Ironically, one “side effect” from Google’s update (for me) has been an increase in “blog comment spam” (spammers post fake comments with links to unrelated web sites).  My blog sites probably rank better, this week, for the search phrases that spammers use to identify “blogs which accept comments.”  (These spam comments are not visible to my site’s visitors.)

Google’s official blog post about the “Farmer”/”Panda” updates:

• Google’s Blog
• Matt Cutts’ blog

Other discussion threads and articles about the Google “Panda” algorithm update:

• ABestWeb-1 and ABestWeb-2
• WebProNews
• TechMeme
• SearchEngineLand-1 and SearchEngineLand-2
• SearchEngineJournal
• Wired

Some lists of impacted sites : Sistrix and SearchEngineLand and Quora

Added March 6: Two more articles about the Farmer Update:

• SearchMetrics: “[B]ounce rates, visit duration, and social reach [are obviously factors]…. [P]ages that are genuinely visually attractive to a user … will be spared by the Farmer Update. Meaning that ranking is going to come down to how a user values a page, as opposed to just what content is on it.”
• ReveNews: Noting the impact on User-Generated Content (UGC) sites: “In the more extreme cases, [the downgraded sites] actually incentivized users to generate this content. And in doing so, they’ve essentially eroded the value that content is supposed have because it’s user-generated.”

Some background: From mid-2002 through late 2004, I explored teaching as a career, but eventually decided that it just wasn’t a good fit for me. As a new teacher, I was  starved for resources, and couldn’t easily identify and obtain relevant and useful lesson plans or teaching guides that could reduce my workload.  (I ended up spending much of my time “re-inventing the wheel,” duplicating the work of thousands of other teachers.)

After I quit teaching in 2004, I created a “literature lesson plan” directory web site, as a part-time effort; after several hundred hours spent designing and implementing the site, I spent five to ten hours per month updating the site.  In July 2007, I sold that web site (and agreed not to compete for two years).  After my interest was revived in October, I learned that the earlier web site has stagnated; that discovery led me to do more research.

By mid-November, I was confident that I could create a new lesson-plan directory, this time for a full range of K-12 teaching and learning materials, in all content areas (integrated with state-specific content standards).

I spent some time designing an ontology and a preliminary database implementation, and soon I began work in earnest.  On January 1, 2011, I “launched” LessonIndex.com — which I believe is already the most useful and relevant directory of “Literature Lesson Plans” for K-12 teachers.

After gathering data from many sources and launching this version of the site as a “proof of concept,” I’m turning back to refine the ontology, database structure, and import scripts so I can expand the site to include resources for all K-12 content areas.

Some relevant links for my new site:

• Short Index page (with links to directory pages for about 200 literary works);
• Big Index page (with links to directory pages for each of 2,034 literary works);
• blog.LessonIndex.com, which provides update information, suggestions for publishers and merchants, and a list of the “Most Frequently-Taught Works” (works with the most resource links).

While my attention was focused on this new project, of course, I stopped posting to this blog.

A privacy policy is one of the most basic requirements for any professional web site. If a web site doesn’t have a conspicuous link to a privacy policy, it simply isn’t very professional, and is less trustworthy.

Consumers constantly evaluate the trustworthiness of web sites they visit, and they recognize that most professional, credible web sites include certain important links, including “privacy policy,” “about us,” and “contact us.” The presence of those links only provides a very subtle reassurance, but the absence of any of those links is a huge “red flag.”

Note that the actual content of the privacy policy isn’t nearly as significant as its presence. A web site could conceivably have a privacy policy that says, “we will track everything you do, and gather as much information as possible, and then sell your information to the worst people on the planet to use for any nefarious purpose.” While that would be quite unprofessional, very few consumers ever click to view a site’s privacy policy.

In March 2002, I posted information about the impact of privacy policies on visitors to an e-commerce merchant, in the I-Privacy Digest email discussion list. The most significant observation, drawn from a brief “slice” of one merchant’s log-file data:

“Of the 359 customers who actually placed orders, only 10 had reviewed the privacy/security policy during the same visit.” (Who Reads Privacy Policies?, in I-Privacy Digest #3, March 28, 2002.)

As an iPhone user, I sometimes find it annoying to navigate through web sites (including my own blog) which were designed with a standard desktop screen in mind.  On the other hand, I also sometimes find it annoying to experience a “crippled” version of a web site just because I’m using an iPhone.

This plug-in also gives users the option to disable the “mobile view” (just scroll to the footer and click on the toggle switch to disable (or re-enable) the mobile view).

There are many “settings options” for this plug-in, and I’m still experimenting with some. I’m currently using the free version, but will likely upgrade to the Pro version ($29) once I figure out how to use some of the features.

Let me know what you think.

Instead, I quickly recognized that some of the headlines were deceptive, as reporters sought to churn a minor clarification into a major story.

Here are some examples that I don’t think are objective (I’ve underlined wording I think is unfair):

• USA Today: “A divided Supreme Court scaled back the well-known Miranda right….”
• Los Angeles Times: “Supreme Court Loosens Miranda Rule” (headline), “The Supreme Court retreated from strict enforcement of the famous Miranda right on Tuesday….”
• Vanity Fair: “In the New America, You Must Invoke Your Own Miranda Rights”
• Detroit News: “Supreme Court narrows Miranda rights….”
• Newsday: “Supreme Court expands limits on Miranda rights”

Several commentators, writing before the Supreme Court issued its ruling, described the case as an attempt to “expand” or “clarify” Miranda, which required that before questioning, suspects must be advised of their Constitutional rights (to have an attorney, and not to incriminate themselves), and police must terminate questioning if a suspect invokes those rights.

But when the ruling was announced on June 1, most reporters and editors immediately transformed the “non-expansion” story into a “limitation” or “retreat” story.

There are two reasons why the media may have sought to magnify the ruling’s importance and impact.  First, the dissenting opinion was written by newly-appointed Justice Sotomayor. Second, when the case was briefed and argued, the U.S. government sought the ruling that the majority delivered — and the government was represented by current Solicitor General Elena Kagan, who is now President Obama’s nominee for appointment to the Supreme Court.

I assume that in the next few days, we’ll hear pundits proclaim that this case demonstrates that Ms. Kagan is more moderate (less liberal, more conservative) than Justice Sotomayor, which might make her appointment seem more acceptable to some Senators (and perhaps less acceptable to other Senators).  Undoubtedly, we will also hear some  pundits complain that Obama should withdraw Kagan’s nomination because she’s too conservative, and different pundits will claim that this case was a “smoke screen” intended all along to make Kagan appear more moderate than she “really is.”

In fairness, the “change” and “limitation” language was not the only “angle” that editors seized upon.  Many other articles instead emphasized the apparent contradiction in the court’s ruling that one must “speak up in order to assert the right to remain silent” (for example, the New York Times editorial, “Speaking Up to Stay Silent”).

__________________________

The actual issue in this case was whether a statement made by a suspect during interrogation, after remaining “largely silent” for nearly three hours, could be admitted at his trial, or should be excluded.

Mr. Thompkins was arrested for murder.  During his interrogation, he was advised of his right to remain silent, and he remained “almost completely silent and unresponsive”.  However, “At no point during the interrogation did Thompkins say that he wanted to remain silent, that he did not want to talk with the police, or that he wanted an attorney.”

After nearly three hours of an interrogation best described as a “monologue,” a detective asked Thompkins if he believed in God, and if he prayed; Thompkins answered “yes” to both questions. The detective then asked, “Do you pray to God to forgive you for shooting that boy down?” and the suspect answered “Yes.” Thompkins’ attorneys sought to have this statement excluded, arguing that by remaining mostly silent for nearly three hours, Thompkins had effectively communicated his intent to remain silent, and police should have ended the interview before then.

The “real issue” in the case, I think, was this: after police insisted that Thompkins read a portion of the Miranda rights from a card, and after police read all these rights to him, and after Thompkins refused to sign an acknowledgment that he “had been advised of and understood his rights,” police did not ask him if he expressly waived those rights.

The Supreme Court was presented with an opportunity to clarify the Miranda decision by deciding whether or not police were required to ask that question (and if so, whether they could insist that the defendant reply; or if the defendant did not reply, whether police should infer either a waiver or invocation of that right).

In a 5-4 decision, the Court ruled that police were not required to ask, nor were they required to infer from silence that a suspect was “invoking his right to remain silent,” which would end the interrogation.

In her dissent, Justice Sotomayor wrote:

The Court concludes today that a criminal suspect waives his right to remain silent if, after sitting tacit and uncommunicative through nearly three hours of police interrogation, he utters a few one-word responses. The Court also concludes that a suspect who wishes to guard his right to remain silent against such a finding of “waiver” must, counter-intuitively, speak—and must do so with sufficient precision to satisfy a clear-statement rule that construes ambiguity in favor of the police. Both propositions mark a substantial retreat from the protection against compelled self-incrimination that Miranda v. Arizona, 384 U. S. 436 (1966), has long provided during custodial interrogation.” (Berghuis v. Thompkins).

While I don’t agree with the majority’s decision, I don’t agree with Justice Sotomayor that this case represents a “substantial retreat,” and I don’t think the majority decision will “limit” or “change” existing law, nor will it change police procedure.

Most journalists adopted the dissent’s view that this decision was a significant change from existing law.  I find this a fascinating example of “pack-mentality media distortion.”

Daily newspapers were a slowly-sinking industry even before the internet.  Total daily newspaper circulation has dropped a few percent nearly every year for decades.  While internet access to news has certainly accelerated the circulation decline, the worst damage has been to advertising revenue.  Lost revenue led to reduced editorial staffs, which meant less local news reporting, and increased reliance on “wire service” news that is readily available online.

I’ve also noticed that in the race to cut costs, many newspapers are turning their newsstand distribution over to “low-bid” distribution companies, who hire the cheapest workers, who often fail to put new newspapers into many of the newsboxes, and who don’t repair or clean newsboxes that are damaged or are covered with graffiti.  In addition to reducing newsstand circulation, this reduces the opportunity to convert new residents into subscribers.

But then I found an interesting blog post, “How Early Newspaper-to-Web Technology Crippled News Industry’s Thinking” (Amy Gahran, Poynter.com) which cited a longer article “How early online newspaper production tools led the industry down the wrong path” (Robert Niles, Online Journalism Review).

These folks suggest that another “weak link” contributing to newspapers’ internet failures may have been the technology that was initially used to publish newspapers and to migrate newspaper content onto the Web.  The crude technology limited the ways that newspapers could use the internet, and the lack of any definite revenue stream from online publishing discouraged investment or independent thinking.

While newspapers remained mired in the philosophies and limitations embedded in their publishing software, internet newcomers experimented and found more successful ways to capture and distribute information online. While newspapers stuck with the idea that “we report the news, and people read it,” internet newcomers discovered that they could invite active participation by readers.

Whenever I think about the “lost opportunity,” I end up thinking about one of my professors in journalism school who was amazed (in 1981 or 1982) by the promise of computer technology: he suggested that newspapers would soon have the ability to print customized newspapers for individual subscribers (reducing printing and delivery costs, while increasing advertising opportunities).  The technology was there, but the newspaper industry moves very slowly, and that opportunity, too, was lost.

Instead, the sinking newspaper industry is grasping desperately to cut costs (fire the reporters who write what readers want; fire the editors who make sure the words make sense) and find new revenue sources (paste ads on the front page) — and those actions are turning away more readers.

Since this blog draws very little traffic, I doubt I’ll see any meaningful revenue, but I decided it would be interesting to see what ads are displayed by Google on my pages.

Please let me know if you see any objectionable ads (either post a comment, or email to Mark Welch at Mark Welch dot com).

The idea of adding a Google AdSense unit came to me in a very roundabout way: I was restoring some very old articles (see the “Web Advertising in 1997″ category), and I was amused to read the 1999 web page announcing the planned sale of the Adbility.com web site.  Reading that page, I noticed that I actually owned and sold the domain AdCents.com in 1999 (before Google created the AdWords or AdSense programs).  Somehow, thinking about that made me wonder which ads Google AdSense would display on my blog pages.

When I’d used AdSense in the past, I often found that it took a while to start inserting relevant ads onto content pages — and sometimes it would display ridiculous ad choices, even after many adviews for a particular content page. However, so far this afternoon, I am seeing highly relevant ads displayed adjacent to category and article pages.

Website tools

The biggest task this week was migrating my “free affiliate-program advice for merchants” from static pages over to the blog; if you’ve bookmarked or posted links to any of the old pages, they should redirect to the same exact article on the blog site.

Yes, I know my content will lose PageRank (and will likely impact Google’s perception of the “age” of the content, even though I’ve tagged each blog entry/article with the date it was originally written). And it is definitely a PITA to manually cut and paste content and then set up a redirect (for every static page) so they point to the correct blog entry.

But once I’m done, I’ll finally be able to make site-wide changes that actually affect my entire web site (without needing to manually edit a bunch of files that are five, ten, or even fifteen years old, dating back through and beyond several earlier content-management strategies I used). And I’m also inviting comments on the old articles (for example, I hope some folks might inform me of the new location of the many links within these articles which come up ’404 not found’ or worse).

Every time I do a “site-wide update,” I’m surprised to see how many hundreds of articles I’ve posted on my site over the years. Many of the older articles are deleted with redirects to my home page (most of these are articles about companies that are long defunct). (I’m actually considering “restoring” a few of those older articles, from backup CDs.)

I’m also surprised to see that my earlier migration of my “perspectives” content (the “opinion & commentary” essays which I first moved over to the blog) wasn’t quite complete, as I hadn’t replaced some of the static pages with redirects to the corresponding blog pages (I assume Google’s algorithms probably pounded those pages with “duplicate content” penalties).

Lots and lots of work, most of it very mundane, but it’s also a trip down memory lane. And ultimately, this work should make my life much simpler in future years, if I ever seek to profit from this content. It’s ironic that after 15 years mostly making a living from internet advertising, affiliate programs, and Google PPC Search, there are currently no ads anywhere on my site!

The migration is definitely not complete yet, and I’m still debating how to deal with some content which doesn’t seem appropriate for the blog site.

While I’d love to join the chorus criticizing both Apple and law enforcement, I simply don’t have enough information to make a reasonable decision about the conduct of various parties. Some key questions: did the “finder” of the phone make a reasonable effort to return it; was it legal to sell or buy the “found” item; was it legal to open and dissect the “found” phone; did Apple act reasonably to find and reclaim the phone; did Apple accurately describe the facts to law enforcement; did law enforcement properly investigate; did law enforcement properly describe the situation when seeking the search warrant; did the judge properly apply the law when issuing the search warrant; did law enforcement act reasonably in conducting the search after recognizing that the “shield law” might be applicable; did law enforcement act properly under the “shield law” both before and after the blogger and Gizmodo expressly asserted its applicability; and did law enforcement allow Apple agents to participate in the residential search?

One key issue, of course, is whether or not a “blogger” is protected under California’s laws protecting “journalists.”  While many people have strong opinions, the issue is one of interpreting a statute. (There’s a helpful column by Gillian Reagan at BusinessInsider.com discussing the California “reporter’s shield law” and how it includes bloggers.)  In short, there is no doubt that the “shield law” applies in this case — but it’s not an absolute protection.

The shield law protects journalists against being compelled to identify their sources or to reveal “unpublished information obtained or prepared in the course of newsgathering activities.” The “shield law” is actually part of the California Constitution (Article I, section 2, subdivision (b)), and in a 2006  decision by the California Court of Appeals (also involving an alleged theft of confidential information from Apple) the court wrote:

“We decline the implicit invitation to embroil ourselves in questions of what constitutes ‘legitimate journalis[m].’ The shield law is intended to protect the gathering and dissemination of news, and that is what petitioners did here. We can think of no workable test or principle that would distinguish ‘legitimate’ from ‘illegitimate’ news.”  O’Grady v. Superior Court (2006) (emphasis added).

The O’Grady case didn’t reach the California Supreme Court, and the ruling doesn’t hold that all bloggers are protected by the shield law. But the facts in O’Grady were remarkably similar to the current Gizmodo situation: both involved disclosure of Apple’s “confidential” materials.

After O’Grady, Apple knew it couldn’t compel disclosure of the iPhone-prototype “source,” and so it chose instead to report the phone as “stolen property” and encourage a law-enforcement task force to instead search the blogger’s home to try to identify the source, while also “fishing for” any other confidential Apple property that might be there.

What stinks here is that Apple knew that it was actively seeking to circumvent the protections of the “shield law.”  By conducting the search without warning while the blogger was not home, law enforcement sought to capture the critical information (the identity of the source) before the blogger could assert the privilege.  As hoped, the search disclosed the identity of the source.

It also seems quite likely that Apple already knew the identity of the source (who found the phone), if he actually did call Apple to report finding it.  If so, then did Apple tell law enforcement?

One key issue is whether the purchase of the “lost” iPhone prototype was legal; was this “stolen property”?  (If it was “stolen property,” then an effort to recover the stolen property, or to identify the thief, might reasonably appear to law enforcement to justify a search warrant, and law enforcement might also reasonably worry that if the search were delayed, the blogger might destroy evidence.  Note that days before the search, the iPhone prototype had already been returned to Apple.)

Gizmodo reported that the person who found the phone:

“… called a lot of Apple numbers and tried to find someone who was at least willing to transfer his call to the right person, but no luck. No one took him seriously and all he got for his troubles was a ticket number.”

Although the phone worked when found, it was completely disabled the next morning; apparently, Apple remotely “wiped” the phone to prevent its prototype software from being disclosed.

At that time, it seemed a distinct possibility (to both the “finder” and to Gizmodo) that the phone might be a hoax, intended to be “found” and reported about.  But the Gizmodo blogger paid $5,000 for the iPhone prototype; it’s hard to imagine anyone paying that much without the belief that it was “more likely than not” that it was genuine, and of course it was the hope that it was a genuine phone that led the blogger to pay for the phone. Once Gizmodo recognized that the phone was genuine, it knew it did not legally own the phone (and Gizmodo asserts that it always intended to return the iPhone prototype to Apple, if Apple would accept it).

Regardless of the “genuineness” of the phone, it clearly wasn’t the legal property of the person who found it.  (When I was a young teen, I found a nice watch in the snow one day, and after failing to find the owner at the scene, I dutifully turned it over the local police department; after the rightful owner failed to claim it within 90 days, the police gave me the watch to keep as my own.)

I started this column by writing that “I simply don’t have enough information to make a reasonable decision about the conduct of various parties.”  The situation certainly “stinks” — it appears highly likely that Apple deliberately sought to circumvent the protection of the “shield law” which is part of the California constitution, and it appears highly likely that law enforcement exercised very poor judgment in obtaining and executing the search warrant.  But all the facts aren’t in yet.