As long as there is any media, pornographers will figure out how use it to purvey their wares. The other week, I mentioned on the Technorati blog that I'd been focusing on some spam scrubbing efforts, including removing porn. Apparently we're not the only social media service taking a look at the bottom line impact of miscreant activities. A few related items of interest percolated recently.
Social network service provider Ning announced their End of the Red Light District. The high infrastructure costs, lack of revenue and administrative burdens (DMCA actions) were among the reasons cited. Sounds very familiar, we get our share of that kind of pointless nonsense at Technorati too.
Today, YouTube posted that they were going to crack down or reduce the visibility of porny videos. YouTube's measures include
As expected in these cases, the trolls come out to cry foul. But this isn't about free speech or puritan ethics, the issue more closely resembles the tragedy of the commons. It's really very simple: these parasitic uses consume a lot of resources but bring no benefits to the host and degrade the service for other users.
Also today, Digg Bans Company That Blatantly Sells Diggs was reported by Mashable. Apparently Digg has directed a cease-and-desist at USocial.net's practice of selling diggs.
It seems to be an accepted truism that social media oft demonstrates, All Complex Ecosystems Have Parasites. Yep, I've talked to folks from Six Apart, Wordpress, Tumblr, Twitter and elsewhere. We're all feeling the pains of success. Over the past month at Technorati, we've purged about 80% of the porn that was active in the search index. Sure, we're not spam free yet but the index is getting a lot cleaner.( Dec 02 2008, 11:42:34 PM PST ) Permalink
Technorati's authority metric is based on a real simple concept: the count of the unique set of blogs linking to you in the trailing 180 days constitutes your authority. By its very nature, it's a volatile metric. The top 100 of a few years ago bears little resemblance to the one today. When some folks observe their authority rising, they twitter w00ts of joy; when it's falling they complain bitterly that Technorati is "downgrading" them.
Authority is not a perfect metric (crawl coverage variations, etc) nor the only important measurement of a blog (traffic and comments are other metrics we'd like to measure), however it is one that Technorati has been objectively calculating for years.
What I find surprising is the surprise (or denial) that some people find when they learn there are consequences to gaming the system. On a fairly regular basis, someone comes up with the wholly unoriginal idea, "Hey, add your URL to my list of links, re-post it and urge others to follow suit to make your Technorati authority explode!" Or some variant of a viral link exchanging scheme. Some folks take the news graciously, "Oh, that's not OK? I had no idea. It won't happen again." But some of these folks get downright hostile, as if the blog authority metric is their god given right to game. These are probably the same people who expect appreciation on their home's property value to be a god given right. News flash: it's not. Since it's (apparently for some) not obvious: the attention you garner in the blogosphere and the price someone will pay for your house are driven by market forces. If your authority is dropping, create posts that are link-worthy. There's no shortcut. Blogs engaging in viral linking schemes stand a good chance that indexing will be suspended or the blog removed altogether from Technorati's index.
Use the blogosphere to converse, to entertain, to teach and to learn. We'll do our best to measure it and to build applications with those measurements. If you want to play games, get a Wii.( Dec 01 2008, 10:40:34 PM PST ) Permalink
I usually don't rant about economics but I wasn't shopping on "Black Friday" (nor will I be tomorrow on "Cyber Monday") - I'm trying to figure out how to tighten my belt. How it is that I, someone outside of the real estate, finance and auto industries that are so problem plagued, am getting caught in our economy's downdraft? Well, let's see.
Last January, Business Week raised the question "When is an institution too big too fail?" Until September of this year, the financial industry's downward spiral meandered along, like a persistent flu. There were bank failures but the conventional wisdom seemed to be that this was the market at work, winnowing the weak. The bad news ebbed and flowed: mortgage failures, rising oil prices and the weak dollar were countered by stimulus package checks, housing sales leveling off or even rising (where prices crossed their local tipping points) and vibrant web 2.0 and green enterprises. There had been bank failures this year but it took the evaporation of really Big institutions, Lehman Brothers and Merrill Lynch to put Business Week's question on everyones lips. To free market purists, the answer is obvious: whatever may come, let the failures fail. But the reality is that when an enterprise is so big that its failure disrupts significant portions of the overall national and global economy, whatever may come of its failure won't be good. Everyone suffers and bigness is the problem. When these companies become indispensable institutions, we should be afraid.
It seems for years there's been a breakdown in accountability. Loan originators could resell their loans and write new ones, no harm no foul. Right? But one of the key problems with that system is that the originators don't have any skin in the game. The have a money merry-go-round and whoever is left holding the paper (big institutions and their investors) draws the short straw. It's total madness. To date, all of the bank failures have resulted in consolidation in some form or another. Lehman is absorbed by Barclays. Merrill by BoA (which already absorbed Countrywide). The big are getting bigger as the competitive field shrinks. Ironically, this perpetuates the problem: bigness. What happens when Barclays or BoA start wobbling next? Now we have yet bigger institutions that are again too big to fail.
Among the remedies dismissed by free-market adherents is one of the Federal Government investing, taking an ownership stake in the banking, insurance and auto giants who have exposed themselves to risk that has subsequently blown up in their faces. "The government won't know any better how these companies should be run" goes the admonishment. But as if it isn't clear by now, the executives paid the big bucks to know how they should be run apparently don't either. As Newsweek explains in The Monster That Ate Wall Street - How Credit Default Swaps Became a Timebomb, the financial industry had no shortage of creativity when it came skirting the liquidity requirements imposed on them in the years following the S&L crisis. Is it really such a surprise? Michael Lewis (Liar's Poker, Moneyball) recounts in The End of Wall Streets Boom (Portfolio Magazine / December 2008), there were those calling Bullshit but things were just going too damned well for those alarms to be heeded.
It's unescapably clear now that the old adage applies, "if it's too good to be true, it probably is." Until recently, I thought this was only impacting me with the difficulty I had getting my mortgage. But no, the cavalier rating agencies ("the fox was guarding the hen house"), excessively leveraged financial arrangements and detached accountability have led us down this financial rabbit hole into what some now describe as a death spiral. It's not just a Wall Street problem, it's spillover to Main street has cacaded down Sandhill Road. Here's the ominous and infamous slide deck from Doug Leone and friends at Sequoia Capital:
As bummed as I am about seeing colleagues depart and seeing my paycheck shrink, I'm actually optimistic about the future. Valuations on real estate seem to be reaching reality: they're hitting thresholds that people can afford with conventional financing. Technology continues to fuel innovation and innovation holds the potential to re-shape markets. Come Inauguration Day, it looks like Obama is coming into office surrounding himself with a team of economic advisors who are committed to preserving free markets but are also not so steeped in ideology that they're paralyzed about how to intervene.
I'm looking forward to this cloud lifting. That's my rant.( Nov 30 2008, 06:05:51 PM PST ) Permalink
Back in the 1990's I used FreeBSD fairly extensively. One of my favorite things about the FreeBSD project was the "ports and packages" system for installing libraries and application software. Since Mac OS X is, essentially, BSD with a lot of updated chrome, it's not surprising that there's a well functioning "ports and packages" system for it, MacPorts. While it's not perfect, MacPorts seems to function and dovetail nicely with everything I use my Mac for, more so than Fink. Sure, dpkg/apt-get seems to work OK on Debian, every effort I've encountered to apply that model elsewhere has left me disappointed... anyway, yum seems to work well enough, I don't expect to use Debian again.
Recently I found myself with a port that would not install,
port install postgiswould bomb out:
"Target org.macports.checksum returned: Unable to verify file checksums" postgisIt's not a very helpful error message. After some RTFM ("man port", imagine that), I figured there musta been some cruft in the way, so I did this:
port -d selfupdate port clean --all postgis port install postgisAnd I'm in business with the latest version of PostGIS. Yes, I coulda installed all of that stuff by hand but MacPorts generally has just what I need in a time-saving way. Note, I do all my MacOS X system administration as root so I'm not typing "sudo" all of the time. ( Nov 29 2008, 03:10:44 PM PST ) Permalink
Since Technorati announced pay cuts for the staff earlier this week, I've been a little worried. The mortgage, an upcoming bat mitzvah (nothing opulent, really), doctor bills... the world won't wait for the economy's doldrums to turn around. I think I'll find ways to to tighten our belts (bag lunches, cancel the gym membership, etc) but if you're currently more fortunate than I am and so inclined, this PayPal Donate button is a way you can help.
If I end up with more than needed, I'll simply donate the excess to a worthy charity.
Public service announcement: your low-tech dowdy public libraries have slicked up high-tech. The old days of long searches through card catalogs and filling out forms in triplicate are gone. Since moving to the east bay several years ago, I've been impressed with the Contra Costa County Library's online catalog that searches all of the branches in the country, online reservations and inter-branch transfers. One of my favorite features is the visual topic clustering.
All said, I'm a fan of the search and clustering technology enabled by AquaBrowser that the CCC library is using, it's had me wondering how well it would perform against the more volatile data set flowing through Technorati.( Nov 27 2008, 11:43:13 AM PST ) Permalink
The incidence of Wordpress compromises I wrote of in the spring is still high but the rate of new infections has dropped considerably. A lot of people learned of their blogs' affliction because they were not getting indexed by Technorati. Props to the folks from Google and the Wordpress team for getting the message out too.
Yesterday's release of Wordpress 2.6.5 doesn't target SQL injection or XML-RPC vulnerabilities, this time it's a cross site scripting vulnerability.
The security issue is an XSS exploit discovered by Jeremias Reith that fortunately only affects IP-based virtual servers running on Apache 2.x. If you are interested only in the security fix, copy wp-includes/feed.php and wp-includes/version.php from the 2.6.5 release package.So jump on it Wordpress users, time to update! ( Nov 26 2008, 07:09:11 AM PST ) Permalink
2.6.5 contains three other small fixes in addition to the XSS fix. The first prevents accidentally saving post meta information to a revision. The second prevents XML-RPC from fetching incorrect post types. The third adds some user ID sanitization during bulk delete requests. For a list of changed files, consult the full changeset between 2.6.3 and 2.6.5.
read the full post
It's been a long time since I've felt hopeful about the outcome of an election. I remember well the civil rights anti-war marches of my childhood. The recent years have felt like a return of profound delusional corruption and polarization that marked the Nixon era. The most peculiar aspect of it is how it came to a head with desperate gasp of Sarah Palin's VP nomination. That the questions about her qualifications were questioned at all demonstrates the height of delusion. While the republican fringe ran to embrace her, those with a brain could only ask "WTF?" and cross party lines. If she had been equipped otherwise between those legs, it would never have happened. It's like reverse sexism, if it had been a man with that background and view points, he would have been laughed out the door as a naive hillbilly. (Yep, I am a reasonably educated urban elitist, so?) Instead, we were treated to tragic comedy in slow mo. The graphic included here is a chart of the blogosphere's mention trajectory for Sarah Palin over the prior 100 days.
The timeline starts off with the igloo phase: practically nobody has heard of her and nobody is talking about her. Then there's the nomination at the republican convention. Followed by interviews, the Tina Fey phenom, the election and ...back to the igloo. So long, Caribou Barbie.
All of the talk about her PR representation, book deals, etc are for naught; she's proven at every opportunity that she has nothing to say that is meaningful and contributing to moving our society forward. Nonetheless, it's amusing to read the conservative bloggers who talk about Sarah Palin as "the future of the republican party." As long as they stick to that meme, they're assuring themselves falling further adrift of where this country is going. Bon voyage, don't let that iceberg hit you in the butt on the way out!( Nov 25 2008, 02:12:45 PM PST ) Permalink
At long last, I'm reviving this blog from dormancy. A lot has happened since my prior posting here. In no particular order:
But they'll wait for another post. It's nice to be back!( Nov 23 2008, 04:17:54 PM PST ) Permalink
UPDATE (2008-04-14):In his posted a responding to the Security Focus alert, Matt Mullenweg noted the wp-pro mailing list as a resource for people who need to find consultants to help maintain their installations. This is great to know.This old server I've been running my stuff on is really long in the tooth and I hate it (the CPU is ancient and RedHat 9 sucks but at the time so many years ago, it was my best option). So I'm migrating to a new host, faster CPU, more RAM, newer OS, new software installs (no more Apache 1.3, g'bye old chum) ... we can rebuild it, we have the technology. I'm not going to post any WordPress updates for a while. If you're one of the folks out there who need help upgrading I know there are folks like The Friendly Webmaster who are available to consult. Unfortunately, I don't know of any WordPress equivalent to Six Apart's Professional Network but I'll be happy to post pointers to you if you're a consultant who can help people out with their WordPress situations. Of course, watch for updates from the WordPress Blog and follow the forums for updates.
Original post follows:
Meantime in Technorati's crawl data, the rate of WordPress site compromises hasn't really changed, there's a ton of WordPress installations that are being taken over. I've also been reading a lot of conflicting data points on the web and in email exchanges. Furthermore, I recently heard from a WordPresser that some of my information is wrong (though specifics were sparse) and I'd like to get whatever clarifications or corrections are necessary. Hopefully I'll hear back, I have no interest in posting inaccurate information; if/when I find out where it's wrong, I'll update here.
So for now, I'd like to thank my friends at ServePath for setting me up for the migration. I'll be working on moving my goods to some shinier digs and forgoing posting any more findings about WordPress for the time being. Peace out.( Apr 13 2008, 10:13:20 PM PDT ) Permalink
UPDATE (2008-04-14): Matt Mullenweg has posted a response to the Security Focus alert, he says it's bogus. I agree that a security alert needs to include more specifics about how an exploit is applied. I'm hoping now that either the author of that report steps forward with details or invalidates the whole thing. I'm disowning the post below (yet) but clearly people are talking about and need to reckon with the facts.
Original post follows:
More WordPress security concerns have come to my attention and it reminds me of the days 5 or 10 years ago when every other day seemed to bring a new exploit with Microsoft's IIS web server, Exchange, Internet Explorer or Outlook. I recall having a conversation with an analyst at the time, we concluded that Outlook wasn't just a chunk of swiss cheese security holes, it was a virus platform. I'm starting to arrive at the same conclusion about WordPress, given the procession of security issues that have come to my attention.
This latest one seems to affect all versions of WordPress (2.3.3 and 2.5 users, you're not safe). I'd seen a report about it here, which lead me to an analysis posted few weeks ago. I've seen a number of blogs with those symptoms (though they were older and I'd assumed they'd fallen victim to the XML-RPC exploit). Assuming this is the same issue, Security Focus says all versions are vulnerable (there's a long list of vulnerable versions and an empty space under "Not Vulnerable:", bad news ). And there's no patch under the "Solutions" tab. Ugh!
My estimation of WordPress is falling through the floor, maybe it's the Microsoft of blogging platforms. If WordPress doesn't respond soon with an aggressive trustworthy blogging response soon, Technorati may have to quarantine indexing all WordPress installations. Sux.( Apr 12 2008, 04:07:58 PM PDT ) Permalink
I found this post about 3rbsmag from the other week that provides some details of a particular WordPress attack interesting. Technorati is still seeing a steady flow of hacked blogs showing up in Technorati crawls. The ones that we can identify as symptomatic of the compromise aren't getting their crawls processed. Some bloggers have noticed that upgrading to WordPress 2.5 is an effective way to clear up those crawl obstacles. It seems like the word is getting out there, but there's still hundreds of vulnerable blogs being compromised every day. Some other WordPress blogs that I've noticed that have upgraded in the last few days include
I didn't post stats last night 'cause my macbook got mad at me for having too many Firefox tabs open, it staged a late-night revolt (it crashed) so I just called it a night. To catch things up, here's latest snapshot of the trailing 90 days of WordPress updates handled by Technorati:
|Version||Count (in thousands)||Change|
By the way, when I'm being good about posting links and dumping browser tabs, you can spot what I'm reading here. If I'm not posting to this blog, I might be posting links there.( Apr 11 2008, 11:12:03 PM PDT ) Permalink
I've seen a few ill-conceived suggestions that the measures we've taken at Technorati to suspend updates of blogs that appear vulnerable are coercive and should be countered. Let's just put this nonsense aside. When the XML-RPC exploits first caught my attention in February (two months ago), I was seeing five or ten, sometimes a few dozen blogs per day popping up on our radar with severely unusual publishing characteristics. I talked to Niall and Matt about it, learned about the hole that 2.3.3 fixed and posted about it on the Technorati blog urging bloggers to Patch or Upgrade Your Wordpress Installation, Now.So here are the bare facts: Around the tail end of March, the problem really snowballed. Kevin Burton put up a series of posts that caught my attention last month so we started comparing notes. This week in Technorati's crawl data, hundreds and sometimes thousands of vulnerable blogs everyday are showing up hacked regardless of rank, language or posting frequency. Why does this matter? All search systems that index links (Technorati, Google, Yahoo!, Ask, etc) have to discount the value of pages that are publicly writable. Wiki's, un-moderated/un-controlled comments and so forth are invariably spammed and that degrades the value of those pages. To prevent blogs from being classified as splogs just because they were hacked, we implemented the change announced at the beginning of this week Vulnerable WordPress Blogs Not Being Indexed. Please read this carefully: In that post, we said we were going to stop processing the crawls if the blog appeared symptomatic. We never said we were "de-listing" or "banning" blogs, yet there are
|Code Line||Patched Release|
I usually restrain myself from responding to trolls but the impacts we're seeing on the blogosphere are too important to let the fallacies and fear mongering go unchallenged. Don't pay attention to those who are trying to profiteer, making hay about Technorati being "bullies" or trying to "tell people how to blog." That's just outright nonsense. Techorati is not doing anything coercive at all, it's protecting the community by quarantining the infected. Technorati is simply suspending updates on the hundreds of blogs that are popping up as being vulnerable and appearing symptomatic of being hacked. Technorati is a small company seeking to be of service to a very large community. Amidst that community, a lot of bad actors (not the Keanu Reeves kind) are expending considerable effort to hijack the fundamental currency of the real time web: time and attention. We would be remiss if we didn't expend our efforts to thwart them.( Apr 11 2008, 10:33:17 AM PDT ) Permalink
The WordPress hack pandemic continues. Sampling the data from Technorati's crawler, I'd estimate there are at least 2500 blogs that did not get updated in our index in the last 24 hours due to being compromised. So while Rome is burning, the WordPress developers continue their violin serenade; the WordPress front page and blog still has nothing new posted alerting the vast majority of WordPress users how vulnerable they are. There's a huge, escalating problem for their community but instead the site is just the usual marketing fluff. It's really past time for the WordPress developers to exhibit some leadership. If Bill Gates can get off his butt to prioritize security, you'd think these dudes could. OK, here we are six years later; I never believed the "trustworthy computing" crap from Microsoft but at least they said something. What we're sorely missing from WordPress is trustworthy blogging.
Check your WordPress blogs and check your friend's. If you're not sure how to talk to your friends about it, perhaps these tips on How To Stop a Friend From Driving Impaired might help:
Seriously folks, send them to the WordPress post about the vulnerability.
- Be proactive. Don't wait for them to get around to realizing that they have a problem
- Politely, but firmly, tell them you cannot let them drive home because you care. Direct them to upgrade wordpress quickly (YMMV with those instructions).
- Drive your friend home. Upgrade their blog for them if they're too lame to do it.
- Call a cab. Tell them to shutdown their blog and use Facebook instead.
- Have your friend sleep over. Sex sells.
- Take the keys away. Help them migrate to Movable Type.
- Whatever you do, don't give in. Kick their asses.
read the original list
We at Technorati have discussed resumption of indexing vulnerable WordPress installations but treating all of the links like nofollow links. This might cause more misunderstanding about the issues than we currently have but it's worth consideration.
By the way, Google's Matt Cutts posted a nice write up with some basic security measures WordPress users should take, Three tips to protect your WordPress installation. These steps won't help you if you're WordPress installation is running a vulnerable version but they won't hurt. I disagree with Matt's recommendation to remove the generator tag - rather than removing it, I would recommend advertising that you're using a secure version of WordPress (2.0.11, 2.1.3, 2.3.3 or 2.5).( Apr 10 2008, 02:33:42 PM PDT ) Permalink
I've been acting on the assumption that WordPress 2.3.3 was a "safe" release. I certainly hadn't spotted any hacked blogs using 2.3.3 but poking around, I find these reports of compromised 2.3.3 blogs:2.5, 2.3.3, 2.1.3, and 2.0.11 -- if that's the case, I'll incorporate that into another update to Technorati's crawler (though to date, 2.1.3 and 2.0.11 have so far been statistically insignificant).
Folks need to keep getting the word out: friends don't let friends run vulnerable installations of WordPress. In the meantime, here's latest snapshot of the trailing 90 days of WordPress updates handled by Technorati:
|Version||Count (in thousands)||Change|