UPDATE (2008-04-14):In his posted a responding to the Security Focus alert, Matt Mullenweg noted the wp-pro mailing list as a resource for people who need to find consultants to help maintain their installations. This is great to know.This old server I've been running my stuff on is really long in the tooth and I hate it (the CPU is ancient and RedHat 9 sucks but at the time so many years ago, it was my best option). So I'm migrating to a new host, faster CPU, more RAM, newer OS, new software installs (no more Apache 1.3, g'bye old chum) ... we can rebuild it, we have the technology. I'm not going to post any WordPress updates for a while. If you're one of the folks out there who need help upgrading I know there are folks like The Friendly Webmaster who are available to consult. Unfortunately, I don't know of any WordPress equivalent to Six Apart's Professional Network but I'll be happy to post pointers to you if you're a consultant who can help people out with their WordPress situations. Of course, watch for updates from the WordPress Blog and follow the forums for updates.
Original post follows:
Meantime in Technorati's crawl data, the rate of WordPress site compromises hasn't really changed, there's a ton of WordPress installations that are being taken over. I've also been reading a lot of conflicting data points on the web and in email exchanges. Furthermore, I recently heard from a WordPresser that some of my information is wrong (though specifics were sparse) and I'd like to get whatever clarifications or corrections are necessary. Hopefully I'll hear back, I have no interest in posting inaccurate information; if/when I find out where it's wrong, I'll update here.
So for now, I'd like to thank my friends at ServePath for setting me up for the migration. I'll be working on moving my goods to some shinier digs and forgoing posting any more findings about WordPress for the time being. Peace out.( Apr 13 2008, 10:13:20 PM PDT ) Permalink
UPDATE (2008-04-14): Matt Mullenweg has posted a response to the Security Focus alert, he says it's bogus. I agree that a security alert needs to include more specifics about how an exploit is applied. I'm hoping now that either the author of that report steps forward with details or invalidates the whole thing. I'm disowning the post below (yet) but clearly people are talking about and need to reckon with the facts.
Original post follows:
More WordPress security concerns have come to my attention and it reminds me of the days 5 or 10 years ago when every other day seemed to bring a new exploit with Microsoft's IIS web server, Exchange, Internet Explorer or Outlook. I recall having a conversation with an analyst at the time, we concluded that Outlook wasn't just a chunk of swiss cheese security holes, it was a virus platform. I'm starting to arrive at the same conclusion about WordPress, given the procession of security issues that have come to my attention.
This latest one seems to affect all versions of WordPress (2.3.3 and 2.5 users, you're not safe). I'd seen a report about it here, which lead me to an analysis posted few weeks ago. I've seen a number of blogs with those symptoms (though they were older and I'd assumed they'd fallen victim to the XML-RPC exploit). Assuming this is the same issue, Security Focus says all versions are vulnerable (there's a long list of vulnerable versions and an empty space under "Not Vulnerable:", bad news ). And there's no patch under the "Solutions" tab. Ugh!
My estimation of WordPress is falling through the floor, maybe it's the Microsoft of blogging platforms. If WordPress doesn't respond soon with an aggressive trustworthy blogging response soon, Technorati may have to quarantine indexing all WordPress installations. Sux.( Apr 12 2008, 04:07:58 PM PDT ) Permalink
I found this post about 3rbsmag from the other week that provides some details of a particular WordPress attack interesting. Technorati is still seeing a steady flow of hacked blogs showing up in Technorati crawls. The ones that we can identify as symptomatic of the compromise aren't getting their crawls processed. Some bloggers have noticed that upgrading to WordPress 2.5 is an effective way to clear up those crawl obstacles. It seems like the word is getting out there, but there's still hundreds of vulnerable blogs being compromised every day. Some other WordPress blogs that I've noticed that have upgraded in the last few days include
I didn't post stats last night 'cause my macbook got mad at me for having too many Firefox tabs open, it staged a late-night revolt (it crashed) so I just called it a night. To catch things up, here's latest snapshot of the trailing 90 days of WordPress updates handled by Technorati:
|Version||Count (in thousands)||Change|
By the way, when I'm being good about posting links and dumping browser tabs, you can spot what I'm reading here. If I'm not posting to this blog, I might be posting links there.( Apr 11 2008, 11:12:03 PM PDT ) Permalink
I've seen a few ill-conceived suggestions that the measures we've taken at Technorati to suspend updates of blogs that appear vulnerable are coercive and should be countered. Let's just put this nonsense aside. When the XML-RPC exploits first caught my attention in February (two months ago), I was seeing five or ten, sometimes a few dozen blogs per day popping up on our radar with severely unusual publishing characteristics. I talked to Niall and Matt about it, learned about the hole that 2.3.3 fixed and posted about it on the Technorati blog urging bloggers to Patch or Upgrade Your Wordpress Installation, Now.So here are the bare facts: Around the tail end of March, the problem really snowballed. Kevin Burton put up a series of posts that caught my attention last month so we started comparing notes. This week in Technorati's crawl data, hundreds and sometimes thousands of vulnerable blogs everyday are showing up hacked regardless of rank, language or posting frequency. Why does this matter? All search systems that index links (Technorati, Google, Yahoo!, Ask, etc) have to discount the value of pages that are publicly writable. Wiki's, un-moderated/un-controlled comments and so forth are invariably spammed and that degrades the value of those pages. To prevent blogs from being classified as splogs just because they were hacked, we implemented the change announced at the beginning of this week Vulnerable WordPress Blogs Not Being Indexed. Please read this carefully: In that post, we said we were going to stop processing the crawls if the blog appeared symptomatic. We never said we were "de-listing" or "banning" blogs, yet there are
|Code Line||Patched Release|
I usually restrain myself from responding to trolls but the impacts we're seeing on the blogosphere are too important to let the fallacies and fear mongering go unchallenged. Don't pay attention to those who are trying to profiteer, making hay about Technorati being "bullies" or trying to "tell people how to blog." That's just outright nonsense. Techorati is not doing anything coercive at all, it's protecting the community by quarantining the infected. Technorati is simply suspending updates on the hundreds of blogs that are popping up as being vulnerable and appearing symptomatic of being hacked. Technorati is a small company seeking to be of service to a very large community. Amidst that community, a lot of bad actors (not the Keanu Reeves kind) are expending considerable effort to hijack the fundamental currency of the real time web: time and attention. We would be remiss if we didn't expend our efforts to thwart them.( Apr 11 2008, 10:33:17 AM PDT ) Permalink
The WordPress hack pandemic continues. Sampling the data from Technorati's crawler, I'd estimate there are at least 2500 blogs that did not get updated in our index in the last 24 hours due to being compromised. So while Rome is burning, the WordPress developers continue their violin serenade; the WordPress front page and blog still has nothing new posted alerting the vast majority of WordPress users how vulnerable they are. There's a huge, escalating problem for their community but instead the site is just the usual marketing fluff. It's really past time for the WordPress developers to exhibit some leadership. If Bill Gates can get off his butt to prioritize security, you'd think these dudes could. OK, here we are six years later; I never believed the "trustworthy computing" crap from Microsoft but at least they said something. What we're sorely missing from WordPress is trustworthy blogging.
Check your WordPress blogs and check your friend's. If you're not sure how to talk to your friends about it, perhaps these tips on How To Stop a Friend From Driving Impaired might help:
Seriously folks, send them to the WordPress post about the vulnerability.
- Be proactive. Don't wait for them to get around to realizing that they have a problem
- Politely, but firmly, tell them you cannot let them drive home because you care. Direct them to upgrade wordpress quickly (YMMV with those instructions).
- Drive your friend home. Upgrade their blog for them if they're too lame to do it.
- Call a cab. Tell them to shutdown their blog and use Facebook instead.
- Have your friend sleep over. Sex sells.
- Take the keys away. Help them migrate to Movable Type.
- Whatever you do, don't give in. Kick their asses.
read the original list
We at Technorati have discussed resumption of indexing vulnerable WordPress installations but treating all of the links like nofollow links. This might cause more misunderstanding about the issues than we currently have but it's worth consideration.
By the way, Google's Matt Cutts posted a nice write up with some basic security measures WordPress users should take, Three tips to protect your WordPress installation. These steps won't help you if you're WordPress installation is running a vulnerable version but they won't hurt. I disagree with Matt's recommendation to remove the generator tag - rather than removing it, I would recommend advertising that you're using a secure version of WordPress (2.0.11, 2.1.3, 2.3.3 or 2.5).( Apr 10 2008, 02:33:42 PM PDT ) Permalink
I've been acting on the assumption that WordPress 2.3.3 was a "safe" release. I certainly hadn't spotted any hacked blogs using 2.3.3 but poking around, I find these reports of compromised 2.3.3 blogs:2.5, 2.3.3, 2.1.3, and 2.0.11 -- if that's the case, I'll incorporate that into another update to Technorati's crawler (though to date, 2.1.3 and 2.0.11 have so far been statistically insignificant).
Folks need to keep getting the word out: friends don't let friends run vulnerable installations of WordPress. In the meantime, here's latest snapshot of the trailing 90 days of WordPress updates handled by Technorati:
|Version||Count (in thousands)||Change|
When I was comparing notes with Kevin Burton, it looks like we each independently found the same A-lister (who shall remain nameless here) that had fallen victim to the WordPress vulnerability on a secondary blog. I think we each independently had passed a "heads-up", I know I was in touch with this blogger a few times in the last two weeks about it. The blog has since been taken down (the URL redirects to a different blog and that redirect target is not vulnerable). This phenomenon is hitting blogs up and down the blogosphere's power curve -- it's neither the A-listers nor the Z-listers who are targetted. Any old vulnerable WordPress installation will do. And as can be seen in the metrics I've posted recently, the number of potential targets is vast.
Bokardo had fallen into the link-spam hole in Technorati's system because of spam defacement (I've since corrected the flagging, we're indexing Bokardo again). Ironically, the same day that Bokardo posted about being zapped in the Google index, the Google Webmaster Central Blog posted My site's been hacked - now what? which details the process of getting out of their purgatory. Unlike the aforementioned A-lister's silence on the matter, Bokardo author Joshua Porter posted about it, to which I say, "Yay, brother!" His case clearly illustrated the basic point: if you haven't upgraded your vulnerable WordPress installation, you're operating an insecure wiki -- any jackass with the exploit can re-write your pages (and worse). And they will.
Shift gears. I've been participating in online community on The WeLL for almost 14 years (yea, I'm paleolithic but I'm young at heart). One of the central ethical underpinnings on the WeLL is YOYOW: You Own Your Own Words. Other people can't quote/repost your words outside of the system without your permission and you need to be responsible for the things you say. In that spirit, I suggest that quality open source projects should adopt a collective You Own Your Own Code ethic. If you release code for other people to do great things with, mazel tov! But take pride in your products by keeping that usage fulfilling and secure. Where are the WordPress folks in getting the word out about the hack pandemic? Why isn't there a Big Red Banner on wordpress.org alerting people to the hazards of not upgrading? Waxing on about all of the groovy features in v2.5 is fine but really, they should be shouting: URGENT! YOUR INSTALLATION WILL BE HACKED UNLESS YOU UPGRADE TO ONE OF THIS FIXED RELEASES OR APPLY A PATCH. It's not like they don't know, both Kevin and I have talked to WordPress developers and posted very publicly about what's going.
Perhaps if Bokardo or the aforementioned A-lister migrated to Movable Type or some other platform and trumpetted about it, WordPress-land would hear the message. Instead of urging people to upgrade, maybe we should be urging them to migrate.( Apr 09 2008, 10:37:03 AM PDT ) Permalink
I've been conversing with Kevin Burton about the WordPress pandemic. We're in agreement that the WordPress community's response to this security issue has been excessively lax. Most of the feedback I've received about yesterday's crawler changes have been supportive; folks generally want more hygienic social media. Kevin is also implementing a change to block spam-infected blogs from Spinn3r's crawls. We're both going to be keeping tabs on this. I'll be developing metrics on the blogs that Technorati is not indexing when they appear symptomatic so that the efficacy (or not) of yesterday's changes are measured. In the meantime, here's an updated trailing 90 days of WordPress updates:
|Version||Count (in thousands)||Change|
Some of the feedback that I've heard from bloggers that haven't upgraded is that the upgrade is a big PITA. Some have asked me for referrals for WordPress consultants to help them get their theme and plugin data rolled forward to a newer version. If anybody has suggestions about where to find reputable consultants knowledgeable about WordPress, please blog about it. If you link to this post and you're not using a vulnerable version of WordPress, I'll even find it on Technorati( Apr 08 2008, 11:53:20 PM PDT ) Permalink
The blogosphere has had its share of maladies before. Comment spam, trackback spam, splogs and link trading schemes are the colds and flus that we've come to know and groan about. But lately, a cancer has afflicted the ecosystem that has led us at Technorati to take some drastic measures. Thousands of WordPress installations out in the wilds of the web are vulnerable to security compromises, they are being actively exploited and we're not going to index them until they're fixed.
We know about them at Technorati because part of what we do is count links. Compromised blogs have been coming to our attention because they have unusually high outbound links to spam destinations. The blog authors are usually unaware that they've been p0wned because the links are hidden with style attributes to obscure their visibility. Some bloggers only find out when they've been dropped by Google, this WordPress user wrote
My 2.2 installation was being hacked into and spam hidden links dumped into index.php. I didn't notice until google decided to ban me (they have now reincluded my site).
To their credit, the WordPress developers have been fixing the issues. They released v2.3.3 in February and patches for older releases to thwart this exploit. More recently, they released v2.5, which in addition to having the flawed XML-RPC code fixed, boasts a number of new features. But from what I can tell, despite brisk uptake many blogs remain obliviously vulnerable and the occurrence of compromised blogs seems to only be accelerating. As of today, here is the count of blogs running WordPress installs that have pinged Technorati in the last 90 days:
|Version||Count (in thousands)|
So at Technorati today, I posted that we deployed an update to the crawlers to abort the crawl if the blog appears to have symptoms of being compromised. We'll probably rescind this measure when the number of vulnerable installations in the distribution above looks a little better (some of the false positives I've found are patched but still have unusual metrics associated with the crawl, so they look fishy). However for the time being, these are just creating a lot of noise and instability in our systems and enough is enough. If you're running an old WordPress installation and you're not getting indexed, stop what you're doing and upgrade. Just Do It. The docs on the WordPress site seem to cover what you need to know and the WordPress Forums should help fill in the gaps.
Digging through the lore, it looks like there have been a procession of security problems with WordPress installations:
Using Technorati membership information, I have personally contacted several hundred of bloggers about this issue. These have included blogs with no authority as well as blogs belonging to A-listers. Many have been grateful for the heads up but none (that I have spotted) have posted about this issue. The blogs that are unclaimed are SOL, I don't have any way to reach them (without groping around their site to find a contact email, though I've done a little of that too). Kevin Burton has made a public plea, Anyone Want to Help Fix these Compromised Wordpress Blogs? One blog that did break the silence (Deep Jive Interests) did so in response to tweets about the issue that Kevin's been facing on TailRank.
But is outreach to bloggers going to be enough to stop the spread of this cancer? Probably not. I think the best way to get the word out is to spread the word, tell bloggers you know to post about it. For their part, what I'd really like to see from the WordPress folks (and all blog CMS developers) are
Building a team of rock stars is cheaper than a team of lower-salaried, less experienced programmers. It's also harder. The notion that there is more economy in the enthusiasm of project contributors and having "more hands on deck", even if they're cheaper hands, is naive. Martin Fowler
If the cost premium for a more productive developer is less than the higher productivity of that developer, then it's cheaper to hire the more expensive developer.You might assume that there's a positive scaling effect with a larger team. Fowler continues
The trouble is that that assumption assumes productivity scales linearly with team size, which again observation indicates isn't the case. Software development depends very much on communication between team members. The biggest issue on software teams is making sure everyone understands what everyone else is doing. As a result productivity scales a good bit less than linearly with team size. As usual we have no clear measure, but I'm inclined to guess at it being closer to the square root.Keep reading the Cheaper Talent Hypothesis.
Trouble is, finding the highly capable and seasoned talent can be a long search. Weeding out the fakers is time consuming, finding the right fit for those who are for real takes longer. And so the search goes on. Technorati is searching; if you're the real deal, call us.( Feb 09 2008, 08:48:53 AM PST ) Permalink
I've worked on a number of different web service and enterprise software products before but never gave one its external name until today. Our release of the Technorati Percolator is the culmination of months of work to harness the vast flow of raw data coming through Technorati to distill a palatable data volume and it's named for the internal moniker I'd been using for it during its development (after all, names with "buzz" and "meme" in them just wouldn't do). While you're looking around at the things we've cooked up in the percolator, make sure you also check out rising links of the day on Blogger Central and today in photos. Today we released them and I mentioned a bit about what goes into them on the Technorati blog. What I didn't elaborate on is what this release means to me on a personal level.
I originally came to Technorati in 2004 after a conversation with Dave fired up my creative sparks about the blogosphere. He had all of these rich conceptualizations about the technology changes in our midst, the social significance of decentralized events, the basic human drives that motivates them, the power of the long tail and the peculiar phenomenon that when you work in the service of others you reap the rewards manifold. I knew I had to work with him to build the ultimate air-traffic-control radar, real time search and meta-CMS systems. The 2004 political season provided an opportunity to work on those problems; the zeitgeist applications that we built to work with CNN's election coverage were thrilling accomplishments.
Since then, Technorati has undergone tremendous growth (regularly chronicled in Dave's "State of the Blogosphere" posts) on the foundation of a search vertical that had no precedent: the real time search of distributed micropublisheds sources. A number of technology changes were necessitated to scale us up; those changes have been likened to rebuilding your jet aircraft's engines at 40,000 feet. A lot has happened since 2004 (the growing pains have been regularly chronicled by the blogosphere) but until now, few of our outward facing accomplishments have excited me as much as the percolator.
There a lot of great sites out there using votes, comments, ratings and other explicit actions that are taken as representative of social gestures. There are also a lot of great sites that use implicit social gestures such as links to identify significant publishes, these are much closer to Technorati's heart. However, our aspirations are to look further along the long tail than most of these other sites can. Bloggers have said they want to see more than "all of the usual suspects", in an October 2007 post, ParisLemon said he wanted
a 'backpage' of sorts where some of us "B-listers" who are on ... everyday under the headlines, could have a chance to have some of our other tech stories showcased
Everyday the percolator is surfacing thousands of things that the blogosphere is talking about; blog posts, news stories and other stuff. It's true, the "A-list" percolates more posts and they bubble up higher; this is basic social software physics and classic power law stuff. But we have put a stake in the ground; we're going to serve bloggers across the power curve spectrum who are producing quality posts and acquiring attention from other bloggers as well as identify where the other attention magnets are by enabling an application that highlights them. When you walk into a crowded party and there are a myriad of conversations going on, you want to find the conversations that are pertinent to your interests and who the thought leaders are in those conversations. For me, today's release marks a new beginning of Technorati playing the role of connector and catalyzer. I hope you enjoy it!( Dec 04 2007, 11:45:40 PM PST ) Permalink
Every family and every community has them. Addicts. Lives twisted by chemical dependency and the accompanying mental illnesses. Maybe I'll never fully understand how lives can wind down into oblivion in that way, given the many opportunities to I consider myself lucky to have never succumbed to such an existence.
From his sister, here's a short contribution to understanding the life, decline and death of one of my teenage cohorts Sherwood Brewer. For now and evermore, I imagine he's partying with Skitchie: boot-a-doot-doot!( Oct 29 2007, 08:59:13 AM PDT ) Permalink
There are blogs that don't take comments (like this one: I don't have time to moderate spam). There are mainstream media sites that are adopting reader comments. There are blogs being published by independent companies with editorial staff. There are big media organizations publishing columns and event streams as blogs. So I'm finding myself asking some basic questions about blogging of late: Is it an indication of maturation or mutation of the blogosphere that there's quibbling about what's a blog and what isn't? Is main stream media's co-opting of blogospheric mores a harbinger of a thermador to some un-televised revolution? Has the little town become too much of a metropolis that twitter, facebook and other social media are the destinations of urban flight?
The basic existential questions of the blogosphere and where its boundaries reside have been open to consideration (and re-consideration) for quite some time. Not a day goes by on the Technorati support forums without a splogger showing up to complain that their spam isn't getting indexed (Note: I'm not saying everyone who has indexing problems is a spammer, I'm saying spammers come rolling in to complain about it). A few weeks ago, Scoble melodramatically lamented that the TechMeme leaderboard heralds the death of blogging":
I was just looking at the TechMeme Top 100 List and noticed that it has very few bloggers on it. I can only see about 12 real blogs on that list. Blogging being defined as 'single voice of a person.' Most of the things on the list are now done by teams of journalists - that isn't blogging anymore in my book.It's true, a lot of the many of the successful blogs have a prolific editorial staff. But death? Really? Why is blogging as an individual practice more or less than blogging as part of a collaborative enterprise? The existence of the weblogsincs, gawkers and huffington posts of the world are manifestations of blogging as a format but are from what I can tell are no less or more blogs than any others. New blogs continue to be created every second, and some of them will eventually develop thriving audiences.
The line between micropublishing and macropublishing is blurring. Reuters recently announced they they're taking comments on stories and Ally Insider's revelation that the New York Times is posting reader comments got a lot of play. In his post about Technorati rankings, Doug Karr doesn't feel that CNN Political Ticker should be considered a blog. So I'm asking myself, when is a blog not a blog?
Sometimes blogs (the narrower Scoble definition kind) provide the primary source for the facts of our times. Other times, it's main stream media that is bringing forth those facts. As the emergence of blogs that operate like main stream media continues and main stream media adopts blogging as a technology and practice, perhaps this is the ultimate outcome of a leveled publishing playing field: changes will flow along many vectors, cross bred practices are inevitable and Darwinistic rules will prevail such that a lot of things that you'd previously not have considered blogs are morphing into them.( Oct 13 2007, 11:22:11 PM PDT ) Permalink
Listen up kids, crime doesn't pay.
Trying to follow a link to Linus Torvalds' railing against subversion, the irony of getting this error heightens the humor:
Microsoft OLE DB Provider for ODBC Drivers error '80004005' [Microsoft][ODBC SQL Server Driver][SQL Server]Transaction (Process ID 134) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. /efytimes/lefthome.asp, line 193Sure, database problems happen regardless of the enabling technology, Microsoft is not unique to this. However, I seem to run into completely fubarred application degradation like this (essentially a BSOD on the web) far more often with ASP and .Net based sites than those enabled by other technologies. Of course, any site architected to require a database transaction to serve a content page (without any user data transaction) is a firing offense any place I'll ever work. ( Aug 19 2007, 09:46:22 AM PDT ) Permalink