Sunday December 05, 2004 When folks say "service oriented architecture" it still cannotes monolithicism to me. An architecture implies a level of structure definition that sounds rigid; can you re-pour that foundation to adapt redrawn plans? Software development agility and loose coupling should reinforce each other. I prefer to think of architectures and ecosystems. A service oriented functionality ecosystem supplies application functionality as a suite of services. Supporting requirements (as opposed to the core business requirements) such security, logging, persistence, redundancy and caching are each handled independently; they in turn may be provisioned as services that higher level services rely on. This is part of the evolution under way at Technorati; some of the changes are evident in Dave's recent posts but some are just revisions that we're quietly rolling out.
Queues and distributed memory caches are natural elements of a such an environment. In the December issue of Linux Journal, Technorati's use of open source building blocks such as memcached is discussed by Doc Searls.
This is the game:
A memcached server (or a set of servers) can be accessed over the network to store things in a table kept in RAM. When storing things, you can specify a maximum age for the cache entry -- if you go back to fetch it and the elapsed time since it was stored exceeds that age, it gets treated as a cache miss.
Storing things in memcached with the timeout parameter and invalidating cache entries works as long as you have consistent mechanism for calculating the key. If internally you're managing "stories" and each one has an "id" attribute that is unique (a primary key), that's a good candidate to store them with. So for instance putting memcache inside a content management system (CMS) "content service" seems natural. In babytalk code:
public Story fetchStory(int storyId) {
Story story = memc.get(storyId);
if (story == null) // perhaps more rigorous validation of the fetched object
return story;
story = StoryDB.findById(storyId);
memc.put(storyId, story, AGE);
return(story);
}
public Story fetchStory(Map atts) {
// encapulate whatever attributes uniquely identify a thing
CacheKey key = new CacheKey(attrs);
Story story = memc.get(key);
if (story == null)
return story;
story = StoryDB.findByAttrs(attrs);
memc.put(key, story, AGE);
return(story);
}
We're in the process of evolving Technorati's infrastructure to one that is loosely coupled, redundant and robust. Our use of memcached is one of the enabling technologies of that evolution.
( Dec 05 2004, 09:22:23 AM PST ) Permalink View blog reactions
Tuesday November 23, 2004 I usually only use "cvs import" to create a new CVS module but it can also be used to do a "bulk add." Maybe it's common knowledge for CVS jockies but it's easy to forget about unless oft-used. Here's the scenario:
Friday November 19, 2004 The only gripe I've heard about Eclipse that I haven't had a good answer for is the absense of Emacs key bindings. Otherwise, what's there not to dig about Eclipse?
I was hopefull that the EPIC plugin would provide at least some of those things for Perl development. This is what I found:
In the meantime, you can enjoy the fruits of this week's labor by pulling it off of CPAN; that's where you can get WebService::Technorati. It's also part of the latest release of the Technorati web services SDK. Thanks to David Wheeler for turning me on to Pod::Simple::HTML ...I'm still trying to figure how he gets it to output nice docs from pod, mine didn't come out nearly that purty. Ah well, I guess that'll be part of next week's Perl fun.
( Nov 19 2004, 10:59:25 PM PST ) Permalink View blog reactions
Wednesday November 17, 2004 It turns out that expat is not installed, grrr. So I fired up Fink Commander and had it gimme some expat lovin'. Tried it again -- bzzzt! This is what I did in the CPAN shell
cpan> o conf makepl_arg "EXPATLIBPATH=/sw/lib EXPATINCPATH=/sw/include" cpan> install XML::Parser-- ding-ding-ding! We have a winner! XML::Parser installed! Thereafter, XML::XPath JFW'd and I'm on my way. ( Nov 17 2004, 04:53:44 PM PST ) Permalink View blog reactions
Sunday November 07, 2004 I poked around the Jakarta bug database and the only mention I could find that close was PR 31442, which described having this
<%@ page language="java" contentType="text/html; charset=UTF-8" %> <%@page pageEncoding="UTF-8"%>and saying that the text was coming back ISO8859-1 when the page is requested as a GET but not as a POST. Well, someone from the Jakarta project marked the bug INVALID glibly saying to ask on the user's mailing list and look at the Connector configuration because it's not a bug. WTF? Are you kidding?
Now I looked around in the Connector stanza's that come in the server.xml and see no mention of encoding configuration attributes. I've got a real simple test case.
<% response.setContentType("text/xml"); %>
triggers no funny encoding behavior, I get the data out as good old utf8 just as I wanted but if I did this
<% response.setContentType("text/xml; charset=UTF-8"); %>
....kablooey! Mangled encoding! That's just wrong. And if it's not wrong, I think it warrants a better answer than RTFM on the Connectors.
And the problem may not just be isolated to JSP handling. Judging from other reports that are turning up in Google's index pertaining to SetCharacterEncodingFilter, it's affecting the filter implemetation as well.
( Nov 07 2004, 02:46:58 AM PST ) Permalink View blog reactions
Saturday November 06, 2004
In the meantime, the Big Lie that waging war on Iraq has some relationship to 9/11 and terrorism apparently has been successfully Jedi mind-tricked into the American psyche and we're destined to have four more years of high crimes and misdemeanors. It just makes me wonder what is up with the rest of the country. Plenty of folks abroad are, evidently, equally perplexed by this election, as we see in a recent Daily Mail cover.
If you're single, there are some Canadians offering asylum. I'm thinking of packing up the family and moving to New Zealand or something. Just to keep track of where I don't want to be, I'm reckoning with the map: ![]()
Source: http://www.electoral-vote.com/
|
Do you live in a state of stupity? Apparently 59,054,087 of you do.
|
Saturday October 23, 2004 Remember Lucilla in Gladiator? Yea, that's Connie Nielsen.
| Here's a guy with two sons and a wife of 7 or 8 years going to fashion shows, art auctions and movie premiers with his Danish girlfriend. Oh, Lars: you're so damned hollywood! Apparently the paparazzi in Denmark have kept tabs on them as well. |
|
Back in the old days o' Metallica we had loads of fun but didn't go to fashion shows, art auctions and movie premiers. We didn't sip fine wines either. Oh well, I hope the dude is happy.
( Oct 23 2004, 07:29:35 PM PDT ) Permalink View blog reactions
Friday October 22, 2004
After raising the notion with Tantek, he plugged the trivial bit to enable this on the Technorati site..
Check it out http://www.technorati.com/cosmos/referer.html (ok, so I'm not very popular in this big 'ol cosmos but anyway...). This is what you do:
Thursday October 21, 2004 OK, I lied. It ain't about me, it's about our new office and the major milestones that Technorati is achieving, the agony of startup setbacks and the ecstacy of... having fun! The details:
Here's Dave's original post.
( Oct 21 2004, 04:34:11 PM PDT ) Permalink View blog reactions
Wednesday October 20, 2004 I've used Eclipse extensively in the past but since there was kinduva a long hiatus in Java development for me, it felt both novel and familiar to install it on my powerbook for the stuff I'm currently working on. I think that liberated feeling of knowing that That Thing Works So I Can Move On Now is referred to as being "Test Infected." Which, unlike the flu, is a good kind of infection. One of the things I've been trying to be more consistent about is writing tests prior to, or if not, concurrently with, application code. I strongly recommend reading Kent Beck's Test Driven Development (who BTW appears to have a work in progress about Eclipse), TDD is a quick read but more than any particular valuable agile techniques per se it offers a good outlook on how to think about what you're doing when you sit down to write code.
| Well, I was writing a few initial test cases for a class I was working on and noticed something new (to me anyway) when I clicked on an error that Eclipse flagged. The non-existence of the method in the called class was identified and Eclipse offered to stub it out on my behalf, this is a huge win as far as expressing an API in a test case and continuously filling in the functionality development "TODO" list. Everytime I want to add a new public method, I'll write a new test case and let Eclipse bootstrap the implementation. | |
|
|
Read it:
|
Now if I could get that level of simplicity and automation working with Perl, Python or PHP, I'd be jammin'! Sure, there are testing frameworks for each of these but Eclipse really streamlines that whole TDD cycle. |
Forget about the flu shot hysteria, get "Test Infected"
( Oct 20 2004, 11:04:06 AM PDT ) Permalink View blog reactions
Tuesday October 19, 2004 Sure, I have a rain parka but that's not the point, I wanted to avoid having my backpack get saturated. So after waiting about 10 minutes, a two car N-Judah rolls into Embarcadero station at aroun 8:40 or so with the robotic audio announcement "two car. mission bay. now approaching...." A crowd piles on; I don't think the MUNI scheduling people have realized that a lot of people work near the ball park.
Out on the surface streets at Harrison, the driver announces that it's the last stop and that the next N would arrive in 3 minutes! Why would they cut the route 3 stops short?! Everyone spills out into the deluge of wind and rain; the shelter there hardly has adequate room, so most of us are getting wet. BTW, that was train cars number 1481 and 1483, in case any MUNI hacks are reading this. Almost 10 minutes later, I ended up getting back on the first train as it showed up on the opposite side of the platform, going the other way -- I was getting concerned that the rain would soak through my backpack and get my powerbook wet. On the way back to Embarcadero station, I saw the "next train" -- it was packed; looked like there would barely be room for all of the folks who'd withstood the rain waiting for it. I finally got another train and got to the office by 9:10am. Thanks for the convenient adventure, MUNI!
( Oct 19 2004, 09:36:51 AM PDT ) Permalink View blog reactions
Saturday October 16, 2004
While the practical problems of test-friendliness seem to be addressed by IoC, it sounds to me like some of the morass-of-deployment-descriptors problem are inescapable. Mapping services into POJOs declaratively (as Spring, Hibernate and Struts all require) isn't a walk in the park; until there's a linter and/or metadata/xdoclet support for easing how services can be mapped in given a specific runtime context, the pain of diddling XML config files and finding errors at runtime are going to be part of the landscape. Nonetheless, automated unit testing without cactus is a huge win in my book -- leaving CruiseControl to run unit tests and having Cactus runs reserved for manually-invoked integration tests sounds like the way to go.
I'll be steering clear of the traditional J2EE containers such as JBoss or Weblogic.
( Oct 16 2004, 09:33:02 AM PDT ) Permalink View blog reactions
Thursday October 14, 2004 So many books, so little time
Someday real soon, I'll build a little app with Amazon's ListLookupOperation to integrate wishlist items with my blog and perhaps fold it into an Attention.XML data source.
( Oct 14 2004, 10:36:56 AM PDT ) Permalink View blog reactions
Friday October 08, 2004 See for yourself, Bush's mystery bulge I hope Kerry brings it up in tonight's debate, may be pat him down.
I will occupy- Metallica, 1986 ( Oct 08 2004, 11:49:12 AM PDT ) Permalink View blog reactions
I will help you die
I will run through you
Now I rule you too
Come crawling faster
obey your Master
your life burns faster
obey your Master
Master
Master of Puppets I'm pulling your strings
The hackathon percolated a lot of interesting ideas. One of the bits of feedback that caught my attention was from Chris Fry:
There are some drawbacks however to not having a WSDL and to not using SOAP. (1) You are bound to HTTP; (2) If you version the contract how do you notify your clients? (3) Related to 1, no SOAP Headers; (4) No public contract other than your documentation.
This sounds great, getting Technorati developers out of the wire-protocol-awareness business (unless they want to be in it) is one of my goals for future development efforts with the Technorati API. The direction I'd like to take it is an "all of the above" implementation where API consumers can fiddle with the low level if they want to (via REST w/XML, REST w/XOXO, xml-rpc or whatever interface to du jour is desired) but also provide a SOAP interface for those who want to use WSDL to skip all of that.
We have some work to do internally at Technorati to get us to that point though.
( Oct 08 2004, 10:58:31 AM PDT ) Permalink View blog reactionsCheck it out:
So what's next from Google? An Orkut that isn't all-Brazil-all-the-time?
( Oct 08 2004, 10:31:29 AM PDT ) Permalink View blog reactions
Tuesday October 05, 2004
If you haven't checked it out, stop what you're doing and check out Mount St. Helens right now!
( Oct 05 2004, 10:03:09 AM PDT )
Permalink
View blog reactions
Monday October 04, 2004
Sure, this is always the time of year when the weather gets a little colder, the sun sets earlier and the volume of junk mail catalogs arriving from the postal service goes up. But what makes this Bad is obvious: the Giants aren't playing baseball until the spring. I just can't wait until spring! Any claims that the lack of October play is Felipe Alou's fault is just silly. The last few years' bad deals with Sidney Ponson and Damien Moss and bad luck Rob Nenn have taken away payroll that coulda been used more productively -- those situations are more blameworthy. But hope springs eternal and this spring we should see Jessie Foppert and Jerome Williams in the rotation and if some good luck (and payroll availability is on our side) Moises should be a Giant and batting behind Bonds.
( Oct 04 2004, 10:04:14 PM PDT )
Permalink
View blog reactions
Wednesday September 29, 2004 As reported on by Dave, the chaos really crescendoed last weekend with an electrical outage at the colo facility. The service is on the mend but we still have a ways to go. The database repairs are proceeding. The hardware upgrades are mostly completed and it looks like we're going to setup camp someplace that will be a huge step up from the ghetto colo we've been in.
Here comes the sun. It's alright.
( Sep 29 2004, 10:37:10 AM PDT ) Permalink View blog reactions
Monday September 20, 2004 One of the recent hassles I've had recently was with a hardware migration that needed to proceed quickly. The clock was ticking down on the disk capacity utilization on some key database hosts. Now suppose one of the sysadmins wanted to perform "preventitive fsck's" and "table consistency checks" -- when you're dealing with over 100 GB (closer to 200 GB, actually) of data, these are not quick propositions. In fact, they might take days. Ergo, just not feasible. Given the time, would it be optimal to sanity check every subsystem's functionality? Perhaps. But when struggling to beat the clock, you just gotta say, "Not now, Poncho!" Sometimes the only effective action is fast action.
First of all, the only times I've ever needed to do a reiserfsck has been after a cold power loss (and reiser is usually fine even after one of those). So the fact that this sysadmin wanted to do a reiserfsck "preemptively" made even less sense. As far as doing a table consistency check, with innodb this is never needed on an anticipatory basis. In my experience, innodb either is able to keep itself consistent with its own journaling or it's just hosed... not a lot of grey in between. Again, the only exception has been in cases of a cold power loss. Sure, sometimes other hardware problems, low level disk defects, will manifest themselves as problems with the filesystem or a database's data file. But usually there are other indicators as well (kernel complaints in syslog, etc). But even with the dependency stack accounted for and checked, it's no guarantee against failure.
Sometimes the optimal course is just the fastest one between where you are and where you need to be. Choosing the deliberate and cautious route, dwelling on unnecessary optimizations, may in fact be the slow and steady road to.... failure! In this case, if we'd followed the course of doing every unnecessary system check possible, we'd have run out disk space and crashed these particular databases.
Stop optimizing. Just shut up and get it done already.
( Sep 20 2004, 11:51:27 PM PDT ) Permalink View blog reactions
Saturday September 18, 2004 Although Apache 2.0 has been out for a while now, the risk for mod_perl and PHP based developers is still high.
I just ran into an interview I did with Linux Guru about two years ago. I was relatively upbeat at the time 'cause I expected that the innovations in Apache 2.0 would be sufficiently compelling that it'd drive mod_perl and PHP developers to "get on the bus." Sadly, that hasn't happened.
If you're serving static content or need to wire up an external application server (i.e. Tomcat via mod_jk), then Apache 2.0 is definitely the way to go. But the vast body of mod_perl modules on CPAN that work well with 1.x but don't with 2.0 does not bode well. Thread safe Perl and PHP development is really the key to Apache 2.0's success within that development community, it seems like it'd behoove RedHat, IBM and other vendors who've bet a lot on the open source integration market to spur this development.
( Sep 18 2004, 05:50:24 PM PDT ) Permalink View blog reactions
Wednesday September 15, 2004 I'm fan of expressing business logic cleanly separated from display logic. It becomes especially important for managing CRUD cycles within an application. In j2ee-land, MVC with struts is the de-facto standard for doing those things and it works pretty well. However, in land o' LAMP, no such standard exists.
I'm currently looking at using Maypole with HTML::Mason. But it looks like (oy!) TMTOWTDI decisions are to be made there:
Tuesday September 14, 2004 I keep hearing from mercenary recruiters from Amazon about technology jobs requiring mod_perl and HTML::Mason knowledge (I tell 'em "No Thanks But Say Hi To Jon For Me" -- I doubt they ever do) . Hearing that one of the topics of conversation at this year's OSCON was the demise of mod_perl came as quite a surprise.
According to the Journal of jjohn, mod_perl's problem is that it's a CGI enabler (psychobabblisticly: it allows web developers to indulge in Bad Things). jjohn sez...
Now, it's not that mod_perl suck (it doesn't) or that it's not useful in some situations (it is), is just that MOST PEOPLE ARE SIMPLY DOING CGI CRAP. That's right, stupid CGI + HTML is a kind of universal Microsoft Fundation Class that works for programmers of all languages.He goes on to give PHP a little lovin'
PHP is a terrible language. Perl has long suffered with the albatross of its highly syncretic origin and it's "organic design". However, PHP is a lot worse. It's a kitchen-sink language where crazy things like mysql routines and GD libraries are part of the core language. While objects were around in PHP 4, few PHP systems use OO style. To put a fine point on it, most of the PHP apps I've looked at are poorly written and a bear to debug.
And yet, PHP is frequently a better choice than Perl for web apps.
Besides the close association to the CGI aspersion, the big problem that Perl and mod_perl suffer from is that it's too damn easy to build templating web component frameworks. HTML::Mason, HTML::Embperl, Template Toolkit, HTML::Template, Apache::ASP and so on and so forth ("but wait! there's more..."). How many goddamned many of these do we need? The overlapping similar-but-different functionality borders on Not Invented Here neuroses. And so at a certain point, TMTOWTDI is a liability. As a programming language, PHP suffers from a similar TMTOWTDI blight. For instance, for file path values, there's pathinfo but there's also dirname and basename, which are completely redundant.
So if you're going to use a language and component system that sucks (and they all do), perhaps the thing to do is to use the one that sucks the least. Despite the OO additions to PHP language for PHP 5 (notably absent: real exceptions), it's a tough case to make that PHP sucks less than mod_perl. Maybe the PEAR libraries and the Smarty component system make it a little more usable. Maybe. Perhaps mod_perl's maturity and Perl's general usefulness inside and outside of the web environment is an enduring asset. But I'm not convinced one way or the other. Screw it! Use mono and ASP.Net!
OK, probably not.
In the meantime, I'm looking into combining Maypole and Mason to get a framework together to support the applications that MVC is appropriate for.
( Sep 14 2004, 04:32:31 PM PDT ) Permalink View blog reactions
Monday September 13, 2004 I've been looking for a way to enable internationalization and localization ("i18n" and "l10n" as they say) uniformly for both Java and Perl code bases. I really like the simplicity of Java's ResourceBundles but it looks like a build tool will be necessary in order to maintain one master lexicon for it and for the Locale::MakeText Perl counterpart.
Currently reviewing Locale::Maketext::TPJ13 and java.util.ResourceBundle for some insight into how to keep it all together. Maybe there is a wheel that's been invented already here, if I have to invent it, it will be an ant task!
( Sep 13 2004, 01:36:37 PM PDT ) Permalink View blog reactions
Sunday September 12, 2004 In today's San Francisco Chronicle there's a review of a pair of books I've had my eye on but haven't had time to read (thank goodness for book reviewers who have the time!).
The Chronicle review, THE HIGH PRICE OF OIL: We pay at the pump, we pay with taxes, and some pay with their lives, raises important questions about the future of, well, modern civilization. Are we heading towards a Madmaxian world of armored mobile homes and gasoline pirates? Do we want our foreign policy to be directed by our desire to preserve our freeway traffic jams? It doesn't seem too far fetched that, were Bush re-elected, we'd be pushed further along into an era of aggressive oil supply protection. A future where the military mission of young Americans will be security duty will be on the oil rigs, for the refineries and tankers and the other instruments of the petroleum industry isn't a future I want to our tax dollars underwriting. The future belongs to hydrogen fuel cells, solar energy and other alternative energy sources.
I'll probably pick up the Klare book on my next book shopping episode.
Other reading of interest:
In the meantime, I'm envisioning an energy-plentiful post-petroleum world, carpooling, taking BART to get to work and telecommuting when I can get away with it. To hedge my bets, I'll be at the Winnebago dealership today to look for rig that I can harden and equip with rocket propelled grenade launchers.
( Sep 12 2004, 11:59:21 AM PDT ) Permalink View blog reactions