Saturday June 05, 2004

Managing Terabytes How does google do it? Indexing <sagan>billions and billions</sagan> of documents everyday isn't easy.

Apparently, one component of the Secret Sauce is the Google File System. Some have even gone on to extrapolate some projections of how they do what they do from the SEC filings. The challenges of managing gigabytes terabytes are impressive, indeed. ( Jun 05 2004, 11:58:01 PM PDT ) Permalink View blog reactions

Friday June 04, 2004

Database Replication and the old Beta versus VHS thing I've been plugging up holes in some application problems relating to MySQL. It's not my favorite database but then life is full of compromises. I really like FreeBSD better but the sheer momentum around Linux development is impossible to ignore. So I use it and make the best of it. However, if there was a native JVM for FreeBSD that was as upto date as Linux' I'd consider switching back.

But seriously folks, there are a lot of things that are just toyish about MySQL. In the same way that programming language features like Object Oriented Programming shouldn't be sad after thoughts as it is in Perl and PHP, basic database functionality (foreign keys, transactions and subqueries) shouldn't be the "new features" for a database. I've always liked PostgreSQL for its more complete SQL implementation but it just doesn't seem to have the momentum behind it that MySQL does.

Anyway, my favorite MySQL bug (this is on MySQL 4.0.18) has got to be the silent failure of replication. There are slave database instances whose replication status I assumed were monitorable by doing "SHOW SLAVE STATUS" and observing "Slave_IO_Running" and "Slave_SQL_Running" (each attribute represents a thread that manages the binlog IO and SQL execution on replicated units, respectively) on each of them. Well guess what? That's not sufficient. Both threads can claim to be running and you might even observe the execution positions changing in sync between master and slave. But lo and behold, the real measure of whether or not the MySQL replication is working correctly is to query your application data! For instance, if the timestamps and/or sequences for some key tables are advancing in the master but not the slave, you're hosed. You might need to myisamchk the slave's tables. You might need to simply restart the database slave instance. You might need to ceremonially sacrifice a chicken. Perhaps a little bitch slappin' and sweet whispers will get it going. I dunno. The bottom line is: MySQL might report that it's replication threads are running and its positions are changing but *SURPRISE* your data isn't really updating on the slave! I need replication support for high availability and read concurrency but bugs like that just suck the big one.

So what is to be done? Does MySQL 4.1 fix this crap? Do InnoDB tables replicate more reliably than MyISAM? Is PostgreSQL even an option here? I don't know anybody using PostgreSQL replication. Sometimes when I read the comparisons between MySQL and PostgreSQL, it just makes me wanna throw my hands up in the air and move over to the The Dark Side (AKA Oracle).

Here are some links:

• PostgreSQL vs. MySQL vs. Commercial Databases
  more general fluff that hard core comparison but still not a bad read
• Oracle, MySQL and PostgreSQL comparison
  even though that can't manage to spell the latter correctly, this isn't a horrible report
• Comparison of Oracle, MySQL and Postgres
  another pretty good side-by-side comparison, though it's pretty dated
• OpenACS wants ACID
  out of date and don't even talk to me about Philip Greenspun (I feel sorry for his dog)... I mean, some of my best friends are into TCL but it all seems so trite.
• MySQL Gotchas
  silent failures cases like the "Whoops, no InnoDB table support" one are the sorts of things that drive me nuts

Perhaps citing these out of date sources amounts to FUD. But I don't think so. For me it indicates how much momentum MySQL has... it's not a better technology but the MySQL peoples have managed to leave old critiques behind them and press ahead with after-thought-ish feature additions. VHS won the videotape format wars because it had momentum, not 'cause it's better. I'd be happy to see newer comparisons, contrasts and benchmarks but this is all I've got handy.

( Jun 04 2004, 01:54:11 AM PDT ) Permalink View blog reactions
Comments [2]

Thursday June 03, 2004

Umlaut emerges in the blogspace After a long publishing hiatus ('cept for the periodic email dispatches with tidbits o' fun), Umlaut is broadcasting again!

Back in those days that I cared a lot about metal, Herr Umlaut and myself were making a lot of it happen. These days, I'm not really in the loop anymore (damn, I don't even know who the hell these people are) but it's still plenty fun to hear the echoing reverberations of all that crap that's irrevocably changed pop-culture.

Welcome to the blogosphere, Umlaut!

( Jun 03 2004, 01:29:49 AM PDT ) Permalink View blog reactions

Thursday May 27, 2004

Knowing people for a long time without knowing them at all Occasionally you discover something about somebody that reveals a new and unexpected aspect. Sometimes you only discover them reading someone else's blog!

I've known Wade Grubbs for years but I guess I never really knew him 'cause if I did, I'd have known that he was into Stratocasters; I have an oldie but goodie that looks a lot like that.

At least he has time to play his. Cheers to you, Wade! ( May 27 2004, 12:08:27 AM PDT ) Permalink View blog reactions

Wednesday May 26, 2004

This morning's ride: BMW 540i This has been the week of beemer carpools! Today I rode with a guy who spent most of the time fiddling with his Blackberry and making/taking calls. Still, he managed pretty good time over the bridge and got into downtown San Francisco pretty quickly.

The 540i interior is relatively cramped compared to some of the other rides I've had lately but the seats are still fine enough. Ya know, BART's seats aren't bad by public transit standards but the leather seats in this ride will beat that anyday. ( May 26 2004, 11:34:17 PM PDT ) Permalink View blog reactions


Better pictures on the Treo 600 I flipped the settings on my Treo's camera, the pictures it takes are a lot better now.

In the Picture's application, if you go into the menu and flip the "Picture Size" option in the preferences from "Small" to "Large" you get a little jump in quality (and "Large" is hardly big at all -- I think it's only a .3 megapixel camera anyway).

So, check out this sign I spotted downtown for a lost dog, sad but true! But at least you can read it! ( May 26 2004, 11:23:52 PM PDT ) Permalink View blog reactions

Tuesday May 25, 2004

Class::DBI riddles Sometimes Perl is wonderfully expressive and an extreme productivity tool; sublime, even. Other times, it is an enigma and a riddle rolled into one.

I recently grabbed Class::DBI (v0.96) off of CPAN to help me wire up some simple objects with lots of database attributes. Now, when roughing out objects in Perl, I typically find it quickest to declare a package in the file scope of my working code -- it's like having a runnable test for the package's code write there in fornt of you. Most of the time, that's just fine and dandy -- as the code matures or gets unwieldy, migrating it out into its own .pm module files is fine. But I noticed that with Class::DBI, all kinds of weirdness can ensue if you declare your packages outside of the .pm module file world. I'd seen goofiness with older version complaining about not finding 'db_Main' in the package and yaddada yada. This time around, I tried putting everything in the file scope i.e. so it looks like this:

#!/usr/bin/perl

use Class::DBI;

my @britons = Criminy->retrieve_all;
map { print $_,$/ } @britons;

package Criminy;
use base 'CriminyDBI';

Criminy->table('criminy');
Criminy->columns(All => qw(id foo bar));

package CriminyDBI;
use Class::DBI;
use base 'Class::DBI';
# someday, I'll have a dog and name him Django Hendrix
CriminyDBI->connection('dbi:mysql:test','django','hendrix'); 

But this totally fails...

Criminy can't SELECT 
FROM   criminy
: DBD::mysql::st execute failed: You have an error in your SQL syntax near 
'FROM   criminy
' at line 2 [for Statement "SELECT 
FROM   criminy
"] at /usr/lib/perl5/site_perl/5.8.0/DBIx/ContextualFetch.pm line 51.
 at ./criminy.pl line 5

However if I put each of those packages in their own .pm module file, it's totally happy. I can't explain it (and I really don't want to, it just sucks). Try it out, here's the mysql schema:

CREATE TABLE criminy (
  id int(11) NOT NULL auto_increment,
  foo varchar(32) default NULL,
  bar varchar(32) default NULL,
  PRIMARY KEY  (id)
) TYPE=MyISAM;
INSERT INTO criminy VALUES (1,'Led','Zeppelin');
INSERT INTO criminy VALUES (2,'Black','Sabbath');
INSERT INTO criminy VALUES (3,'Deep','Purple');

Anyway, the short answer: don't take any shortcuts. Declare your packages in their own file and Class::DBI will glide along swimmingly. ( May 25 2004, 11:52:22 PM PDT ) Permalink View blog reactions

Monday May 24, 2004

Riding in the back of a beemer sedan Today's ride over the bridge was quick and comfortable, the BMW sedans are rich in leg room.

And the comfy leather upholstery was certainly a big step up from other recent rides, sometimes casual carpool just rules! Check out the BMW interior. Next year for my birthday, can I get one of these? I'd like the black exterior, tan interior, please. ( May 24 2004, 09:35:57 AM PDT ) Permalink View blog reactions


Moblogging with my Treo 600 The PictureMail service that SprintPCS has always seemed a little bass akwards. It doesn't allow you to send pictures directly to someone but instead can only send them a link to view it on SprintPCS' web server.

That might not be a bad thing per se but when I first wanted to setup my blog with a little procmail magic to post pictures, I was stymied by the limitations of PictureMail. Well, it turns out that Buzznet's moblogware supports fetching the fetching the PictureMail content and posting it!

So I'm gonna give it a go, I'm posting stuff from my Treo 600 there in separate blog simply named Pictures on my Treo 600 (hey, who says these things have to have clever names?).

( May 24 2004, 01:08:36 AM PDT ) Permalink View blog reactions

Sunday May 23, 2004

What would Google do? The blog index spam issue has to be something that our friends at Google have already accounted for.

Google already has a lot of Secret Sauce in their page rank heuristics. Presumably they've already got some automated Google bomb detection but I sure would be interested in knowing the specifics of how they accomplish that. ( May 23 2004, 01:47:00 PM PDT ) Permalink View blog reactions


Further Ruminations on Blog Index Spam Every time I think of easy ways to squash blog index spam, I can think of countermeasures that They Who Eat Their Young might employ to circumvent those efforts.

Perhaps the Vote Links is a bad idea after all (sorry, Kevin). It would be trivial for blog index spammers to catch on and merely add their thumbs-up vote in the links in the content they're clogging the ping stream with. Perhaps there's a sunnyside to this issue, in the same way that comment spam can provide fodder for anthropological amusement, maybe there's something valuable in blog index spam.

Well, no. All spammers should deserve to be treated to a merciless onslaught of pain. Period. ( May 23 2004, 10:54:38 AM PDT ) Permalink View blog reactions

Saturday May 22, 2004

Secure Blog Pings The spammers are onto us. They realize that they can inject their undesired noise into the stream.

If you want to have the appearance of having lots of attention on the net, the barrier to entry is not terribly high. Install some blog software, setup a five or six blogs that link to your site about lonely, sexy milfs in your area, octane booster for your libido and deals-of-the-century for mortgages and [badda-bing!] make a thousand DNS entries for each of these blogs. Then, whenever you update your handful of blogs with your wonderful content, programmatically ping all of the wonderful recipients of update notifications and.... [drum roll]

Ta-da!

You've spammed the blogosphere.

It's my considered opinion that this problem is going to continue to swell as more spammers catch on. As anyone who's had a friend descend into a Mister-Hyde's-gone-AWOL-on-a-heroin binge dirtball can attest, low life scumbags are often quite resourceful. We've already seen that demonstrated contending with comment spam. The underlying problem is that the event capture engines promiscuously accept anything into the stream. It's as bad as having an open relay in the SMTP universe... millions of mail servers in Asia and Eastern Europe can't be wrong!

Blog posts can be fingerprinted and checked for duplication but next thing you know, we're going to require bayesian filters -- I can easily imagine how to defeat the duplication checks; to catch a criminal, you have to have the capacity to think like one, I suppose. Weblogs.com already makes sure it doesn't take a ping for the same blog too frequently within a duration of time, but that doesn't address any issues concerning authenticity.

Anyway, the underlying problem with SMTP is that you can pretty much claim to be anyone and send mail to everyone when the SMTP server is an open relay. By extension, the ping stream suffers from the exact same problem.

I propose that the ping services become a network of trust. Pings should be identified with secure tokens; one way cryptographic hashes with regularly expiring keys would keep just about everyone except the NSA from anonymous pinging. Those found abusing the ping stream could have their ids revoked. That way, the only events making it into the ping stream would be known and identified entities. I believe that the earlier this is put in place, the sooner the blogosphere can wall itself off from purveyors of canned pork by-product products. ( May 22 2004, 05:32:34 PM PDT ) Permalink View blog reactions


UFO tribute a la Pearl Jam Some of my favorite noise is old UFO, and apparently I'm not alone.

Last week (May 14, 2004), according to a report from The Umlaut, Mike McCready from Pearl Jam gigged at The Showbox (Seattle, Washington) in a tribute set that included classic Aerosmith and UFO:

1. Doctor Doctor
2. Lights Out
3. Too Hot To Handle
4. Shoot Shoot
5. Loser
6. Rock Bottom w/Spinal tap tease
7. Sick As A Dog (Aerosmith)
8. Nobody's Fault (Aerosmith)

Sounds like an old time style Headbangin Hoe-Down! ( May 22 2004, 09:47:02 AM PDT ) Permalink View blog reactions

Friday May 21, 2004

Bill Gates' Clue Inkling (But Only An Inkling) I suppose sooner or later it was inevitable that the swarm of motion in the blogosphere might cause a wobble in The Borg's orbit.

Bill Gates stood before a crowd of top CEO's and waxed on about where he see things going. Now, if you'd read any of his drivel (like The Road Ahead) where he positions himself as a technocaster, you know that at least half the time, he's full-o-crap. But that doesn't matter when you're the richest schmuck on this rock, does it?

"Another new phenomenon that connects into this is one that started outside of the business space, more in the corporate or technical enthusiast space, a thing called blogging. And a standard around that that notifies you that something has changed called RSS."

Bill, RSS and notifications are orthogonal. Don't you have a technical editor review your speeches before you get to the podium? RSS is a confined set of metadata. Period.

What would Bill blog? Posts about how great SCO is? How much he hates Mac OS X cause it's so much better than anything he'd come up with? ( May 21 2004, 02:21:27 PM PDT ) Permalink View blog reactions

Thursday May 20, 2004

Neifi Perez Hits A Home Run What's the world coming to when the Giants' sluggers are slumping and mister shallow blooper hits a two run home run?

Neifi Perez has only hit five home runs since 2001. I mean, he's a great infielder but perhaps the threat of having Pedro Feliz become the everyday SS was enough to awaken his inner-slugger. This was their game to lose and for a while there, it looked like that might be the it was gonna go. But here ya go:

	Runs	Hits	Errors
San Francisco Giants	5	10	1
Chicago Cubs	3	8	0

( May 20 2004, 02:43:50 PM PDT ) Permalink View blog reactions


Identifying Blog Index Spam If there's a burst of new links to a URL, it could mean either something very interesting is happening there or perhaps it's blog index spam.

Does the blogosphere need a "report this url as spam" service where any blogs that link to it are immedately suspect? Perhaps a points system... if a blog has legit URLs than the links to spam that have managed to get inserted into the content will should score as strongly. Perhaps this is case for Vote Links as my colleague Kevin Marks will assert. ( May 20 2004, 11:50:08 AM PDT ) Permalink View blog reactions


Blog Index Spammers Must Die! What will they think of next? I wrote a little application to keep an eye on what's under discussion in the blogosphere on a particular topic and next thing ya know, some bozo has posted hundreds of clone blogs to stuff the search index.

Last night at the Technorati Developer's Salon I showed off something I wrote that uses Technorati's API, Who's Talking About The San Francisco Giants, Powered By Technorati. It uses Technorati's search and bloginfo APIs and orders the most recent results by the blog's rank. Lo and behold this morning, there's some bastard who owns the mooseblogs.com domain who has hundreds of aliases pointed to the same blog postings for buying/selling tickets. Among the events mentioned are tickets for the San Francisco Giants, thus killing the usefulness of the search index. The whois specifics for this bastard looks like this:

Registrant:
   WAI
   10105 W. 126th Terr
   Overland Park, Kansas 66213
   United States

   Registered through: GoDaddy.com
   Domain Name: MOOSEBLOGS.COM
      Created on: 02-Apr-04
      Expires on: 02-Apr-05
      Last Updated on: 04-Apr-04

   Administrative Contact:
      Walls, Tom  twalls@kc.rr.com
      WAI
      10105 W. 126th Terr
      Overland Park, Kansas 66213
      United States
      9134848289      Fax -- 
   Technical Contact:
      Walls, Tom  twalls@kc.rr.com
      WAI
      10105 W. 126th Terr
      Overland Park, Kansas 66213
      United States
      9134848289      Fax -- 

   Domain servers in listed order:
      NS1.ABAC.COM
      NS2.ABAC.COM

So if any kind readers in Overland Park, Kansas would like to pay a visit to 10105 W. 126th Terr and kick the perpetrator's ass, the world would probably be a better place for it. I'll loan you a Barry Bonds bat to help get the job done. ( May 20 2004, 10:58:26 AM PDT ) Permalink View blog reactions

Wednesday May 19, 2004

Thermonuclear Pitching Randy Johnson throws a perfect game in Atlanta and Jason Schmidt pitches 9 innings of one-hit baseball. Are the planets in an unusual alignment?

It's about time the Giants show some life in their play. Sad that it was still only a one-run game, the offensive anemia long ago crossed the threshold into patheticness. Thank goodness the Dodgers have dropped five in a row; the Giants are only seven games behind them in the standings! Someone tell Alou to get a clue: ditch Dustin Mohr or Deivi Cruz and bring Brian Dallimore back in the line-up! ( May 19 2004, 09:36:40 AM PDT ) Permalink View blog reactions

Tuesday May 18, 2004

Speaking of outing, Bill Gates uses Mac OS X Well, they do develop applications for the Mac platform.

But it's still funny to see it captured in a snapshot. Wondering what he's thinking about, "Maybe if virii were written with a portability layer, these suckers would get infected too... hrm" ( May 18 2004, 03:36:35 PM PDT ) Permalink View blog reactions


Keep It Real, JBoss The blogosphere is abuzz with the apparent exposure of the JBoss groups' shadowy marketing tactics. Are they really skulking around pretending to be a fan base when in fact they were posting about themselves?

The signal to noise ratio on TheServerSide's Forum have never kept me glued to the conversations but they recently outed users who share locations, which hopefully, by reducing anonymity, will raise the signal level a bit. Though recently enriched with a $10M investment from Intel, the JBoss group may have to spend some of the funds earmarked for engineering on PR instead. There's certainly no shortage of bile being offered for one of the most ambitious open source J2EE code bases, is there? "Anonymous Fakers!" declares Mike Spille. Lots of downloads evidently doesn't translate into lots of love.

Is this a redux of the fraudulent reviews on Amazon thing?

I'm wondering now if the comment-throughput on all of these blogs hosted on jroller.com have crashed their server; those blogs were inaccessible for a while today. Ya know, JBoss can run embed Tomcat as its web container (the server identifier says "Apache-Coyote/1.1" which in all likelihood is Tomcat). Hah, maybe they're running on JBoss!

OK, probably not. But nothing is shocking, is it? ( May 18 2004, 03:20:33 PM PDT ) Permalink View blog reactions
Comments [1]

Sunday May 16, 2004

objects versus ad-hoc data structures in PHP I'm in the midst of changing how a complex data structure is defined. This data structure is returned from a user defined PHP function (i.e. not built-in to PHP or one of its extensions) and the semantics of how the data should be populated needs to change. The quick-n-dirty hacker in me appreciates how easily the data can be assembled and assigned values but my inner object oriented programmer bristles at the kind of mess that has ensued.

There are tons of books from all of the usual suspects (O'Reilly, Wiley, SAMS, etc) about how to program with PHP but leveraging its crude support for OOP and employing best practices around that is seems to be pretty thin in all of the literature I've reviewed. The discussion I've seen on the web about object oriented PHP have also seemed weak; the commenter's who snivel about how complex OOP with PHP is underscore how toyish the use of PHP seems to be in many projects.

Before sending mail or posting comments insulting me for not appreciating PHP's simplicity, make sure you tell your Perl programming friends as well. I've been an adherent to OO in Perl for a number of years (though admittedly fell out of practice while my work was J2EE based) -- my gripes about prevelant practices amongst Perl programmers would be similar the above critique of PHP. In Perl, if you have a big blob of data and you want to find out what the heck is in it, you pretty much have to rely on Data::Dumper or, in PHP, print_r and then pick-apart the output to suss the structure that was created. Perl's man pages have several entries about lists of lists and and a data structure cookbook and so forth but the value of defining objects that provide a structural representation of the data (and of course, semantics) are too often overlooked in real software projects. If the changes I'm currently working on implementing had been using an instance of a class, I'd be looking at the object interface and making changes to the method calls. Instead, I'm looking at a big tangle of opaque data structures.

I could go on and on about how must PHP literature closely binds business logic and UI logic and the hideous morass that that creates, but I guess I already griped about that sufficiently in my high level rant about J2EE versus PHP. <sigh>

The code base I'm currently working on has a long way to go before the business data is captured in clear class definitions and there's a greater level of transparency into the relationships of the things within the application, but we'll get there. ( May 16 2004, 09:46:10 AM PDT ) Permalink View blog reactions

Friday April 30, 2004

Randomizing a list Needing to to randomize the contents of a Perl array, I was just about to write the code myself but decided to snoop around CPAN first. Lo and behold, I found that there was something in the Perl distribution already that did what I need.

As of Perl 5.8, there's a module in the standard library with a boat load of things you need to with a list from time to time. It was nice to stumble upon the List::Util class; it's got a method called shuffle that does the trick. Here's an example of getting a randomized list using shuffle:

use List::Util 'shuffle';

my @list = ('a'..'z');
my @shuffled = shuffle(@list); 
# @shuffled is randomized, @list is unchanged
print "@shuffled\n";

There are more useful methods like min, max and sum that should make dealing with aggregations of values much easier. ( Apr 30 2004, 11:45:45 PM PDT ) Permalink View blog reactions