The Power of Prototyping

This is the first in what will be a series of blog posts for the Mozilla-Knight Journalism Challenge. This week we heard from Aza Raskin, former creative lead for Firefox and head of user experience for Mozilla, talking about the power of prototyping for understanding problems and building momentum behind your solution.

Continue reading “The Power of Prototyping”

World leaders delay climate agreement beyond Copenhagen Summit

Today at the APEC summit, President Obama announced that top world leaders would not form a binding, global agreement on CO2 emissions reduction at the Copenhagen Summit this December.

They didn’t give any firm dates, beyond saying that such an agreement is not likely before the second half of 2010, which basically means 2011.

This is both disappointing, and an opportunity.

Disappointing because a binding agreement, even one that is a deep compromise, would at least establish momentum towards a low-carbon global economy.

An opportunity because citizens worldwide have more chances to press for strong action at the time a decision is finally made.

We should now shift our focus at Copenhagen, though not diminish our energy. The message is clear: the scientific community that has brought us so much progress and wealth over the last century are now telling us to drastically cut CO2 emissions or face terrible consequences. The community of the world overwhelmingly backs these conclusions and is ready to take action to switch to a new, sustainable way of living on the planet.

As Mr Gore rightly says, the only thing missing is political will.

Experience importing an existing Drupal site to Aegir

(Cross-posted from the Aegir group on Aegir is a brilliant new framework for managing web sites built in Drupal – upgrading, migrating, enabling, disabling and so, SO much more.)

Hi all,

I just wanted to share my experience migrating an existing site to Aegir, in the hope that others will find it illuminating and it will help them avoid some of the pitfalls I encountered.

Firstly, I made sure my site was completely checked into Subversion, and then checked it out on the Aegir host in the platforms directory as “mysite-1.0”. I had decided after reading the available documentation that the best way to bring an existing site into Aegir is to import the whole Drupal distro as a platform and then migrate the “default” site into another existing platform (the latter step isn’t really necessary, but I wanted to avoid having dozens of platforms for all my existing sites – defeats the point of Aegir somewhat).

Secondly, and before I imported the site, I renamed the “sites/default” directory to “sites/”. Aegir ignores the “default” site. I then created a symlink, “sites/”, because that’s how Aegir does it.

Third, I imported my database. Of course.

Now it was time to import the platform in Aegir. I crossed my fingers, toes and eyes (then uncrossed my eyes because I needed them to see the screen… and my fingers because I needed to type… so at this point only my toes were ensuring good luck came my way, and clearly toes aren’t enough because…)

BAM! It worked!



Time to import the site. Breathe. Breathe. It can’t be this easy, can it?

It wasn’t.

Okay, here I’ve hit my first problem. I wanted the site to have a generated alias to, but I think I either forgot or didn’t see the entry box because this wasn’t baked into the generated Apache config in config/vhost.d, and my site (I later found out) wouldn’t appear when the DNS got changed. Whoops!

Lesson 1: Make sure you enter your domain aliases when you import the site in Aegir.

Secondly, the import task failed because of a missing file on a mysqldump, “/dev/fd/3”. Turns out on Debian Lenny you need to make sure you’ve got udev installed so the smart mysql script can create temporary devices containing mysql credentials.

Lesson 2: On Debian, make sure you have udev installed.

But my script was halting at another point now. It was complaining that I wasn’t using a password for MySQL! That’s weird… checked my settings.php, all seems well. Turns out the issue is that my password started with a “#” symbol. Turns out one of our components here doesn’t like certain punctuation in passwords. Similar problems will occur if your password contains } or ) or various other symbols that could be interpreted as PHP/shell/MySQL delimiters of various kinds.

Lesson 3: Don’t include punctuation in your passwords, for now. Hopefully this will be fixed shortly.

Ran it again. Got a bunch of permissions errors on files, typical stuff and I fixed it.

So, finally I got the migration script to run but, critically, by this point I was running it from the command line so I could see this debug output. This has ramifications later.

When the migration script completed, there were a lot of errors to do with missing packages. See, I’d assumed that drush or provision or one of the other smart tools underlying this system would detect the modules in the source platform and, if they were missing in the target platform, it would automatically download them.

Whoops, nope, turns out that’s not the case, for a whole bunch of really good reasons.

Lesson 4: Make sure you have all the required packages installed on your target platform before you migrate.

At this point I decided to flex my Unix command-line muscles and concocted a simple command line to port and enable the packages. As we’ll find out, I’m not as smart as I thought I was.

cp -R source-platform/sites/all/modules target-platform/sites/all
ls target-platform/sites/all/modules | xargs -y enable

See, this assumes there is just one module per directory name, and that the name of the module is the name of the directory. As y’all in Drupal-land know, this just ain’t so. Chalk one up for Java-guy-learning-Drupal.

So, thinking I was done and readying my martini, monocle, slippers, and wallet for the inevitable glory and prestige, I pointed my browser at the site:


I dropped the martini, the monocle fell out, and my crossed-toes drew blood inside my tartan slippers as I surveyed the damage. Missing images, ill-rendered Javascript, formatting completely gone, fonts in TIMES NEW BLOODY ROMAN. This was no good, no good at all. I was building a professional site, not MySpace. Back to the typing-board.

Okay, time for a few more lessons:

Lesson 5: Copy the themes, not just the modules. I had forgotten that my site’s theme was a Zen subtheme, and you can’t have a subtheme without the super-theme.

Lesson 6: As above, enable all the modules you need. Though this shouldn’t be necessary if the modules are actually present in the target platform when you migrate – Aegir was clearly making a good-faith effort to enable everything when I ran the migrate script.

Lesson 7: If you don’t migrate using the web interface, then your site will be copied as the Aegir user, not as www-data. Then you’ll find that suddenly imagecache can’t create thumbnails of your images, for example. At the very least, do a “chgrp -R www-data sites/”. Hopefully this will be enough.

Lesson 8: There are plenty of Drupal modules that hate having their site renamed. ImageAPI / ImageCache is one of them. Moving from “default” to “” meant that I had to go into the database (shudder) and do an “UPDATE files SET filepath = REPLACE(filepath,”sites/default”,”sites/”);” so imagecache could find the files. This is a design flaw in the way that Drupal or this module handles files, and multi-site should be fixed to handle it. I strongly believe no files should have their paths stored relative to the system root in a multi-site framework, for obvious reasons.

So, after fixing permissions, filepaths, enabling required modules, testing and re-testing, I had my site imported and migrated onto Pressflow 6.14.56 and working well.

Then my client changed their DNS, and the site didn’t appear, and they got angry and changed their DNS back. Because on my final successful migration, Aegir rewrote my apache configuration without the Site Alias of

We re-launch today. Second time lucky. Aegir still has some rough edges for newbies, but I feel really excited that I finally have a consistent way of managing all my sites. Migrating your existing sites into Aegir is totally worth it, and I hope that my lessons above help a few people to get into this wonderful new world.

The Number One Hit for “Arsehole”

(Cross-posted from Tawtof)

Ah Google. Any company that can have three copies of the Internet just kind of lying around is awesome to behold.

And Google Image Search is a modern marvel. All the speed and power of Google Search, now with the nerve-rattling immediacy of images.

However, despite the best efforts of thousands of PhD graduates, image searching is still flat-out wonky.

Text? Ooh yes, we can search text! Is the word I’m looking for in your page? Yes/No. Done.

(I am over-simplifying. The job of then ranking every page on the internet in order of relevance is absolutely horrifying to contemplate, but nevertheless kind of a solved problem.)

And we all know that typically the number one Google hit for a particular word is a definition of that word, or the home page of a company called that word, or very often a serious Wikipedia article summarising the complete history of human endeavour around that word. Serious, relevant, useful.

But image search is another beast entirely.

There is no canonical image for anything. Not even close. Unless it’s an image of an image, and even then it could be a forgery.

And for images of concepts it’s even worse. In text-land, we could point to a bunch of dictionaries for the definition of “wind”, and perhaps even highlight the best definition. By contrast, trying to ascertain the top 10 photos that describe “wind” would be basically pointless.

Same goes for “nancy“, or “shape“, or “old“.

The photos that appear may all contain those things, but they’re hardly the canonical definition of them. The idea that you could rank all the images in the world by relevance to a single word is so painful to contemplate that the very thought would send most PhD graduates flinging themselves off the nearest definition of “balcony”.

All of which is why I love typing “arsehole” into Google Images.

Because – no matter the consequences – whatever comes up is still, by definition, the number one arsehole on the planet.

According to who?

According to machines.

The machines that decide for us that Wikipedia has great definitions (it does), that you have new email and that this is the only new email you have, because we have decided all this other stuff is spam and there’s no need for you to even look at it.

So, what do the machines think is the number one hit for Arsehole?

A young black man on the London tube, looking a lot like he doesn’t want to be photographed.

The story that accompanies the article is hosted on, a self-described “blog that brings you news & views from a critically rational social individualist perspective”. The Samizdats are a crew of London-based Civil Libertarians that sprang out of led by Perry De Havilland, a “trans-Atlantic Entrepreneur” who casually reels off phrases like “The core of what makes this so wrong lies as usual at the meta-contextual level”.

The whatty-what-what?

When you can discredit the core of someone else’s argument using a word and concept you invented then you’re not just dealing with meta-context, but meta-language and possibly even meta-credibility.

Person A: “You can’t say that. You’re a blurglefincher.”

Person B: “A what?”

Person A: “Blurglefincher. It’s a word that means ‘the person who just lost this argument’.”

Perry rightly dismisses a purely-Utilitarian view of the world (as one must with a purely-anything view of the world – cf. Quantum Mechanics), but the cognitive momentum carries him way out into into the intellectual deep-end where every edifice that society has constructed to keep us fed, educated, safe and healthy has to be disposed of because it somehow affects individual liberty.

For example, they refuse to discuss the Second Amendment in the US in the context of gun control because they reject the very notion that a State should be granting us “rights”. Uh oh.

So, back to the black kid who appears when you search for “arsehole”. It appears at the top of an article.

The article (helpfully titled “Does anyone know who this asshole is?“) asks for help finding the young black man, as he is alleged to have assaulted a Samizdata community member called Jackie.

This act has incensed the Samizdata community, who rail against the unhelpfulness of Police, the lack of civility in society, and the fact that they’re not legally allowed to carry around weapons to they could defend the woman’s honour in the most direct possible way.

So what was the nature of the assault? They helpfully provide a link to Jackie’s blog where she details the attack. In point form:

  • She just spent hours detailing the attack to the Police
  • The attack was unprovoked
  • She said that there was some “physical violence”, but didn’t to go into detail
  • She said that the perps followed her down some stairs shouting abuse, and then caught the same train as her (which is how she took the photo)
  • Later, the police apparently arrested one of the men.

Now, at no point do I want to imply that there was no physical attack or that it wasn’t completely awful. I actually feel terrible for this woman even for the details she did describe (the verbal abuse).

The point I want to very very carefully make is that we have absolutely no idea what happened, not even from Jackie herself. No details on whether they had weapons, how the attack came about, whether it was a nudge or whether it was rape.

Hey, you know what? That’s completely okay. It’s a private matter for Jackie, her family, and the police.

And also, apparently, hordes of gun-toting new-Libertarians who want to bring their own brand of hate-filled vigilante justice down upon the head of this chubby black kid for [INSERT CRIME HERE].

Because it turns out that they’re a bit odd.

Turning briefly to the comments on the original article, we find plenty of thinly-veiled racism.

Hmmm. It just struck me that Baker Street is the stopping off point for many heading to and from the Baker Street mosque. You dont suppose……?


And a dash of homophobia along with brutal and luridly-ironic vengeance:

Sickening. He looks like a pansy and pantywaist just waiting for someone to chin jab his flabby ass in unconciousness.

I hope he dies while painfully violated.

And then downright weirdness, like referring to Somalians a “dirt-scratching savages”, or asking that the police force “DOES ITS FRICKIN JOB INSTEAD OF ATTENDING DIVERSITY LECURES”.

But it gets worse than the threats.

I was in Barbican market this afternoon (3;30pm) and saw someone who I thought looked like a spitting image of the shitbag in your picture. As I walked by him, staring as I tried to figure out if I really was looking at the piece of trash, he gave me a hearty and somewhat aggressive “Alright mate!?” About 5”²10”³ 150kg. I would say he’s about 15yo. I suspect he studies in the Barbican area and just finished with school.

Okay, so now we’ve gone from “let’s find this person” to the very brink of “kill all people that look like him”. He certainly fits the description though: black, male, and reacts badly to being stared at in a crowded place. Get him!

Imagine what could have resulted if, as many of the bloggers and commenters claim, everyone in Britain should be allowed to carry concealed weapons? Not only would the crime have been worse, the retribution could have been catastrophic.

If this is the alternative to Government, I’d rather live in Myanmar.

There are so many ironic twists to this story I feel dizzy. For example, if it wasn’t for justice-crazed civil libertarians we would have such a need for governments and police to protect us from their vigilante whims. If it wasn’t for people demanding the free trade of deadly weapons there would be a much need to have one. The fact that they encourage a “meta-contextual” view of the world that respects each person’s point of view, yet pay no heed to the fact that this kid might have his own story to tell about what happened. And so on.

So in the end, in a weird kind of way, Google was absolutely right. The top hit led to biggest pack of arseholes I’ve ever seen.

The Internet is an amazing place. Just to cleanse the mental palate, I leave you with the number 2 hit. Goodnight.


Mah Finance Sector Is Too Big

[Note: Credit for these ideas must go to many people, particularly Joshua Zeidner and Matthew Slater, both of whom are working on mutual credit, LETS and other alternative currency systems. Check out the Matt’s Complementary Currencies module for Drupal and, shortly, Josh’s JUNO API for connecting these systems together]

I’ve kind of tuned out of my daily Huffington Post emails, but today one arrived that I found pretty interesting (The Dominance of the Financial Sector Has Become a Mortal Danger to Our Economic Security). The title is a bit overblown and the article is selective and shrill, but most of the facts speak for themselves.

Some context: In recent months (well, longer than that really) the Republican party in the US has been trotting out the tired Reaganite “tax and spend liberals” rhetoric to attack every policy Obama tries to implement: the “Cap and Tax” policy for regulating carbon output, the “Socialist” medical system that will push up taxes and several even less creative redefinitions of what’s actually happening.

Meanwhile they crow about the stifling effect of regulation on the financial industry, complaining that the reversal of Reagan’s deregulation and Bush’s nails in the coffin will also roll back America’s prosperity to the mid-80’s.

Two problems here:

  • First, that much of the prosperity created after deregulation isn’t real. That’s why they call it a bubble. It’s built on unrealistic speculation.
  • Second, that the US Financial industry is so bloated and powerful that around $1 out of every $12 spent in the US goes to the financial sector, an industry, by some measures, produces very little.

You want to talk about tax? How about an 8.5% tax that goes straight into private hands instead of funding the public interest? A tax paid by every man, woman and child in America every day on everything, and one that disproportionately affects the poor.

Or how about the tax imposed on every US citizen by the private medical industry? The amount of money spent on prescription drugs in the US is rising by 20% every year, yet it would be impossible to argue that each year sees a 20% improvement in their citizens’ health.

There is a place for regulation in both the financial and medical industries. There is a place for government institutions that genuinely advocate and, yes, manipulate these markets on behalf of the common interest, because time-and-again history has shown that completely unregulated markets quickly become distorted by vested interests and that the perceived “wealth” created flows disproportionately to private hands rather than the public good.

I call this effect “trickle-up economics”. See what I did there?

So why not just start regulating?

Have you seen what’s happening in the US and other capitalist economies? Now that they have become deregulated, these industries (and I haven’t even touched on the energy sector yet) have become so powerful and tightly controlled by a few large companies that they can buy political influence and prevent the reimplementation of regulations. Here in Australia, our weak-kneed politicians dare not offend large industries with forward-looking legislation to protect the environment, and so we get pathetic half-measures. Just this week our conservative opposition party almost imploded over the question of whether they should even debate legislation to combat climate change. And this in an age when the overwhelming public and scientific consensus says act now.

So what can the little guy do?

Stop using Money… WTF?

We’ve tried protesting. We’ve tried writing to our representatives. We’ve tried to shop thoughtfully and buy Australian, buy Green, buy Ethical. We’ve subscribed to MoveOn, GetUp, Facebook groups, YouTube channels, mailing lists of every political stripe. Yet political rhetoric, and particularly legislation, is completely, provably out of touch with what people actually want.

I think the time has come to devalue money. Think about it: How often do you use actual cash? Me, not that often. At least half my expenditures are on cards, using credit. My speculative ventures (I have a few) use Australian Dollars to subdivide shares in companies, nominating value. Most of the money I shift around is not real – it’s credit in one system flowing into another. It has ceased to be important whether it’s measured in Australian Dollars, US Dollars, Rupees or bottle-caps. Yet I pay fees to move all this non-existent money around. I am paying the bank some insane amount of money just to decrease a number on one computer and increase it on another, only to have it flow back the other way a week later.

What’s more, modern currency systems are no longer backed by physical resources such as gold – they are all Fiat Currencies. Money has become, for better or worse, simply an accounting system – a way of mutually determining who owes who what.

So what’s to stop me and my friends from adopting our own currency for a particular project? Say, for example, we want to build a web site (I’m a programmer). It’s a club with membership fees. The site needs to be designed, built, documented and maintained and all these things take time, and this is before we have any members.

So we create a pool of credits, say a million credits. Let’s call them SiteBucks. This is just like shares in a company. We allocate SiteBucks to particular tasks – 2000 for a design, 10000 for programming, and so on, as decided by the founders of the venture. The people who work on the site accumulate SiteBucks, and we store each person’s tally in a register. When the site is built, we may have expended 20% of the available SiteBucks, spread around everyone who did the work.

So far, this has operated just like any other kind of speculative venture.

The interesting part comes next. How much longer can you sustain your “virtual” currency? How else could you spend it? As membership of the site grows, or offshoots are created, could you continue to use SiteBucks to pay for stuff?

The answer is, surprisingly, yes. The only thing you need is an exchange rate. Some notion of how your SiteBucks translate into other things, whether it’s currency or apples.

This is where computers come in. Computers are insanely good at storing registers of numbers, lists of members and, most importantly, modelling complex systems like exchange rates. Thank you financial industry – you’ve invented suites of tools that can balance hundreds of currencies off against each other. Now let’s scale that up to thousands or even millions of currencies. Easily done – computers are fast and cheap.

But wouldn’t this just result in chaos? Surely most currencies will not be traded enough to have a stable exchange rate against every other currency

Well, yes, but even so most speculative companies fail before they hit the market anyway. Just because you’ve invented a currency doesn’t mean you’ll grow to the size that it can be meaningfully exchanged.

Also, when your venture reaches a size where it’s producing mutually-agreed value, you can swap your unstable currency for a more stable one. Over the last 40 years or so that would typically have been US Dollars. Now it can be anything. I mean, fuckit, it could be WoW Gold. It doesn’t matter.

Or you can continue with your private market in your private currency and only trade for other goods “at the edges”, for goods and services that can’t be produced internally. This is what’s called a Community Currency or LETS (Local Exchange Trading System). A great site discussing issues around community currencies is

But isn’t this illegal? Actually, no. Creating your own physical currency is illegal. Creating a “points system” that a closed group uses to track mutual credit levels may be in a gray area, but I’d love to see a government try to detect and eliminate it if it’s all happening inside a private computer system.

What about tax? The basis for a Fiat Currency is that it’s the means by which the government demands tax payments. Creating your own currency is a neat way of avoiding paying tax on things like Capital Gains or sales taxes because no money is changing hands. Republicans and Libertarians rejoice! Overdoing it may land you in jail though.

What effect will this have on the Real Economy(tm)?

Oh, a whole bunch! Some positive, some negative. In real terms (i.e. productivity) the economy should grow because of the reduced friction for capital to enter speculative ventures. But the one effect I’m interested in is this: there will be a huge shift in the balance of economic power. The general population will be empowered to shrink the Dollar-economy because Dollars won’t make up as much of the traded currency. This will reduce the influence of companies that have spent most of their energy accumulating financial and political power and not so much of it actually helping people.

In other words, more of the energy you expend working will go towards actual value, and less into hidden, private taxation by banks and other large, opaque, private industries that have flourished in the past 20 years.

I’ll sign off now – no doubt I’ll come back and edit this at some point. Interested on people’s thoughts on this though. I’ve begun meddling in a few virtual currency projects, so if anyone’s interested in hearing more let me know.

A Story About Stories

This is a piece I performed for the Northcote Storyteller’s Club today. It’s been an odd day, preceded by an even odder night – but that’s another story.

I hope you enjoy it, whoever you are. The first part is a long and rambling semi-justification for the story that follows. It’s based on the recent crash-landing of US Airways flight 1549 into the icy Hudson River in New York City.

Northcote Storytellers Gig 2009-01-18

I’m an avid consumer of the news. I consume the news in the same way some people smoke cigarettes. If I have a moment of boredom, I take a long slow drag on international politics and forget about my own troubles for a while. And that pretty much tells you how relevant the informational component of modern news is to our own lives.

But there’s something vaguely unsatisfying about it. Something in the cruelty and randomness of reality that makes news stories less like a long slow drink of water and more like a shot of tequila. News is not contemplative. Journalist’s attempts to draw lessons from it are straightjacketed by the facts. The universality of the tale is quashed because you can’t legitimately retell it in each culture’s own context. If George W. Bush says something horrible that highlights a flaw in the modern democratic systems, and Australians journalists reported it as if Kevin Rudd had said that, there would be hell to pay.

So in order to make these global “important” stories work everywhere they have to be amped up for each market and demographic by swirling computer images and aggravated talk show pundits. The importance and universality of these stories is painstakingly carved by news reporters out of a reality that is far more arbitrary, cruel and random than anyone would like to admit. And the end result is that we’re left feeling a little bit confused and powerless, where a really good yarn might show us how to apply a lesson in our own lives.

So as the bulk of our stories have moved from the exaggeration and customisation of a thousand retellings in a hundred languages to the shoehorning of facts into a compelling narrative that you can maybe sorta kinda draw a lesson from, has anything been lost?

Uh, yes. Look at what happened to the Bible when it went from stories to news.

What I find absolutely fascinating about the Bible is that clearly it went through decades or centuries of oral retellings and written versions, along the way making Christianity one of the world’s more powerful and interesting and relevant religions, until someone decided there ought to be a canonical version. “That’s about fantastical enough! Leave in the bit about the fish, take out the bit about turning a little girl’s ears into legs, and let’s call it version 1 point oh. Matthew, Mark, Luke and John. The Gospel of Ted is too saucy, The Gospel of Steve tells people to ignore the Romans and concentrate on building really nice gardens, and Joanna’s gospel has all those tacky rhyming couplets that will be a bitch to translate.”

Incidentally, the way they selected which Gospels to include way more stupid and arbitrary than that: A Greek guy called Irenaeus, tasked around 150 AD with creating a canonical Bible, decided they should select four gospels from the many available, because there are four corners of the earth, four winds, and four beasts of the apocalypse. The first is completely wrong, the second is a useless simplification, and the third won’t matter until 2011. What an entirely appropriate way to turn an oral tradition into canon, and thus fiction into fact.

And so it was pretty much at that point that the Bible started becoming increasingly boring and irrelevant, and began being enforced by the sword as much as the word. Without being allowed to fundamentally alter in the retelling, it became increasingly difficult for the Bible stories to stay relevant and exciting. Who’s to say that if the Bible had been allowed to evolve as much in its later years as it did in the earlier ones, we not might have had Jesus riding dirt-bikes and making base-jumping to rescue a box of kittens from a runaway speedboat?

So what I want to start today’s news down the road, to retell it in a style that gives it a little room to breathe and grow and come to life. And, since most of the news we hear is so far away and disconnected from our own lives anyway, I reckon this makes the news more accurate and useful.

The Hungry Dragon

There once was a hungry dragon. He ate and ate and ate. He ate the earth to fill his belly, and he ate the air to fill his lungs, but no matter what he ate, the dragon was still hungry.

Every morning at 8am the dragon would taxi out of his cave, his tummy rumbling, his mouths yawning, his tail roaring and his wingtips blinking, for he was hideously deformed and everything was in the wrong place. And he would summon the local chieftains.

And the dragon would ask the chieftains to whisper in his ear and tell him where to fly, for the dragon was blind from his many deformities.

This had been happening for many years. When they were young, the chieftains would fly on the dragon for the sheer joy of being up in the air and surveying their beautiful village. The dragon was young and lithe. He would swoop high in the air and low to the ground. The chieftains would describe for him the world that lay below, the peasants waving from their sunlit fields of corn and orchards of fruit, the church steeple that seemed so high from the ground but so small from the sky, and in his mind the dragon could picture it all, and they were happy.

One day, the wealthiest villagers approached the chieftains and asked if they could climb on the dragons back and fly across the land so they could sell their wares to distant villages.

At first, the chieftains were taken aback. “It’s just for fun”, or “It’s too dangerous”, they would say to one rich man. “It would take too much food, and would make the dragon tired”, they told another.

But over time, the chieftains grew bored, and they grew greedy, and they got mortgages just before the market dipped, and they agreed to take just one or two of the wealthy villagers on the hungry dragon’s back in return for some gold.

This went on for many years, and as time went by and the village grew larger, more and more villagers wanted to climb on the dragon’s back, and the chieftans wanted more and more money.

To convince the dragon to fly ever further, they had to feed more earth into his belly , and more air into his lungs. The dragon grew larger and more deformed, and the chieftains wore silly hats and name tags and stayed in exclusive lounges with free filter coffee and 148 cable channels wherever they went, and the back of his head was painted white and PG-rated films were projected onto it, and some of the wealthiest and most self-important villagers paid ten times as much to sit near the front in a different coloured seat with a cup-holder, and the dragon got fatter and hungrier and fatter and hungrier.

Until one day, much like the previous day and the one before that, at 8am the dragon taxi’d out of his cave. The chieftains climbed onto his neck and whispered in his ear while the wealthy villagers snuck onto his back and flicked absent-mindedly through a dog-eared copy of yesterday’s Wall Street Journal.

And once the chieftains had filled his belly with earth, they ordered the dragon up into the sky to fill his lungs with air.

On this day the dragon was hungrier than ever, and as he rose above the earth he smelled something. It was more delicious than earth, and more substantial than air, and without even a single thought he dove straight for it. It was a flock of geese.

The dragon opened his roaring mouths and filled them with delicious geese. He filled them and filled them until they stuck in his throat and he lost consciousness and he fell towards the earth.

At this point, nothing the chieftains could do made any difference. They shouted in his ear and pulled at his wings but the most they could manage to do was turn him away from the earth and towards a wide but cold river.

With a terrifying splash the dragon dove into the icy water. It filled his nostrils and his throat, washing away the flock of geese and decades of earth and his mind regained consciousness in a snap as his body experienced a completely new sensation. And for the very first time in his life, the dragon felt completely at peace, and he bobbed in the river, slowly sinking down and filling himself with the icy water as they wealthy villagers scrambled from his back.

And before he finally drowned, the hungry dragon had one last, long, slow, satisfied thought:

“Oh! That was the problem! I was thirsty!”

Sending HTML Email with Wicket part II: Converting links

In my previous post, I showed how you can use Wicket‘s HTML rendering engine to render HTML emails by faking a request/response cycle.

In this post, I’ll show you how to use an IVisitor to change image and anchor URLs to be absolute instead of relative. This is absolutely essential in order to make your HTML email work – otherwise all your images can’t be found, and your links point to your own mail server.

The trick is to use Wicket’s IVisitor to add a TransformerBehaviour to all the Images and Links that uses a regex to transform the URL after render but before the page is returned.

The code for the IVisitor is below:

private final class RelativeToAbsoluteUrlVisitor implements IVisitor {
        private final String requestPath;
        private Pattern urlPattern;
        private Class<? extends Component> componentClass;

        private RelativeToAbsoluteUrlVisitor(String requestPath, Class<? extends Component> componentClass, String attributeName) {
            this.requestPath = requestPath;
            this.componentClass = componentClass;
            urlPattern = Pattern.compile(attributeName+"="(.*?)"");

        public Object component(Component component) {
            //if this component is of the specified class, update the URL attribute to be absolute instead of relative
            if(componentClass.isInstance(component)) {
                component.add(new AbstractTransformerBehavior() {

                    public CharSequence transform(Component component,
                            CharSequence output) throws Exception {
                        log.warn("Transforming component output: "+output);

                        Matcher m = urlPattern.matcher(output);

                            String attributeValue =;
                            int start = m.start(1);
                            int end = m.end(1);

                            //convert relative to absolute URL
                            String absolutePath = RequestUtils.toAbsolutePath(requestPath, attributeValue);

                            log.warn("Got absolute path '"+absolutePath+"' from relative path '"+attributeValue+"'");

                            //construct a new string with the absolute URL
                            String strOutput = String.valueOf(output);
                            String finalOutput = strOutput.substring(0, start)+absolutePath+strOutput.substring(end);

                            log.warn("Returning updated component: '"+finalOutput+"'");

                            return finalOutput;

                        return output;
            return IVisitor.CONTINUE_TRAVERSAL;

Then we override the onBeforeRender() routine to traverse the component hierarchy and add this behaviour to the appropriate elements. Note that I haven’t shown how you get the current absolute request URL, as in my system this is proprietary. There’s plenty of example code floating around on how to do that, anyway.

    protected void onBeforeRender() {

        final String requestPath = MyCustomWebRequestCycle.get().getCurrentUrlAsString();

        IVisitor imageVisitor = new RelativeToAbsoluteUrlVisitor(requestPath, Image.class, "src");
        IVisitor anchorVisitor = new RelativeToAbsoluteUrlVisitor(requestPath, Link.class, "href");

        visitChildren(Image.class, imageVisitor);
        visitChildren(Link.class, anchorVisitor);

So there you have it! All the bits and pieces to create HTML email with Wicket. There’s one more catch though: You have to generate these emails in the same process as the Wicket Application. Calling Application.get() outside of the main process results in an error. In my system, I get around this by generating the HTML email source every time the user saves my Newsletter bean, which means that when it’s finally sent (in the background), it just sends the pre-generated HTML. Easy!


Render a Wicket page to a string for HTML email

Something that’s very desirable to do in Apache Wicket is create HTML emails using Wicket’s brilliant component-oriented markup.

I’ve been working on this problem on and off for ages — it’s tricky because of teh way that markup rendering is so deeply tied to the requestcycle, which in turn is deeply dependent on the httpservletrequest — with good reason, too. That’s where Wicket gets its autoconfiguring magic from!

So in order to use Wicket to create HTML emails, we need to fake the request/response cycle. I wrote this convenient method that renders a bookmarkable page (pageclass + pageparameters) to a string:

protected String renderPage(Class<? extends Page> pageClass, PageParameters pageParameters) {

        //get the servlet context
        WebApplication application = (WebApplication) WebApplication.get();

        ServletContext context = application.getServletContext();

        //fake a request/response cycle
        MockHttpSession servletSession = new MockHttpSession(context);

        MockHttpServletRequest servletRequest = new MockHttpServletRequest(
                application, servletSession, context);
        MockHttpServletResponse servletResponse = new MockHttpServletResponse(

        //initialize request and response

        WebRequest webRequest = new WebRequest(servletRequest);

        BufferedWebResponse webResponse = new BufferedWebResponse(servletResponse);

        WebRequestCycle requestCycle = new WebRequestCycle(
                application, webRequest, webResponse);

        requestCycle.setRequestTarget(new BookmarkablePageRequestTarget(pageClass, pageParameters));

        try {

            log.warn("Response after request: "+webResponse.toString());

            if (requestCycle.wasHandled() == false) {
                requestCycle.setRequestTarget(new WebErrorCodeResponseTarget(

        } finally {

        return webResponse.toString();

One other thing that’s desirable to do is change all relative links in the email to absolute URLs — something that Wicket makes super-easy, if you know how. That will be the subject of my next post.

Free SMS service notifications using Google Calendar

Today I had a small revelation.

I was wracking my brains trying to figure out the SMS messaging provider to use to send myself service outage notifications for my clients’ web sites. Given that I have just a handful of clients so far, it makes no sense to use a provider that requires a minimum monthly or yearly spend.

Ideally of course, I’d like to spend nothing at all, and in exasperation I finally threw my hands in the air (they’re detachable) and whined: “Google sends SMS’s for free – why is it so hard for everyone else?”

(answer: not everyone has billions of dollars)

And then came the revelation: Why not create a command-line tool that uses Google’s Calendar API to create events 6 minutes in the future that have an SMS notification set for 5 minutes prior to launch? That way, within a minute you get a notification sent to your phone for free within 1 minute. Sweet!

So, here’s the code (it’s in Java… sorry)

* Simple command-line notification command that uses Google Calendar ATOM API to create
* a single event 6 minutes in the future with a 5 minute SMS reminder
* @author Daniel Walmsley

import java.util.Date;
import java.util.List;


* This is a test template

public class GCalNotifier {

public static void main(String[] args) {

     * Command line args: 
     * username
     * password
     * calendar name (e.g. "Notifications")
     * TimeZone offset (in hours)
     * event start offset (in minutes)
     * event end offset (in minutes)
     * title
     * description

    try {

        // Create a new Calendar service
        CalendarService myService = new CalendarService("GCal Event Notifier");
        myService.setUserCredentials(args[0], args[1]);

        String calendarName = args[2];
        Long tzOffset = new Double(Double.parseDouble(args[3])).longValue() * 60 * 60 * 1000;
        Long startOffset = new Integer(Integer.parseInt(args[4])).longValue() * 60 * 1000;
        Long endOffset = new Integer(Integer.parseInt(args[5])).longValue() * 60 * 1000;
        String title = args[6];
        String description = args[7];

        // Get a list of all entries
        URL metafeedUrl = new URL(
        System.out.println("Getting Calendar entries...n");
        CalendarFeed resultFeed = myService.getFeed(metafeedUrl,
        List<calendarentry> entries = resultFeed.getEntries();
        for (int i = 0; i < entries.size(); i++) {
            CalendarEntry entry = entries.get(i);
            String currCalendarName = entry.getTitle().getPlainText();
            System.out.println("t" + currCalendarName);

            if (currCalendarName.equals(calendarName)) {
                sendDowntimeAlert(myService, entry,
                        title, description, startOffset, endOffset, tzOffset);
        System.out.println("nTotal Entries: " + entries.size());

    } catch (AuthenticationException e) {
    } catch (MalformedURLException e) {
    } catch (ServiceException e) {
    } catch (IOException e) {

private static void sendDowntimeAlert(CalendarService myService,
        CalendarEntry entry, String title, String description, Long startOffset, Long endOffset, Long tzOffset) throws IOException,
        ServiceException {

    String postUrlString = entry.getLink("alternate", "application/atom+xml").getHref();

    URL postUrl = new URL(postUrlString);//was: ""

    CalendarEventEntry myEntry = new CalendarEventEntry();

    myEntry.setTitle(new PlainTextConstruct(title));
    myEntry.setContent(new PlainTextConstruct(description));

    Date now = new Date();

    Date startDate = new Date(now.getTime()+startOffset);
    Date endDate = new Date(now.getTime()+endOffset);

    DateTime startTime = new DateTime(startDate.getTime()+tzOffset);

    DateTime endTime = new DateTime(endDate.getTime()+tzOffset);

    When eventTimes = new When();

    // Send the request and receive the response:
    CalendarEventEntry insertedEntry = myService.insert(postUrl, myEntry);
    System.err.println("Got response for: "+insertedEntry.getTitle().getPlainText());
    for(When when : insertedEntry.getTimes()) {
        System.err.println("When: "+when.getStartTime()+" to "+when.getEndTime());

    //5 minute reminder
    Reminder reminder = new Reminder();


Don’t forget, you’ll need to download the Google Data APIs and put their JARs in your classpath before this will work!

Personally I use this with Nagios. I always use the same args for the calendar offsets, so I’ve encapsulated most of my settings (except title and body) in a script.


export SCRIPTDIR=/opt/calAlert
export PW=mySecurePassword
export CAL=Notifications
export TZOFFSET=10
export ENDOFFSET=12
export TITLE=$1
export BODY=$2

export CURRDIR=pwd

export CLASSPATH=”${SCRIPTDIR}/calAlert.jar”

assumes GData libs are in “libs” subdirectory of SCRIPTDIR

for jarfile in $(ls “${SCRIPTDIR}/lib”)
echo lib/${jarfile}




Training Junk Mail filter using Apple Mail and GMail IMAP Connector

Like most people, I get literally thousands of spam messages a month. I never see them, of course, because they’re filtered out by Gmail’s incredible spam filtering system.

This is all well and good if you’ve got a Gmail account, but what if you want Google-quality filtering for your business or other mail accounts?

Well, thanks to Gmail IMAP support, we now have a massive and incredibly accurate data set for training Apple Mail’s junk filter. Gmail IMAP lets you browse folders other than your inbox. So once you’ve added it to your Apple Mail account list, simply browse to the Spam folder, select all, then mark those messages As Junk.

This simple act will train Mail’s bayesian spam filtering system on everything in your GMail spam folder, no doubt instantly improving its performance somewhat. In my case, the spam folder usually has 3000-4000 messages in it, which is fairly hefty and it would certainly take a lot of clicking to get that kind of data by hand.

Thanks Google!