On the Edge of Occam’s Razor

Due to my ISP, PlusNet, trying to kill off their CGI hosting platform, and preventing users from logging in (which I need to do, to update WordPress, take backups, etc.), I’ve decided to move this blog. All pages hosted at ccgi.gumbley.plus.com will eventually die (reminding us of the impermanence of all things).

What better place to move the blog to than WordPress.com?

Henceforth, my blatherings may be found at http://mattgumbley.wordpress.com – a much nicer URL!

Also – and I’ll start to publicise it on the new blog – my new .org under which I hope to share my larger scale works: http://devzendo.org – a Zendo (Zen meditation hall) for developers who practice agility mindfully.

So, see you over there…..

On Saturday 31st October 2009, I went to see The Enid at the Elgiva Theatre in Chesham – the first concert I’d been to for many years, and was completely blown away by their performance. I saw them many years ago in Stoke at the Wheatsheaf – a venue far too small for their epic music.

The Elgiva was a great venue; not packed, although some would have preferred there to have been seating. The sound was excellent, and their performance was tight, performing older pieces as well as the latest ones.

In case you’ve been hiding under a rock since 1973, The Enid are undoubtedly the absolute masters of their genre, ambient, symphonic, euphoric rock – in fact, it’s difficult for me to characterise them, as they’re unique. They’re progressive, but as Robert John Godfrey points out, they’re not prog rock. Their music isn’t like anything else I typically listen to – I have many favourite musicians and groups, but The Enid transcend that – their music is on a scale that could provide the soundtrack to a life. I’d wholeheartedly agree with Tommy Vance who stated that Robert John Godfrey is probably one of the finest composers this country has ever had.

An old poster I had from one of the Stoke gigs stated “It is the mark of a closed mind that knows what it likes, and likes what it knows.” – if you are interested in expanding your musical tastes in directions you never knew existed, check them out.

When I’m on my desert island, could I take my Enid collection?

When I started this project (when it was still a single desktop application for home financial management), I hoped that it might attract a community of users and developers.

Writing a software product that has quality is no small task. Anyone can lash together some code quickly, slap it up on SourceForge and hope – and many people do, leading to what David Heinemeier Hansson describes as “two people playing with it for five minutes” (See previous post, with quotes from David taken from the FLOSS Weekly Podcast #79 – http://twit.tv/floss)

Creating well-designed, aesthetic software, that meets the users’ needs, with all the features they have come to expect from modern desktop applications, plus excellent documentation, community infrastructure (website, forums, mailing lists, etc.) that’s translated and localised to their language and locale and having a coherent architecture that passes intensive quality metrics – is a lot of work for one developer.

I admit I have given no thought to internationalisation or localisation, at present. I will dedicate a release cycle or two to it later – I hope that the project will be of interest to developers who have more experience of these issues than I do, and who speak languages other than British English.

As noted in earlier posts, the goals of the project have diversified into providing a general-purpose embedded-database-backed desktop application framework. I hope that this might be of broader use to more users and developers than the narrower opportunities afforded by simply a financial application.

All the development to date has been stored and built on a server on a private network – as I open up the project, these facilities will have to move to public hosting facilities. Again, SourceForge and the like offer attractive facilities to developers here, but are not at all usable by end users (See How to successfully compete with Open Source software by Patrick McKenzie for a critical view of SourceForge as being unsuitable for end users.).

I’m currently using Subversion as a version control mechanism; this has been great so far, but the central repository on the private network is a bottleneck to my preferred style of working: rapid commits of discrete pieces of work. I do the majority of work on the project on the train, only connected sporadically via GPRS/HSUPA modem. Since the private network is not accessible remotely, I can’t commit when I’m travelling. All items I’ve been working on during commuting are in my working copy, and require sorting out into discrete commits. Sometimes there are overlaps between items, and if I were to open this to the community, it would make code review of independent pieces of work difficult. Not having the entire history available whilst working disconnectedly is also less than ideal. So before opening up, I’ll convert the projects to a distributed version control system – I’ve already chosen Mercurial: Git is not a consideration since its developers do not seem overly concerned about cross-platform issues; Bazaar was a contender, however Mercurial passed the “Has an O’Reilly Book” quality test. (Mercurial: The Definitive Guide.) I don’t know how easy it would be to set up my own repository hosted on my own public server, so I’ll enquire about hosting the code on BitBucket.

Another small barrier to entry for other developers is the domain from where it is available. I have a perception that people might be less likely to contribute to a project that seems like it only belongs to ‘a guy in his bedroom’: The gumbley.me.uk domain and uk.me.gumbley Java package I have been using to date is like a private house: you may come in if invited. A community hall is open to all, more welcoming. Therefore, a “proper” domain has been obtained – which, in accordance with my preference for polishing until ready, will be revealed later. Suffice to say that I intend to develop it as a community website, made highly accessible primarily to potential users of the software, as they shouldn’t have to wade through a developer-centric design. I’m using Drupal to manage it – a thoroughly fantastic CMS; I recommend it highly. There will also be a grand renaming of the Java package structure of all the projects prior to hosting on BitBucket.

The build infrastructure is more than just source code control: I’ll need:

• a continuous integration server (are there any publically hosted Hudson servers?)


• a Maven repository (I am aware of Sonatype’s OSS repository, and will be enquiring about this – there are a few artifacts I’m using that are not available from the central repository, and a few I have deployed in odd ways on my private repository – these issues will have to be addressed first).

I have been reading two excellent books on the community aspects of developing Open Source software: Karl Fogel’s Producing Open Source Software, and Jono Bacon’s Art of Community. I’d thoroughly recommend these to anyone contemplating an Open Source project – especially if they want it to have any longevity. Another series of articles that will help the Open Source craftsman are by Kirill Grouchnikov, and may be found at Pushing Pixels: Party of One: Surviving a Hobby Open Source Project. (Note that I disagree with Kirill’s point about documentation: “If your documentation does not have an immediate answer – it’s your fault. In fact, the documentation is a chicken and egg problem – if your library needs documentation, it’s unfortunately your fault as well.” – I believe documentation to be an essential part of software, especially frameworks, languages and libraries: how many people would be Ruby programmers now, if Programming Ruby had not been published?

The list of technical things I have to provide have ballooned following their advice: bug tracker, mailing lists backed by forums for nontechnical users, IRC (could be difficult, given my semi-connected, only-on-the-train-and-frequently-going-through-tunnels mode of working).

There are also the political / governance aspects that must also be addressed.

There’s a lot to do – and at the moment, only me to do it. No wonder then, that only 15% of projects identified by Kirill in his articles achieve viability.

I’ve been deliberating over when to release the framework for some time. The accepted wisdom in Open Source projects is to “release early, release often”. However, I’d prefer to wait until the code is at a sufficient level of completeness, quality and usability.

I’ve found very few developers in agreement with this; they favour an early release, so that feedback can be given. I’d rather polish it to my own satisfaction first – and not potentially waste some of their time, as the code is in considerable flux.

One voice I heard recently who seems to agree with my stance was David Heinemeier Hansson, creator of Ruby on Rails. David has strong opinions about design, and it shows throughout Rails, which is a wonderful framework (part of this wonderfulness comes through from his choice of Ruby). In an interview with Randal Schwartz on the FLOSS Weekly podcast #79, he says, on the work leading up to the release of Rails:

I didn’t want to put something out there that was going to languish, like a ton of other Open Source projects. If I was going to put something out there, I’d put it out there with enough effort that it’d have a fighting chance to make a difference for people. So, I think for about six months actually, after I had a completely working framework sitting on my hard drive I said ‘screw release early, release often’; I’m going to polish the hell out of this thing. I’m going to write the documentation; I’m going to build up some marketing sort of intensity around it; I’m going to do all of these things that a lot of Open Source projects do not. And as a result of that, they have a tendency to not get very far. I wanted this to have an impact; I wanted this – for my time to be worthwhile. I was not going to put all of this freaking effort into bundling this stuff up that meant two people playing with it for five minutes; I mean, that’s freaking wasted effort. So, it took about another six months of me talking it up, polishing it up, writing documentation, doing all these things before I felt ‘Hey, it’s good enough to release it’.

I’m greatly reassured by this – and if polishing until it’s ready worked with such spectacular success for Rails, then perhaps “release early, release often” is not always the right path to follow.

Other interesting parallels here are that Rails is a framework extracted from an application, as is the MiniMiser framework; Rails allows you to write database applications easily, and this is the intent of the MiniMiser framework, albeit with Java and targetting desktop applications; David is right about Java being verbose; I quite like it for its static typing, and the excellent IDEs and toolchains – I’ve never felt totally happy with the dynamic languages in this regard, although I feel that Bruce Eckel is again right, when he argues for “strong testing, not strong typing”. I just prefer both. And if Java isn’t entirely your thing, then rest assured, I’m trying to enable users of the framework to work in JRuby.

There’s also the issue of control over the project, and singular vision that I agree with. David, again:

In the beginning, I could absolutely control the vision of where this thing was going to go, which is also why, for about at least a year, Rails had one committer – me. There was plenty of people submitting patches, but there was one committer, and I was that, because I wanted to make absolutely sure that this had a clarity of vision; it had a very targetted design. I did not want it to become this sort of mess that in many ways was what drove me away from PHP. [which is] wonderful [in many ways], but consistency of vision and design is not [one of its qualities]. It’s just a grab-bag of so many people injecting their ideas into how this thing is going to be designed. I wanted something that was a ton more streamlined than that. And the only way I knew how to secure that in the beginning was that I was going to be the one making the decisions.

So it’ll be a little while before I’m ready to publish – but the day is definitely approaching.

A picture really does tell a thousand words. These are the words in the classes of the current version of the MiniMiser framework – only the main code, not the unit tests (it’d look very skewed by the word ‘Test’, if I included those!).

find . -name "*.java" -print |
~/utils/bin/wordleize-java-class-names.pl

	
#!/usr/bin/perl

while (<>) {
  chomp;
  my $class = $_;
  $class =~ s-^.*/--;
  $class =~ s-\.java$--;
  $class =~ s/([A-Z])/ $1/g;
  $class =~ s/(^| )[A-Z]( |$)//g;
  $class =~ s-^ +--;
  $class =~ s- +$--;
  print "$class\n";

}
	

The foundations of the application were built by considering a few of the first use cases:

• The application should have a single main window, whose position and size is saved on shutdown and restored upon startup.


• Users want to create new databases, with optional encryption. I used a wizard framework for this, as there were several choices to be made.


• Users want to open databases, being repeatedly prompted for the password if it’s detected that the database is encrypted.


• Users want to open multiple databases simultaneously, and switch between them.


• Users want to be able to close the currently selected databases, or all open databases.


• Users want the set of open databases to be saved at shutdown, and upon restart, those databases should be re-opened, with the last selected database being made current.


• All the above should be accomplished from the application menu, with the appropriate menu items being greyed out (disabled) when not appropriate. e.g. If no database is open, the ‘File/Close’ item is disabled;


• A small number of recently-opened databases should be present under an ‘Open recent’ option of the File menu; the order of this should change to reflect usage patterns.


• This necessitated the storage of user preferences in a file, separate from any database


• Once a database is open, allow the user to choose from several views into that database, effectively, the main screens for the application, each view being on a separate tab in the main window. Remember the open views, and the currently active one, restoring it upon restart.

Some software components were chosen during this initial work: Log4J for logging (I’ve always used it, and that’s not going to change just because Sun wrote their own system and shipped it with Java) JUnit 4 and EasyMock for unit/integration tests; Spring for dependency injection; H2 as the embedded database, accessed via Spring’s JDBC Templates (with the possibility of using Hibernate if the DAO-based approach became too onerous); Swing as the GUI framework; the Netbeans Wizard library for wizards. I’m using the JGoodies Plastic Look and Feel for Windows and Linux; Quaqua for Mac OS X.

Build tools: Eclipse, Checkstyle, EclEmma, m2eclipse. Also, some infrastructure: Subversion for version control; Maven 2 for the build; Hudson for continuous integration. Numerous metrics and static analysis tools running in Maven and Hudson to aid investigations into quality.

Writing the code for these led to several other pieces of work, some of which bordered on YAGNI (You Ain’t Gonna Need It), but were essential in any ‘quality desktop application’:

• There was no actual financial management functionality present at this stage, and so the database schema would inevitably need to change over time to incorporate new functionality. So, ensure that the code and database schema versions were recorded in the database, to allow for schema evolution upon software upgrade.


• No application is complete without an About Box, but this one was useful, giving developer credits (essential, if a community was to form around the application), and license information.


• Although using test-driven design and development, with continuous integration throughout, there would inevitably be times when something went wrong and a user would have to repair their database, so a rudimentary facility for diagnosing database problems was added, in the form of a special view screen containing an SQL console.


• This view wasn’t to be available unless explicitly requested, so a Tools / Options dialog was added, to which parts of the application could contribute options panels.


• The settings chosen in the options panels were stored in the user preferences file.

I considered the initial user experience, and how the user might engage with the developer community, send feedback/bug reports, and receive notification of new releases:

• Provide a ‘welcome screen’ that explains the overall facilities of the application. Show this automatically on first startup, but not on subsequent startups; allow it to be started from the menu.


• Provide a ‘What’s new in this release’ screen that shows a nicely-formatted change log of changes and enhancements in this release of the software. Allow this to be easily shown from the initial welcome screen, and the menu.


• Allow the detection of fatal exceptions via a problem reporting dialog that would provide a user-friendly synopsis of the problem, and sufficient detail that could be cut-and-pasted into an email, to be sent to the developers.


• So that users might discover new releases of the software, add an update availability checker. Note: this is not an update download system as in Firefox


• If an update is available, inform the user via a nicely-formatted display of all new functionality from the version the user is running, to the latest release version.


• This update check would have to go out over the Internet, and some users are rightly concerned about applications that ‘phone home, so before any update check could take place, the user must be asked whether this is acceptable to them; this choice must be stored in the user preferences.


• If they allow the update checks, schedule a background check for updates (only runs once in a given day), and also allow the update check to be manually triggered from the menu.


• Applications that spew many popups into the user’s face when they start up are infuriating if you just want to get work done. Although it’s inessential to the correct operation of the software, I felt that the welcome / what’s new messages on first startup / upgrade were important enough to be shown automatically. An update check would happen soon after every startup, and if the user opened the application each day, this could rapidly become tiring, and they’d want to turn the feature off. So, a simple message queueing system was introduced such that parts of the system that wanted to get the user’s attention would post a message to a queue. The presence of any messages in that queue would enable a button on the main window that would pulse gently, but not in an obtrusive manner. The user could click the button and see the message list – or, they could completely ignore it.


• Allow for certain messages to be presented with a ‘Don’t show this again’ checkbox whose state is persisted to user preferences.

That’s quite a bit of functionality, and apart from a few screens in the ‘new database wizard’, there wasn’t much in the way of personal financial management functionality present. It was time to rectify that.

However, it was mid-December 2008, and Christmas was approaching. The above code was written using an old IBM Thinkpad running Ubuntu Linux, on the train during my morning commute. (This ‘hour a day’ will be a topic for another post) Over Christmas, the development office kindly provided by South Eastern trains was no longer available, and any spare time I found was spent working on another ongoing project for a while, until mid-February 2009. Then, I switched to a Mac as my main development system. I also had an idea for another project – also a desktop application using an embedded database as its storage.

... to be continued.

I’d decided to scratch my itch, and write my own home financial management application. But should it be a web application or an installed ‘fat client’ application?

The web has its benefits: cross-platform; zero installation for the user; data is always there in the cloud; developers get to write in wonderful languages like Ruby.

The web has its disadvantages: there is still cross-browser woe (i.e. IE); having to write too much low-level user interface code (toolkits help, but you’re still trying to work with the modern equivalent of a 3270 terminal); developers get write in wonderful languages like JavaScript; hosting costs the provider in terms of availability, security, backups; if the users’ connection is down, they can’t use the app (although Google Gears helps here – it’s a workaround); if the ‘series of tubes’ is clogged then the user experience is impaired, etc.

One of the use cases I had to consider was encryption of the data. The existing commercial application I aim to replace allows individual databases to be encrypted, and this was something I had to provide. If I were to develop my application as a web application, I’d have to host every users’ data in the same database (MySQL, PostgreSQL, etc.) and ensure there was no way a hacker could compromise anyone’s data – tough. Security between the browser and the server would have to be provided via SSL, requiring a certificate that probably wouldn’t be cheap, especially for an open source project.

The languages and toolkits issue also swayed me away from the web. Much as I love Ruby, I can’t kick my Eclipse/Java habit. I still have the option of using JRuby in the project. Swing UIs can be attractive and responsive – plus, Swing is a proper GUI toolkit, not a series of disparate technologies glued together with duct tape.

User installation of applications is not without problems – the user would also have to install the correct version of Java, or I would have to ship the application with a JRE. I didn’t want to have to do that – so I made the choice that an installed JRE would be a prerequisite. This is a no-brainer on the Mac and relatively painless on Ubuntu. Windows remains a little painful with regard to Java installation, since there is no centralised package management – every application provides its own update installer these days (and there will be more about this later, when I discuss how I plan to address this in the application itself). On Windows, the Java update checker runs in the background, and nags the user to update – as do countless other applications. This might be fine for technical users such as myself, but for the user base I hope to target, it’s awful. Ubuntu and the Mac have a better system for update notification – at least for Apple-provided code on the Mac: you’re notified about updates in one place, and you trust it. Several non-technical users I know ask me “Should I accept this Java Update it’s offering?” These users have to work out for every application that does this whether they should accept. They also presumably get to find out that the mass of implementations of update systems are quite variable in quality – which might be the reason they’re afraid of using them.

Once the JRE’s in place, they’d have to download and install the application itself. This has to be made as painless as possible. There’s a great post about this – see How to Successfully Compete with Open Source Software – that I wholeheartedly agree with; I won’t reiterate the points made there, except to say that on the project’s home page, there would be a prominent link to enable the user to download for the platform of their choice; possibly involving some platform-sensing as is done on e.g. OpenOffice.org’s download page.

As for installation, this had to be tailored to the three platforms:

• On the Mac, you drag the application’s icon to the Applications folder (yes, it’s that simple, it need not be any more complicated, as you’d expect from the gold standard in user-centred computing)


• On Windows, you run an installer, typically a .MSI file


• On Ubuntu, add a repository to enable the app to show up in their usual package manager.

That didn’t sound too painful – although it imposes some additional platform-specific build tool requirements and infrastructure burden on me, but there wasn’t any choice – ‘download from a SourceForge page’ and ‘run this Ant script to install’ are not viable options.

...to be continued.

Things appear to have been pretty quiet round here for a while; the odd post here and there, with microupdates via Facebook. This does not mean I’ve been idle – in fact, I’ve been busier writing Open Source Software than ever.

Now it’s time to start publishing information, and, more importantly, code.

A couple of years ago, I had (and still have) a classic problem that Open Source alleges to solve. We use a piece of proprietary software. This is a very well-known application for managing home finances. We’ve upgraded this a couple of times, upgraded to Windows XP, and it still works very well. However the manufacturers announced in 2005 that they’d no longer be supporting the UK version. This leaves us with several problems. If our XP system needs reinstalling, we’d have to reinstall this application. Would the online activation still work? Would any updates still be available? If Microsoft shipped an update that unfortunately broke compatibility with this application, we’d also have problems. (To their credit, Microsoft are very good at maintaining backwards compatibility with older apps – at the expense of a clean operating system – and this works in the favour of end-users, not kernel developers. We have to be pragmatic: Who’s the customer?)

We could, of course, switch to another application. I was considering Microsoft Money, but now Microsoft have stopped producing this entirely. I looked at several Open Source home financial management packages, but for various reasons, didn’t take that investigation too far: lack of maturity, lack of cross-platform compatibility (the application we use is the last one tying us to Windows XP), ‘interesting’ user interfaces and feature sets. I suspect they’ve improved in the last couple of years – indeed, I notice some new ones have started.

Also in the last few years, the web has finally become viable as an application platform, although cross-browser problems are still there. See Bruce Eckel’s recent post about this. There are some online apps in this area, but I worry about security, connectivity and usability.

So I decided to scratch my itch, and write my own application. I like being creative!

Over the coming weeks, I’ll write about the development so far, and where I take it from here.

...to be continued.

It was only a matter of time, I suppose – first Quicken UK, and now Microsoft Money has been killed off.

Apparently you can get comparable products from your bank. My experience of my own bank’s offerings leads me to differ on this.

I have some concerns about storing all my data in the cloud, especially important sensitive information – I’d much prefer to have this data on my own hard disk, backed up, and encrypted.

Ubiquitous Internet access is almost here, although it’s still not as resilient as I’d like – network outage is still a problem, and we’re still contending for bandwidth. Standards-compliant high-performance browsers are here, and for more performance, there’ll be NaCl soon enough (if you think AJAX is fast, try Facebook Chat!) – however, I feel there’s still a place for the downloaded, desktop application. 

This is one application area where Open Source is an ideal fit. Intuit and Microsoft decided that it wasn’t cost effective to continue producing their products in the face of the economic downturn, and web applications from banks. Open Source however is not so affected by the downturn – my itch is still present, despite the economy. Not much we can do about the web applications – they’ll be developed, and eventually will have reasonable performance. In the meantime, now is the time to fill the hole produced by the exiting incumbents in this area.

Watch this space…..

The magic number seven, plus or minus two. Classic paper. Don’t have time to read it.  Executive summary: you can only handle a small amount of stuff.

Also, you only have finite time. I’ve been trying, moderately successfully, to learn the programming language Haskell, for a month or two. After reading Paul Graham’s Hackers and Painters book, and surveying the collective opinion of the lazyweb, I think it’s likely that learning a proper functional language will make you a better programmer in whatever your main languages are. The Pragmatic Programmers’ advice to learn one new language per year keeps you flexible. (And probably doesn’t hurt their book sales!) I try to write in a functional style (although it’s too painful in Java), but wanted to write in something purer. After a brief investigation of LISP (the obvious first choice, given the source of the impetus to Go Functional) which didn’t survive a bout of Parenthetic Aversion, and an attempt at re-reading old university texts on Prolog, I settled on Haskell (out of Haskell, Scala and Clojure) – mainly due to The O’Reilly Factor.

IMHO, given that time is finite, anything in computing can be safely ignored if there are no good, up-to-date technical books on the subject that are edited-by-actual-editors. There are many fine websites and tutorials, howtos, etc. out there, written by skilled authors – but print carries weight (and not just physically!). There are a great many technical books that have been edited by idiots, or not edited at all: the entire output of certain publishers goes largely ignored based on earlier editing disasters. For me, it has to be an O’Reilly book. Scala has a very good book (one, ATM), and Clojure has a pragprog book coming out RSN. But the presence of Real World Haskell meant that this was a language that had broken out of its ivory tower, and meant business.

The book is available online, and is excellent. I soon found myself wanting further coverage of various aspects of the language, so went searching…

Since most of my study time is on the train, I’ve collected many web pages of literature that I’m hoping firefox will store in its cache for me and not lose if it decides to ditch my session. This means that every time I fire up the browser, it has to go and reload these pages – all 31 of them (plus others, for various other projects). This is not good – I’ve had it lose my session every so often, but the main problem is I now have too much web-stuff crowding my brain. I’ve become tempted by quantity over quality, making further progress difficult.

And so, in a fit of cleanliness, I vow to only have a maximum of five resources open for any one project, at any time.

If it will help anyone else, here are my top five sources of information on Haskell:
Why Haskell Matters
Learn You A Haskell For Great Good
Yet Another Haskell Tutorial
Tour of the Haskell Syntax
Real World Haskell

The haskell.org wiki has all this and far, far more, but the thing I’ve noticed about learning Haskell is that it’s not like anything I’ve done before – going from Forth/C/C++/Java/Perl/Ruby/Python is tractable – switching to Haskell requires extensive mental rewiring: I’m sure it’s all “For Great Good”. It just has to be sipped slowly, and assimilated one concept at a time – something I find the O’Reilly book excellent for. All the pages on monads have been closed for now, and although I wanted to scratch multiple itches (see aforementioned lack of time), I’ll have to forego articles on concatenative languages. Although I’ll be back…

Of course, I have more than one project on the go at the moment and so there are browser tabs open all sorts of other projects – my attempt to start Getting Things Done again must start again soon, methinks.

But all of this spring cleaning got me thinking how I should best organise study, projects, tasks – I need a PIM system that syncs between laptop, PDA/iPod and the web… study is currently organised with a combination of multiple browser sessions, Delicious bookmarks, eBook reader’s recollection of the current books I’m reading, multiple Freemind maps. I hear good things about Zotero as a means of capturing research material. If I didn’t mind being trapped on the Mac, there’s also DEVONThink, which is probably better, but would require me to upgrade to the latest OS, which is a PITA I don’t need right now – also, any tool I invest time and money in today and for the future has to be cross-platform, and open source.

Hmmm….