Bootstrapping IT

June 1st, 2014

Reclaim your inbox

An e-mail manifesto

Over the last couple of years, I've become painfully dependent on Gmail for email and archiving. Being a developer, I know what this implies for my privacy, and the fact that Gmail's domination of email creates a monoculture that cannot be healthy.

However, the Gmail offering is incredible good. I can manage multiple e-mail addresses, store gigabytes of data, search and filter at an incredible speed and always access this from everywhere.

It's impossible to match this by using the email service supplied by my local domain farmer. Over the years, I've made a couple of attempts to migrate away from Gmail, but always turned back. Somewhere along the process, I lost terminal velocity, productivity and worse, my emails.

1. The Client Problem

Thunderbird, the de-facto open source e-mail client, has been slow to keep up with modern standards, as the Firefox organization decided to reduce the amount of resources spent on development. Little alternatives remain in the open source eco system: Evolution does a reasonable job in trying keep up with Microsoft products, but fails to do so for me in usability.

I've tried MUTT and I am impressed by this age-old dinosaur, but the learning curve is steep. Having a local mail client is one thing, but things look even worse if you're used to using a web-browser to read your e-mail. Zimbra is hard to maintain and set up, and offers very little comfort for those fleeing from Gmail. Something like Squirrel-mail really isn't up to my usage pattern for e-mail.

2. The Server Problem

Maintaining and setting up an e-mail server, despite numerous resources and options out there. Before long, I found myself spending hours on multiple configuration files full of acronyms and abbreviations, standards and protocols I had no use for.

E-mail is as old as the internet, and it shows. On the surface, SMTP (Simple Mail eh!) is a simple, straight forward protocol. But traditionally, mail servers consist of a hodge-podge of programs that are either MTA, MUA, MSA, SMTP or a combination these. There's the choice for unix-users, or running a PostgreSQL database just for storing authentication credentials, command line tools for generating config files, encrypting passwords and god knows what more.

3. The Protocol Problem

The most modern standard for transferring e-mail seems to be IMAP. It's a tough protocol that can do everything including the kitchen sink. For me, IMAP seems to be up there in the league of SOAP and XML, together with LDAP and god knows what other enterprise standards are out there.

Compared to setting up a simple REST service serving JSON almost directly out of Elastic Search or MongoDB it feels clumsy and inefficient. My experience in dealing with IMAP confirms this: in all those cases where I had to use it, it has been slow, error prone, crashing my client, or simple unable to import a large inbox like mine within an acceptable time span.

A suggested approach

The migration cannot be done overnight, as there are too many hurdles to be taken at once.

  1. First, build A simple REST API to serve my inbox

Build a thin layer on top of the common "Maildir" format. Browse e-mails for a given account in a given folder. Store credentials as either a config file or simple MongoDB.

Stick with Maildir for interoperability and efficiency (Storing Gigabytes of data in a Database quickly becomes inefficient). Use something simple to index and search mail within a directory.

Have a simple schema to define which account can use what e-mail address for sending and reading. Use something like nodemailer in the beginning, to avoid having to run SMTP on my own. Possibly use "Sendgrid" or "Amazon SES", even Gmail for now.

Use offlineimap to synchronize and have massive amounts of data. Use mu to build a search service per-inbox.

The server should support THREADING like JWZ has described.

  1. A simple web-client that can browse, search and compose e-mail.

Leave the hard stuff out for now, and concentrate on something that is snappy and easy to use. Use page.js and browserify. Possibly explore web components...

  1. A node.js mail-server to accept incoming e-mail

Use [simplesmtp]. Dump mail into Maildir. Synchronize using offlineimap. Implement a middleware to handle Maildir logic.

  1. A node.js smtp server to send outgoing email


Your Password Is Wrong

May 23rd, 2014

One of my little annoyances in my day-to-day browsing is websites telling me that my password is wrong. They offer me somewhat of the following advice:


This example roughly states the following translated into English:

The password must be at least 8 characters long, and contain at least a capital, a digit and a word-character. Do not use spaces or any of the following characters: ?,#&<>:'"?=%\

Now I cannot imagine how spaces, ampersands or dashes would introduce any kind of technological challenge, but mucking about I found that ümlouts weren't allowed either, Limiting my options to a-z, A-Z and 0-9 indeed. The password hint also gives the following example of a proper password (translated): Secret01.

To be honest, I found this rather insulting. The following picture from xkcd illustrates the problem quite well:

So how does a miscreant like this actually becomes part of a large, corporate website, that is supposed to manage payments for my business? I'm not sure, but my guess, someone didn't do his research and spent less then five minutes of thinking about this, followed by a couple of hours coding some algorithm counting characters in a string for both server side and client.

( Then covered with at least two unit tests, one for server, one for client...)

So, what should have been the correct function, you ask me? Easy, wikipedia has the answer. It's as simple as the following formula:

Formula: L * Log N / Log 2

If we translate that to javascript, and assume the length of N is the size of the printable character set of ASCII, the following function returns the entropy:

function entropy(str) {
    if (!str) return 0;
    return str.length * (Math.log(alphabetSize(str)) / Math.log(2));

BADABING! How about unit testing that? To improve upon our assumption that the user can only use the 96 visible ASCII characters we should make N dependent of the size of our character set, and thus introduce a small bonus for using characters outside the ASCII, and less entropy for lowercase characters only:

function alphabetSize(str) {
    var c, size = 0,
        alpha = true,
        alcaps = true,
        digits = true,
        punct1 = true,
        punct2 = true;

    for (var i = 0; i < str.length; i++) {
        c = str[i];

        if (str.indexOf(c) !== i) continue;

        if (alpha && 'abcdefghijklmnopqrstuvwxyz'.indexOf(c) !== -1) {
            size += 26;
            alpha = false;

        if (alcaps && 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'.indexOf(c) !== -1) {
            size += 26;
            alcaps = false;

        if (digits && '0123456789'.indexOf(c) !== -1) {
            size += 10;
            digits = false;

        if (punct1 && '!@#$%^&*()'.indexOf(c) !== -1) {
            size += 10;
            punct1 = false;

        if (punct2 && '~`-_=+[]{}\\|;:\'",.<>?/'.indexOf(c) !== -1) {
            size += 22;
            punct1 = false;

        // non-ascii character set.
        if (c.charCodeAt(0) > 127) size += 1;

    return size;

function entropy(str) {
    if (!str) return 0;
    return Math.round(str.length * (Math.log(alphabetSize(str)) / Math.log(2)));

console.log(entropy( 'correct horse battery staple'));
// 132
// 68

I can only imagine where to take it from here. Display nice progress bars, rainbows, ponies and kittens... and be helpful to your user.

Code is available as a package here and published to :)

Security is not a business decision

May 1st, 2014

As a developer, you're in the business of making decisions, literally: if this, else that.

We often deflect responsibility for the products we build on a number of points for a number of reasons: Maybe we're no good at it (graphic design), maybe we need a specialist (UX person), of course, we want to get payed (Sales), we need to win this war (Atomic Bombs).

Often this is good. The people making the other decisions are probably better capable of doing so then you, and besides, you want to focus on coding anyways. However, when it comes to security, the decision process suddenly seems fuzzy, as doing security the right may bring some extra cost to development time and budget.

When that happens, multiple stakeholders are involved and influence the technical decision process: Sales wants feature xyz to be implemented tomorrow so they can deliver on a deal. UX wants to have a signup flow that gets people in without a single credential, or your public API goes without HTTPS because your boss doesn't get you a good SSL certificate, or worse, you don't bother to figure out how to set it up.

When a decision becomes complicated, we sometimes give up, and ask the others to provide a solution, and this is wrong. As a developer, you are probably the most knowledgeable person in your organization when it comes to security. You know that users pick bad passwords, that rainbow tables exist, that it can be a matter of minutes to hack into any phone, brute force a password database, and facilitate identity theft trough back doors in your software.

You know that any of these things happen regularly and that criminals are out there. And when a breach occurs, you know you're toast. You may have to work the nights and weekends to restore lost data, reset accounts and fix server security issues. New updated versions of your app will have to hit the stores overnight and you can only hope that people will update software in time before you have to go public. Not only is this going to hurt you personally, it will hurt development, and it will hurt your business and public relations.

Therefore, security is almost never a decision you should leave up to management or any other part of your organization. If you can check any of the following items on this checklist:

  • Allow consumers to remain fully anonymous during signup
  • Allow people to submit credentials and data over an unsecured connection
  • Allow your product to be exploited for collecting identities or scamming
  • Ask for too much privileges on social networks, operating system or a phone
  • Don't escape user data on your public pages
  • Forgo on backups
  • Have one master password for a number of things, and forget to change it regularly
  • Include third party software from untrusted sources
  • Keep production data around for development and store it on your local network with guest access
  • Neglect to monitor your application's behavior and traffic
  • Provide WIFI and network connections to members outside of your organization
  • Send sensitive data out by e-mail to co-workers and third parties
  • Store passwords unencrypted in a database, or hash everything using the same salt
  • Trust user input and become sloppy about the input you accept from consumers

You should ask yourself who made that decision. Did you leave it up to someone else to decide on security? Did you allow yourself to be soothed (This will not happen to us, we're too small, we don't have that many users, the data we have is not that important...?) or bullied into releasing a product that contains security risks (We'll fix this in the next release, or we can't be strict about passwords and users because we need to reach our target)?

Remember that in the end, you will probably be held accountable if anything happens, and finger pointing is no use here. The bad stuff already happened, and it doesn't matter who screwed up.

As a rule of thumb, keep close to the following principles:

  1. Don't store production data on development laptops
  2. Use public/private keys to connect to version control and log in to servers
  3. Write down new passwords on scraps of paper, and never submit them by e-mail
  4. Keep important documents in one place, and don't send them around
  5. Store password and account data in a password database like KeePassX or GnuPGP
  6. Submit credentials and API requests over HTTPS. Always.
  7. If you must share your office connection, setup a Guest WIFI with Airport isolation.
  8. Rotate passwords. Keep them strong. Keep them secret.
  9. never trust user input.

As a developer, you probably follow these rules already. Still, you Maye have checked one or two items on the above checklist. I know I have done so, on a couple of occasions. But remember that this is a risk that you allow your organization to be exposed to.

Bootstrapping IT

April 7th, 2014

While bootstrapping a small enterprise myself, I've started to look at saving costs on IT, since I'm now bankrolling this operation. Even if you can spend the many, I believe that this still is a sound practice, as long as you don't start acting penny-wise and pound-foolish.

Here's a couple of my tricks:

Don't use AWS.

Most of the cloud offerings are, in fact, relatively expensive. To rent a medium machine on AWS for a month, you start looking at a bill of around USD 50 per month.

That's just the bare machine, so you may actually spend quite some more for additional services like EBS, SES and what all. AWS comes, for a all of it's freedom, without any real support, no automated backups, and charges you on your credit card. (As a European, I don't even own a credit card currently)

You'll get a lot better deal from places like Linode, or Rackspace. In my case, a good old local provider provides an ultimate deal at less then 10 euro's a month.

When you expect a traffic spike for some reason, serve static pages, and off-load things like a signup form to places like Google forms, Wufoo or Mailchimp. Don't go all out the first time on "scaling your infrastructure", You'll have plenty of time to catch up with those sing-ups in the week after that.

Free is Good

There's tons of free stuff out that that makes for great infrastructure. For example, MMS is totally free as long as you're in the free tier. New Relic has server monitoring for free, so there's no excuse not to use it, and Google Analytics will give you valuable insights from the start.

Sendgrid has a free offer, and so does Mailchimp. If you're not sending out mail more then a couple of times a week, you'll be fine with that.

There's really no reason to pay for hosted domains by Google as long as you're team is small enough for you to help everybody setup a new gmail account and forward their mail to that account.

Asana is great for team collaboration and free as long as you're all on the same team. Bitbucket is free and offers issue trackers, team collaboration and unlimited private repositories.

All of these services provide excellent paid versions as well, and form a sound foundation for any web-based service.

Cheap is better

Business plans for phones are usually a rip-deal for small companies. Ask people to buy their own minutes and give all of your employees a phone allowance. Sales may spend that on minutes, where development may buy the extra data-plan and a beefy android phone, and marketing will the latest Apple gizmo. Everybody is happy and you don't need to deal with that.

When you ask people to bring their own devices to work, you have no control over what they bring, even if it's a virus invested monster. For anything besides graphic design and development, a mac mini will do just fine.

My WLAN does not need an apple logo to work. In fact, neither does my keyboard, mouse, screen, or even computer. I work best on a Linux desktop with plenty of RAM and an SSD drive. I don't remember the last time I actually counted the number of cores, so old machine from the previous generation will do. Chances are your development team will regard Linux for work as a big plus.

In the end, buying everybody a nice tablet for Christmas is cheaper and more fun then providing them with the latest in hardware for work.

Compartmentalized refactoring

March 31st, 2014

I got confronted with the following piece of code:

this.handle = function( err, obj, arg ){
    util.extend( this, util.extend( arg, obj ) );

And I knew I was in trouble. I was floating in a battleship lost in deep-space. There's very little this function can tell us by looking at the arguments and what to expect from them.

When salvaging a spaceship, first thing to do is get live-support back online, at least for some compartments containing vital controls like communication and helm.

Getting a grip on routing and the flow of control is the first hurdle to take when dealing with an inherited code base the size of a small apartment building. This is where I started working on arguments-validate.

The idea? simply to compartmentalize functions, documenting their arguments as part of the working code, and testing all of my assumptions. And make sure it blows up when those assumptions prove to be invalid, so to have a baseline from where to work on. That simple library became my crow-bar for prying open the custom framework I inherited, and was supposed to work on for the coming months.

After isolating the compartments, it was time to start venting in new oxygen one function at a time, until I was able to move from one end of the ship, containing the helm and communication controls, towards the engine room. During this process, the design of the ship did not structurally change.

During my investigations, I found a hidden gem full of valuable data inside the Mongo database. Being able to control the flow of plasma I started moving back to the control room, replacing a custom built routing library with spare parts I brought along: express.js and nginx to serve static content.

This allowed me to cut away large parts of the internal structure, replacing them with modern components while keeping the overall operations going. Lots of effort was wasted trying to re-erect the cargo hold, handling image uploads effectively and scalable as this proved to be an important part of our business, and one of the weakest links in the whole structure, finally replacing it with a flexible image backend.

I replaced the engines with mongoose, cutting away yet layers of code dealing with our database. All this time, I was effectively reducing the size of the overall code base while trying to add new functionality.

One month into this process, disaster strikes. A meteorite the size of a small car disrupted all operations, as Twitter finally updated their API to version 1.1. We now had to deal with an immediate urgency in parts of the code I had not touched before: A single page client app that had done all of the communication with Twitter before.

In the end, I abandoned ship, taking my tools and spare parts with me. We rebuilt our vessel in the months after that in the silent seclusion of an icy planet orbiting a small red dwarf. The lesson I learned? Build small parts that have documentation, so you can hand of jobs to other engineers, create a solid foundation from emerging practice, share knowledge and learn from best practices established by others. Make yourself redundant as a maintenance engineer.

Air locks on your left, lifeboats to the rights, engines in the back. Development flourished and we pushed out new features as fast as we could. For about eight months, we were in a good place, or so we thought. Until the fuel ran out. It was time to abandon ship again, and jump in our lifeboats. We launched and set for new stars.

I've had a wonderful time at Whatser, and I was thrilled to bootstrap a development team and make it work. The love and admiration for Node.js and MongoDB have grown over time, as the community is maturing and the tooling is becoming more awesome every day. It's been a career move I've never looked back on with any regret, except the fact that we did not make it work fast enough and good enough in time.

Many thanks to the people I've worked with for their comradery, energy, love and good food and drinks.

So long, R.I.P. Whatser

Remember variable hoisting

March 26th, 2014

note to self: remember variable hoisting

Remember variable hoisting? sometimes we need a quick reminder:

    function hoist(o){ = 1;
        var foo = o;
        return + 1;

    var n = hoist({});
    // 2

    var m = hoist({bar:5});
    // 2 OUCH

I know. It's a contrived example. But it's also the simplified version of a pieace of real live code I happened to come across today hunting for a mysterious off-by-one error.

There's a reason putting those var's in the top of your function!

Shortening an ObjectID

March 18th, 2014

I wanted to shorten a MongoDB ObjectID to use in URL's for aesthetically reasons and in case a customer would have to read it out loud on the phone in some emergency cases.

Now an ObjectID is a 12-bit binary value that is commonly represented as a 24 character hex string, e.g. something similar as 52690bb1fcfcb7429fbf61eb. That's rather long, especially when you want someone to read it to you on the phone.

One of my favorites for shortening an integer has always been base36:;
    // e.g. 1395177031142 => hsxodqzq

It's a lot shorter and it looks great - only 1..9 and a..z, perfect for reading out loud on the phone and urls, and the .toString method is built in.

Unfortunately, this would require us to convert our 12 bit binary into an integer, and the maximum value for an integer in javascript is 2^53 == 9 007 199 254 740 992.

It won't work:

    parseInt('52690bb1fcfcb7429fbf61eb', 16) < Math.pow(2,53);
    // false
    parseInt('52690bb1fcfcb7429fbf61eb', 16).toString(16)
    // '52690bb1fcfcb80000000000' OUCH

Another option is to use encode the binary string into Base64, we'll gain a couple of characters:

    new Buffer(id,'hex').toString('base64')
    // 'UmkLsfz8t0Kfv2Hr'

This'll gain us 8 characters, ending up with only 16 digitis. But now we have to pronounce things like "Captial U, small m, k, capital L" over the phone. Besides, base64 is not url safe as it includes a number of aditional characters like + and /, meaning they'll appear url- encoded to the user in the works scenario (so now your customer has to read out things like: %2F%3D%2B)

There's a neat alternative called base62, but it requires some custom encoding schema that bypasses the whole conversion to number, and it's not built in to the Buffer object. I've used a module called int-encoder:

    var encoder = require('int-encoder');

    // 62 character custom alphabet

    encoder.encode('52690bb1fcfcb7429fbf61eb', 16);
    // 'HkHMHK2qaXj3jgBb'

    '52690bb1fcfcb7429fbf61eb' == encoder.decode('HkHMHK2qaXj3jgBb', 16)
    // true BINGO

This would also work for base36, but resulting in a slightly longer string:

    // base36 alphabet

    encoder.encode('52690bb1fcfcb7429fbf61eb', 16)
    // 'craxu6pf9mo57l21895' // but 19 characters

After playing around with this for a while, I've come to the conclusion that these really are your only options unless you're going to use partial opbject ids, or store some kind of shorter identifier along with your id (for example, only the last 8 bits).

Writing clever code

March 8th, 2014

Today I published a small module called ellipsize. The problem at hand doesn't seem to warrant a whole module by itself, as the problem itself seems trivial to implement.

However, it is not. My first approach was a simple truncate and appending the string with an ellipse like so:

    function ellipsize( str, max ){
        return str.substring( 0, max - 1 ) + '…';

This was not good enough, as could break in the middle of a word or even leave a space just in front of an ellipse. Still considering this problem worth less then 1% of my active brain, I looked back once, and saw the solution: truncate the string, find the last space remaining, truncate the string up-to that point and then add the ellipsis. Clean and simple in a one-liner.

    function ellipsize( str, max ){
        return str.substring( 0, str.substring( 0, max - 1 ).lastIndexOf(' ') ) + '…';

Blinded by my own cleverness, I did not realize there's quite a bit wrong with this piece of code.

First off, it would not handle empty strings properly, returning a single '…'. It also does not play nicely with strings not containing any white space characters. And last but not least, if the character at max - 1 was an actual space, the outer substring is not needed at all.

If I had thought this over for more then one minute, I could have come up with (at least) some of these edge cases myself, but I didn't. In the end, discovering them and fixing them wasted other people's time and mine.

As always, when dealing with user input, allocate more then 1% of your active brain, because there will always be unexpected input. Check for edge cases to avoid your clever code from blowing up an entire app.

Today I've re-written that function and put it on NPM so you won't have to write it. I hope I thought long enough about it. check it out: ellipsize

Why each is better

February 18th, 2014

When transitioning from perl to node.js, I was delighted to find that "map" and "forEach" were natively available. I had alreay gotten very used to jquery's $.each and felt that I should not need to write another for loop in javascript again.

Not so nice was still the lack of support for iterating over an object, having Object.keys as the clunky workaround, as for( my key in object ) would still give me those pesky jshint warnings about possible undefined keys and all.

My previous experience with large scale javascript was actually coffeescript. Since iterating over an object's properties is a non-issue in coffeescript, I was never confronted with the problem. And I had already made the choice not to use coffeescript on the server.

Using native forEach and map on the server ( not to mention every and some ) is nice, but once you get into the habit of it, going back to the dreary world of the client, there's browsers to maintain and you can forget about these functions. And then I remembered that backbone actually shipped with someting called underscore. If you are not using backbone, there is no reason to use underscore and I would always prefer to go with lodash.

Rewriting a loop like this:

    var value = obj[key];

Using lodash has a lot of advantages in terms of maintainability and compatibility. And, more important: readability.

_.each(obj, function(value, key){

The difference is minimal, but those extra characters can mean the difference of fitting within the 90 character boundary, and not needing to mentally "parse" that Object.keys.

Besides, this will work on any older browser, while the previous example will not. In the end, it will make your code look more similar ( not having to choose a different iteration approach depending on object or array saves a decision as well, and will not tempt you to use for( .. in .. ) )

All of this at the cost of one require. And in some cases, it even performs better too

Things my body can do

February 10th, 2014

I've discovered something my body can do.

Just about thirty years to late, I've recently discovered something my body can do. It can run. It can run and run and run mintute after minute.

Not too long ago I finally was fed up with myself enough to start running. I knew I was capable of running for about three minutes, up to the point where every muscle in my body would hurt and I was sweating like a pig, gasphing for air like a fish on dry land.

So I started running those three minutes, take a break, and run three minutes more. After a couple of these short runs, I discovered that I could do more. Run for five, maybe six.

Until now, I thought running was about having the right state of mind. A good soundtrack. The right time, the right food. This time I approached differently, as something I should be capable of doing.

After running like that for a week or two, maybe three times a week, I wasn't progressing. That was, until someone told me to go running in the dark. I started running my usual routine, two times three, five, six until I reached a really dark area in the park and started to forget about my stopwatch. By the time I felt I needed a break I had run for almost ten minutes.

I've been running in the dark since then. I've enjoyed streaming rain, cooling my face as it was getting red hot, and cutting winds that made my eyes tear. And I started running then minutes, five minutes, and up to now, thirty without stopping, increasing my pace as I feel good.

I never knew my body could do this. And now it can.

Reduce is transform

January 10th, 2014

A blog post by Aria Hidayat attended me to the fact that I'd never actually used Array.prototype.reduce much.

The name always confused me, I associated the function with something like:

var sum = [1,2,3,4,5].reduce(function(sum, n){ return sum + n; }, 0);

A nice but verbose way of summing the integers of an array, and not somethign you need in a day-to-day ad-hoc fashion. ( If I were to sum lots of arrays, I'd propably create a utility called "sum" ).

This is where I stopped reading and thinking about reduce, and I was wrong. While reading the source code of (forms)[] I realized there is a whole use case for sum I never thought of. The pattern where reduce could have been transform instead.

Whenever you write code that does something like the following pattern:

function escapeValues( obj ){
    var escaped = {};

    _.each(obj, function(value, key){
        escaped[key] = escape(value);

    return escaped;

You're transforming one data structure into another. This function feels particularly clumsy because of the extra var statements and having to define the pairs array upfront. Face it, it's a lot of code for something that a perl user would write as:

my @pairs = map { my $p = {}; $p->{$_} = $h->{$_}; $p; } keys %{$h};

( exuse me my perl )

If you rewrite the kv function using reduce, you win quite a lot:

function escapeValues( obj ){
    _.reduce(obj, function( escaped, value, key){
        escaped[key] = escape(value);

        return escaped;
    }, {});

In this case, we only got rid of one line of code, not counting newlines. But in similar cases, rewriting an _.each into a transform has improved my code immensely. When counting the newlines ( I am a fan of whitespace for readability ), the net gain is 3 lines. Thats almost 30% less.

That's 30% more code to fit on your screen. That's 30% less lines to scroll trough when debugging.

There Is No Order in Object Keys

January 3rd, 2014

This is going to be somewhat of a rant.

I've seen perl developers doing this. I've seen ruby developers doing this. In fact, python developers don't do this, they have list comprehensions and tuples for that, but I now see plenty of javascript devs do this.

They rely on the order of object ( or hash ) keys for correct functioning of their software. I've had team members doing this, and told them about it. I've had to rewrite large pieces of code just because someone did not think of it.

Today I was implementing a dropdown that allowed the user to select a country. Of course, the right thing to do is to sort the options alphabetically by value, so when going trough the list of countries you will find your country in some expected place.

We are using the forms to (partially) render our form elements and validate input. The way you define options is by passing in an object of key value pairs.

So I had to tell my team there's no way we're going to be able to display the list of countries alphabetically in a reliable way. I was even challenged by the other developers, as they persisted that as long as you "don't touch" the object, everything is fine. Yeah. Of course it didn't, as forms does quite some touching itself inside.

Now there are two options here: write an ad-hoc solution for rendering "select" elements ourselves and work around forms in the case of a "select". We have a lot of forms with select widgets, so that's not an option.

The other option is to open up a pull request to forms, and allow me to specify options as an array of objects. I'm considering the latter, but given the compact nature of the code it'll take me a while. We left the list of countries in it's wanky, pseudo random state for now ( and solved it by implementing the Google address look-up widget ).

But I'm a sad panda again for discovering this common mistake in a valuable piece of software.