“I’m mostly blogging this for my own future use, to be able to find how to do something I remember doing before. There you go, future me.”

A friend and fellow blogger Denny Gentry reminded me of the value of helping out my future self by capturing a handy piece of code within a blog post.

Last week I had to back port a fairly complex project from Visual Studio 10 to Visual Studio 9. Microsoft likely has some tools laying around to achieve this, but I’m often tasked with rapidly moving our projects over to linux or other operating systems, and I decided to write a short script to solve the problem of cross platform project building once and for all.

All that’s needed now are cmake, a ruby interpreter, and two lists of files. One list is for libraries (order matters, last dependency last) and another for executables. One caveat is that if the system you’re ultimately moving too doesn’t have cmake you may need to delete any project dependencies on CMakeLists.txt in the created project (I had to). By default cmake watches for changes to this file and remakes project files, which will lead to build errors without cmake’s installation.

create_cmake.rb

create_cmake is a simple ruby script and sample lib which takes a list of libraries (with paths) and executables and generates a CMakeLists.txt file. The CMakeLists.txt file will generate build files on many platforms include linux, os x, and windows in the form of makefiles and visual studio solutions. See cmake.org for all supported build systems.

The create_cmake.rb script

With sample inputs:

library list:

  one_lib
  another/lib
  third_lib

exec list:

  test/test.cpp

In this example to generate the CMakeLists.txt file first a lib.list and an exec.list are needed. See the included repo files for examples of formats (file lists where order matters, put your latest dependency last on the lib list).

In the project directory run:

ruby create_cmake.rb lib.list exec.list test > CMakeLists.txt

Then from the project root type

cd build
cmake ..

And cmake will determine a default build system for your architecture.

You may specify targeted builds by typing:

    cmake

This yields a listing of supported platforms. Here’s an example of what showed up on my home system:

The following generators are available on this platform:

    Unix Makefiles              = Generates standard UNIX makefiles.
    Xcode                       = Generate XCode project files.
    CodeBlocks - Unix Makefiles = Generates CodeBlocks project files.
    Eclipse CDT4 - Unix Makefiles = Generates Eclipse CDT 4.0 project files.
    KDevelop3                   = Generates KDevelop 3 project files.
    KDevelop3 - Unix Makefiles  = Generates KDevelop 3 project files.

To create an Xcode project enter:

cmake -G Xcode (path to CMakeLists.txt file)

Stevey’s Google Platforms Rant
I was at Amazon for about six and a half years, and now I’ve been at Google for that long. One thing that struck me immediately about the two companies — an impression that has been reinforced almost daily — is that Amazon does everything wrong, and Google does everything right. Sure, it’s a sweeping generalization, but a surprisingly accurate one. It’s pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn’t let me show it to anyone, even though recruiting loved it.

I mean, just to give you a very brief taste: Amazon’s recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they’ve made to level it out. And their operations are a mess; they don’t really have SREs and they make engineers pretty much do everything, which leaves almost no time for coding – though again this varies by group, so it’s luck of the draw. They don’t give a single shit about charity or helping the needy or community contributions or anything like that. Never comes up there, except maybe to laugh about it. Their facilities are dirt-smeared cube farms without a dime spent on decor or common meeting areas. Their pay and benefits suck, although much less so lately due to local competition from Google and Facebook. But they don’t have any of our perks or extras — they just try to match the offer-letter numbers, and that’s the end of it. Their code base is a disaster, with no engineering standards whatsoever except what individual teams choose to put in place.

To be fair, they do have a nice versioned-library system that we really ought to emulate, and a nice publish-subscribe system that we also have no equivalent for. But for the most part they just have a bunch of crappy tools that read and write state machine information into relational databases. We wouldn’t take most of it even if it were free.

I think the pubsub system and their library-shelf system were two out of the grand total of three things Amazon does better than google.

I guess you could make an argument that their bias for launching early and iterating like mad is also something they do well, but you can argue it either way. They prioritize launching early over everything else, including retention and engineering discipline and a bunch of other stuff that turns out to matter in the long run. So even though it’s given them some competitive advantages in the marketplace, it’s created enough other problems to make it something less than a slam-dunk.

But there’s one thing they do really really well that pretty much makes up for ALL of their political, philosophical and technical screw-ups.

Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon’s retail site. He hired Larry Tesler, Apple’s Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally — wisely — left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn’t let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they’re all still there, and Larry is not.

Micro-managing isn’t that third thing that Amazon does better than us, by the way. I mean, yeah, they micro-manage really well, but I wouldn’t list it as a strength or anything. I’m just trying to set the context here, to help you understand what happened. We’re talking about a guy who in all seriousness has said on many public occasions that people should be paying him to work at Amazon. He hands out little yellow stickies with his name on them, reminding people “who runs the company” when they disagree with him. The guy is a regular… well, Steve Jobs, I guess. Except without the fashion or design sense. Bezos is super smart; don’t get me wrong. He just makes ordinary control freaks look like stoned hippies.

So one day Jeff Bezos issued a mandate. He’s doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion — back around 2002 I think, plus or minus a year — he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

His Big Mandate went something along these lines:

1) All teams will henceforth expose their data and functionality through service interfaces.

2) Teams must communicate with each other through these interfaces.

3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4) It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. Bezos doesn’t care.

5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6) Anyone who doesn’t do this will be fired.

7) Thank you; have a nice day!

Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.

#6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at Wal*Mart, and is a big genial scary man who used the word “hardened interface” a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made LOTS of forward progress and made sure Rick knew about it.

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon’s vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon’s dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

- pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.

- every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.

- monitoring and QA are the same thing. You’d never think so until you try doing a big SOA. But when your service says “oh yes, I’m fine”, it may well be the case that the only thing still functioning in the server is the little component that knows how to say “I’m fine, roger roger, over and out” in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it’s indistinguishable from automated QA. So they’re a continuum.

- if you have hundreds of services, and your code MUST communicate with other groups’ code via these services, then you won’t be able to find any of them without a service-discovery mechanism. And you can’t have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.

- debugging problems with someone else’s code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.

That’s just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they’re not supposed to trust external developers.

This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.

At this point they don’t even do it out of fear of being fired. I mean, they’re still afraid of that; it’s pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they’ve come to understand that it’s the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it’s the right thing because SOA-driven design enables Platforms.

That’s what Bezos was up to with his edict, of course. He didn’t (and doesn’t) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.

You wouldn’t really think that an online bookstore needs to be an extensible, programmable platform. Would you?

Well, the first big thing Bezos realized is that the infrastructure they’d built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel’ o’ other services browsable at aws.amazon.com. These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.

The other big realization he had was that he can’t always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn’t use the goddamn website. It’s not even super clear whose mom he was talking about, and doesn’t really matter, because nobody’s mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I’ve just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.

I’m not really sure how Bezos came to this realization — the insight that he can’t build one product and have it be right for everyone. But it doesn’t matter, because he gets it. There’s actually a formal name for this phenomenon. It’s called Accessibility, and it’s the most important thing in the computing world.

The. Most. Important. Thing.

If you’re sorta thinking, “huh? You mean like, blind and deaf people Accessibility?” then you’re not alone, because I’ve come to understand that there are lots and LOTS of people just like you: people for whom this idea does not have the right Accessibility, so it hasn’t been able to get through to you yet. It’s not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software — or idea-ware for that matter — fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an Accessibility failure.

Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there’s more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.

But I’ll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.

So yeah. In case you hadn’t noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I’ve worked at. But I will never get this little rant published, and you’ll never get it read, unless I start to wrap up.

That one last thing that Google doesn’t do well is Platforms. We don’t understand platforms. We don’t “get” platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.

But no. No, it’s like our tenth or eleventh priority. Or fifteenth, I don’t know. It’s pretty low. There are a few teams who treat the idea very seriously, but most teams either don’t think about it all, ever, or only a small percentage of them think about it in a very small way.

It’s a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they’re building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I’m concerned, it’s none of them. Stubby’s great, but it’s like parts when you need a car.

A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.

Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don’t get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: “So is it the Stalker API?” She got all glum and said “Yeah.” I mean, I was joking, but no… the only API call we offer is to get someone’s stream. So I guess the joke was on me.

Microsoft has known about the Dogfood rule for at least twenty years. It’s been part of their culture for a whole generation now. You don’t eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.

Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that’s not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there’s something there for everyone.

Our Google+ team took a look at the aftermarket and said: “Gosh, it looks like we need some games. Let’s go contract someone to, um, write some games for us.” Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.

You can’t do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don’t have a Steve Jobs here. I’m sorry, but we don’t.

Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn’t need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.

I apologize to those (many) of you for whom all this stuff I’m saying is incredibly obvious, because yeah. It’s incredibly frigging obvious. Except we’re not doing it. We don’t get Platforms, and we don’t get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.

So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don’t “get” much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to msdn.com, and spend some time browsing, and you’ve never seen it before, prepare to be amazed. Because it’s staggeringly huge. They have thousands, and thousands, and THOUSANDS of API calls. They have a HUGE platform. Too big in fact, because they can’t design for squat, but at least they’re doing it.

Amazon gets it. Amazon’s AWS (aws.amazon.com) is incredible. Just go look at it. Click around. It’s embarrassing. We don’t have any of that stuff.

Apple gets it, obviously. They’ve made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dogfood. And you know what? They make pretty good dogfood. Their APIs are a hell of a lot cleaner than Microsoft’s, and have been since time immemorial.

Facebook gets it. That’s what really worries me. That’s what got me off my lazy butt to write this thing. I hate blogging. I hate… plussing, or whatever it’s called when you do a massive rant in Google+ even though it’s a terrible venue for it but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it’d be pretty easy to just go. But Google is home, so I’m insisting that we have this little family intervention, uncomfortable as it might be.

After you’ve marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn’t look because I didn’t want to get too depressed), head over to developers.google.com and browse a little. Pretty big difference, eh? It’s like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.

Please don’t get me wrong here — I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They’re kicking ass as far as I’m concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.

I’m just frankly describing what developers.google.com looks like to an outsider. It looks childish. Where’s the Maps APIs in there for Christ’s sake? Some of the things in there are labs projects. And the APIs for everything I clicked were… they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it’s all snouts and horse hooves.

And also don’t get me wrong about Google+. They’re far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producters.

Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs — Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it’s hard for them to get funding for it because it’s not part of our culture. Maestro’s funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it’s a fluffy rabbit versus a T-Rex. The Docs team knows they’ll never be competitive with Office until they can match its scripting facilities, but they’re not getting any resource love. I mean, I assume they’re not, given that Apps Script only works in Spreadsheet right now, and it doesn’t even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.

Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook — that is, the stock service they offer with walls and friends and such — is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.

You know how people are always saying Google is arrogant? I’m a Googler, so I get as irritated as you do when people say that. We’re not arrogant, by and large. We’re, like, 99% Arrogance-Free. I did start this post — if you’ll reach back into distant memory — by describing Google as “doing everything right”. We do mean well, and for the most part when people say we’re arrogant it’s because we didn’t hire them, or they’re unhappy with our policies, or something along those lines. They’re inferring arrogance because it makes them feel better.

But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we’re being fools. You can attribute it to arrogance, or naivete, or whatever — it doesn’t matter in the end, because it’s foolishness. There IS no perfect product for everyone.

And so we wind up with a browser that doesn’t let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I’m actually going blind. For real. I’ve been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they’re quite brazen about it, and Fuck You if you’re blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.

It’s not just them. It’s everyone. The problem is that we’re a Product Company through and through. We built a successful product with broad appeal — our search, that is — and that wild success has biased us.

Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers… if only they could be monetized somehow… you can see how he arrived at AWS, in hindsight.

Microsoft started out as a platform, so they’ve just had lots of practice at it.

Facebook, though: they worry me. I’m no expert, but I’m pretty sure they started off as a Product and they rode that success pretty far. So I’m not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.

Maybe they just looked at us and asked: “How can we beat Google? What are they missing?”

The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don’t do internal service-oriented platforms, and we just as equally don’t do external ones. This means that the “not getting it” is endemic across the company: the PMs don’t get it, the engineers don’t get it, the product teams don’t get it, nobody gets it. Even if individuals do, even if YOU do, it doesn’t matter one bit unless we’re treating it as an all-hands-on-deck emergency. We can’t keep launching products and pretending we’ll turn them into magical beautiful extensible platforms later. We’ve tried that and it’s not working.

The Golden Rule of Platforms, “Eat Your Own Dogfood”, can be rephrased as “Start with a Platform, and Then Use it for Everything.” You can’t just bolt it on later. Certainly not easily at any rate — ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it’ll be ten times as much work as just doing it correctly up front. You can’t cheat. You can’t have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

I’m not saying it’s too late for us, but the longer we wait, the closer we get to being Too Late.

I honestly don’t know how to wrap this up. I’ve said pretty much everything I came here to say today. This post has been six years in the making. I’m sorry if I wasn’t gentle enough, or if I misrepresented some product or team or person, or if we’re actually doing LOTS of platform stuff and it just so happens that I and everyone I ever talk to has just never heard about it. I’m sorry.

But we’ve gotta start doing this right.

But for specialized system simulations the user base may be drastically smaller, as small as a solitary user. The strategies for testing are different, and often only iterative usage over time can promote confidence in the software.

A colleague out west is in the process of setting up test cases for software I wrote over the past several months. Much of the local testing I completed while developing the models and glue was based on a straight forward scenario where I could easily visualize and verify the correct system state at each processing step. To keep my development anchored I had a baseline was generated by mature external programs, and was iteratively compared to intermediate outputs.

The process of validating code is equal parts art and science. Science is leveraged to compare data products to what is known either from external sources or engineering judgement, and art is leveraged to judge what is unknown and to efficiently sample a complex nonlinear parameter space. Resilient code isn’t just written, it’s surface is smoothed by the coarse winds of regular usage.

The model code I recently delivered is going to be run through the paces and shaken down to mitigate any lingering bugs and identify any limitations which I haven’t already documented. I offered up a rough set of runs that will further validate the models as well as a brief mention of edge cases.

The edge cases I referred to weren’t what the current model was designed to handle, but the limits of what it’s capable of representing. I suggested that my colleague investigate the breaking points of the model after he’s satisfied with the baseline tests, so that analysts will have a basic understanding of the model’s capabilities. Hyper specialized systems simulations rarely allow for broad testing beyond the transient efforts that they support. Odds are that edge case testing will have to wait until after the current version is stamped and released.

The YQL or Yahoo Query Language was what I attempted to use to provide a standard feed interface. Response formats returned to the callback functions varied at the top with data.response.entries or data.response.results. Then within these json structures the entries and results had keys to describe urls. This lead to a variety of handlers and comparisons like obj.link.href, obj.link[0].href, obj.origLink, etc. The differences arise between atom, rss, version releases, and the particular provider. In theory I could write or find a JavaScript/CoffeeScript Swiss army knife of feed parsing, much like the fine Nokogiri Ruby gem I long took for granted.

It’s sad when feeds, the open “lingua franca” of the Internet, are such a mess that hitting unique APIs is both simpler and more reliable. At least with feeds the formats have a slower divergence rate compared to one shot APIs, but middleware is advised when juggling with variations.

• Google Alert feeds
• YouTube Video Search
• Stack Overflow Questions tagged CoffeeScript
• and even my CoffeeScript blog posts

I still have plenty of work to clean up the parsed feed response, as well as overhaul the video javascript embed. I was surprised there wasn’t a default javascript video search bundle (I’m using the best I could find and it’s terrible looking).

When it’s complete the code will perform a number of simultaneous up to date searches for CoffeeScript content.

The source is available at my CoffeeScriptTracker repo. Click the above image to see the Demo

For the past few days I’ve been supporting a software delivery out in Arizona¹, and to set the atmosphere I’ll emphasize that it gets warm here in the summer. The team I’ve worked with has provided fantastic support and kicked it up into high gear over the past month. They’ve gone the extra distance by extending the initial delivery date (first date was too optimistic), freed up local top notch developer and analyst time to support integration (sharp team), and minimized meetings (have I told you lately that I love you?).

While cutting corners on a rapidly approaching cut off date, I performed all manners of unsavory software hacks. Most of those were forgivable faux paus, but one in particular I feared would come back to haunt me. You can safely guess the outcome based on the post title.

Hacking the Earth

An Earth model that we developed in my shop back east is driven by a keyed input file, and is composed of an assortment of tables and polymorphic transformations. These structures support line of sight modeling differences for spherical or WGS84 Earth geometries, as well as varied atmospheric conditions.

The host service that this class lives within has a specified copy constructor and as I looked through the nested pointers in the class, I opted for file based reconstruction. And that served just fine and dandy until yesterday.

It turns out the host simulation is normally run on all manner of dynamic mounted drives and, all input files require specification in the main configuration files. Uh oh! This is done to allow relative paths to function with user set paths and run from any location. As we bumped into the issue around 6pm local time yesterday evening, I volunteered (prompted by the disapproving glares of my team) to correct my wrong doings and fix the code before we left last night.

I was desperately seeking any shortcut to avoid having to write copy constructors and clone functions for the Earth model and all its members, but didn’t see any way around it. My refactoring began with a wordpad editor and a browser download of a few svn accessible files. I was pretty foggy by this point as it’d been a long and busy day which began ages ago when I woke up 3am local time. My poor choice of editing environments lead to a quick failure, and a swap to the tech integration lead’s ² sweet Mac and Win VM setup, with Visual Studio loaded and ready.

Pro Tip: Well designed IDEs are a life saver when you’re bleary eyed and the alternative is Wordpad.

The top level of recoding revealed another layer of polymorphic composed object pointers. Like a Russian doll, each pointer referred to a contained base class which in turn required clone and copy constructors to satisfy the parent objects copy. The clock ticked by for a few hours while I hacked my way downward, adding clone methods as I went to all base and inherited classes. We finally left off last night at the bottom of the stack, with just a single class left, with each of it’s children having no further contained classes. At that point I felt guilty for holding up my colleagues Eric and Mikhail who deserved a healthy break and had long since completed their own tasks, so we called it quits for the night.

Lesson Learned

A Sucker Punch quote comes to mind:

“Don’t write a check with your mouth you can’t cash with your ass.”

Technical debt has a way of demanding compensation at inopportune moments, and while it’s not inevitable, it is mostly avoidable. In this case if more care was taken with specifying polymorphic base classes with clone and copy constructors, my refactor work would have been trivial. This morning I’ll review last night’s dicey hacks and ensure that checks for self assignment are done in the copy constructors, while validating the new code.

Notes:

1. For background see A Tale of Two Sims, and it’s sequel.
2. Eric has been doing a killer job pulling the code all together with meeting “point defense” by Greg (epic tech management). The entire team has been a pleasure to work with: Mikhail my partner in modeling, as well as Dan and Cort for c++ wizardry. We’ve had some rough spots the past couple of days, but it’s all coming together.

In the last couple of years I’ve crawled through thousands of blog posts, web docs, how to guides, tech reviews, video tutorials, along with a couple dozen technical ebooks¹. I simply can’t get enough quality documentation on languages, libraries, sdks and frameworks which kick start my enthusiasm and imagination.

I’ve developed a fine appreciation for well crafted tutorials while delving deeply into the world of modern tech docs. Master crafted guides start small and build up in simple steps, while focusing on the elegance and value of the specific tool under review. Installation and dependencies are boiled down to bare essentials to remove any potential impedance to adoption.

A moment of interruption during a first impression can turn off potential devs or trigger their attention to drift to other sources and technologies. Hackers are short on time, and quick to jump to another solution they can hastily patch together.

One recent example that highlights the advantage of minimized dependencies is my early experience with CoffeeScript. I’ve found it much easier to dive in with client side CoffeeScript examples which require only a browser and Jeremy Ashkenas’ JavaScript interpreter, than to work through server apps which depend on node, several packages, along with databases and interfaces (redis, mongo, couch, etc). I’d love to become intimately familiar with all those layers and plan to dig into cs server apps, but at the beginning they’re a distraction.

The critical resource for developers is not discovering sufficient material, but freeing up ample time to work through and understand relevant examples. Competition is fierce in the war to earn mind share from potential partners, and if you want to win it’s worth the extra effort spent honing any tech tutorials or documents. Docs are the first and most important surface with which devs contact your technology, whether it’s an open source library or an API so make them amazing and keep them up to date.

Common Traits of Polished Tutorials

I’ve danced about a few of the keys to effective documentation, and now it’s time to boil those features down into one definitive hit list:

• Start with great technology:
  Document products that don’t suck. If the library, language, or framework you’re documenting is horrible, and you absolutely must document it…




• Focus on the good parts:
  Douglas Crockford did a fantastic job of highlighting the best aspects of JavaScript, learn from his example by diving into the best aspects of the tech you’re reviewing




• Start small, build up in steps:
  Don’t begin with the most complex example you can imagine just because it shows every possible configuration of a given tech. Start small with hello world, and build up by adding one tool at a time. As a tutorial author you’re a tour guide. Bring devs along for a magic carpet ride by starting on the ground. A drive by pickup at Mach 1 makes for a messy start




• Minimize dependencies:
  The less I have to do to begin a tutorial the better, browser based walkthroughs are ideal. You don’t have to include you’re own c compiler but consider a binary instead of having devs build Erlang from scratch²




• Stylize the experience:
  Use large sparse elements, quality fonts, and generous whitespace to separate tutorial steps. Let me chew on precisely what’s going on in one example before proceeding into something more complicated. Experienced devs can always fast forward simple steps and white space to jump into the deep end

I could extend this list indefinitely with pet peeves and desired aspects of brilliant tutorials, but this is where reader feedback does a much better job. Please pitch in with documentation features you’re enamored with, or mistakes that doc authors keep repeating. I continually strive to make my “how to” posts more fun and effective, and your input helps me achieve that goal.

Notes:

1. I grab inexpensive ebooks on topics that I want more cohesive references on. My preferred format is ePub so that I own the file and can read it on my phone, tablet, laptop or desktop. But I have picked up a few texts that are Kindle only. Not all the books I purchase are for cover to cover reading, and unfortunately the references age quickly (ie I upgraded The Rails 2 Way to The Rails 3 Way because it was obsolete in under a year)




2. Cheap shot on local CouchDB installations. The database and bundled http server depend on building a local Erlang environment and boy does it take a while. The build is only necessary if you’re running something customized like GeoCouch. Nifty Mac, Linux, and Windows single server installs are available from Couchbase.

My goal was to pass a class method to setTimeout, and have it loop. Below I show each step and what was missing along with it’s compiled JavaScript.

First cut:

The class GameOfLife contains an “update” method which takes in a single argument with a default parameter. The argument is compared with an internal member variable and if equal the time out id is cleared and play is paused.

If you’re familiar with JavaScript you may notice that the update method passed to setTimeout is inaccessible. I wasn’t reviewing the compiled JavaScript at the time, so it took me a little longer to see. Eventually I came to the same conclusion.

Second take:

This version of the passed method I enclosed in an anonymous function wrapper and tried slapping the function onto the window object to make it accessible everywhere. This turned out being a bad idea because the window object doesn’t have all the other members and methods the update function relies on. But it did bring me closer to working code.

Third time’s a charm:

Finally, after several interweb searches and rereading CoffeeScript docs it struck me. This is the perfect time for the FAT arrow. This time I passed in an anonymous function with the fat arrow

=>

to bind the method always to the GameOfLife class it was called from, and removed the update function from the window object (whew). Eureka, it worked!

You can see the wiggling results at my html5 life page.

Implementing translations and adaptions of graphical JavaScript snippets has proven to be an easy transition into the basics of 2D web graphics. In fact the process has been so much fun that I defy you to devise a more entertaining method of learning CoffeeScript¹.

After mimicking a few intriguing examples, I contemplated how to combine the effects without overburdening the browser. Each small script pushes even Chromium’s JavaScript interpreter to its limits. The analogy of each canvas layer to a separate music track lead me to consider pre-rendered gifs, but that implementation would lose any interactive features. Alternatively I could write all the image updates to the same canvas element, and render it to the browser once.

I can’t imagine smoothly rendering multiple canvas elements simultaneously without a form of parallelism, lower level code, or dramatically shrinking each canvas. It turns out there are quite a few tricks I can try to speed up both individual and parallel rendering².

Life in 2D

The world of 2D pixel graphics unerringly shot my attention like an arrow to Conway’s game of life. Conway’s game set the stage for cellular automata, a broader field of self replicating patterns (see otomata for a music variant). A simple set of rules and initial state gives rise to complex self perpetuating patterns.

The universe of the Game of Life is an infinite two-dimensional orthogonal grid of square cells, each of which is in one of two possible states, live or dead. Every cell interacts with its eight neighbours, which are the cells that are horizontally, vertically, or diagonally adjacent. At each step in time, the following transitions occur:

1. Any live cell with fewer than two live neighbours dies, as if caused by under-population.
2. Any live cell with two or three live neighbours lives on to the next generation.
3. Any live cell with more than three live neighbours dies, as if by overcrowding.
4. Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

Life with CoffeeScript

My latest conversion from JavaScript is a non-canvas table based implementation of life from Conway’s Game of Life FAQ. When time allows I’ll add an interactive canvas version. Click the image below to see it:

The relevant coffeescript source:

Notes:

1. Server side CoffeeScript in Henri Bergius’ gist (bottom of post) on the Falsy Values node.js tutorials is a strong contender, but requires a couple of additional steps to dive in (installing node/npm)
2. Web Workers are one way to perform calculations leveraging multiple cpus. Each completed canvas would need to be handled by a separate thread, and the initial worker API limited their ability to modify DOM elements. The good news is the browser vendors have augmented the API to allow imagedata to be manipulated by workers, but it’ll take some tuning to leverage efficiently (limit data copying, distribute processing to workers)
   Pixel manipulation source
   
   Additional references: stack question, mandlebrot renderer and this comment

The effect runs much smoother in Chrome/ium than the versions of Firefox (5.0 beta) and Safari (latest) I have installed locally.