Simon’s Blog

Sun Microsystems Inc ( SUNW ) is shortly to make its Grid Compute Utility offering generally available, initially to customers based in the US. In his blog post, entitled The Network Is The Computer, Sun’s President and COO, Jonathan Schwartz, gives some of the details. The question, then, is - does Sun’s offering looking like it has the right ingredients for success? My view is that they’re not quite there yet - read on to see why.

Anyone who has had a need for some kind of supercomputing capability in the last ten years, will have at least explored compute farms as an option. Around eight years ago, which is when we set up our first compute farm, this was starting to become the approach of choice. This is because it was exceptionally cost-effective - you could build a super-computer simply by linking together commodity x86 boxes running Linux. However, while it was at least an order of magnitude cheaper than the super-computers that came before, it was a pretty painful experience because both the OS platforms and grid software were pretty immature (read, riddled with bugs). Today, things are much better. However, it’s still a pain to set up and run compute farms. Far better to outsource it. And that’s what people do. I was told yeterday, for example, that Google doesn’t run its own data centers - it outsources that part of the business.

However, what Sun has announced is taking things another step further along from outsourcing. That is, they’re offering a compute utility, which will be shared with many other users i.e. it isn’t customised to meet your needs. It’s a fantastic idea, because by using software technology to manage the sharing of resources efficiently, potentially huge economies of scale can be achieved. There are, though, a number of issues that people wanting wanting to use computing like a utility (similar to the way people use electricity to power their homes and businesses) must consider.

The key consideration is that in order for a utility computing service to be useful, it must meet the needs of your application. Now, before the utility computing enthusiasts jump up and shout, “You’re missing the point - you need to tailor your needs to what the utility can provide”, let me clarify my meaning. It’s true that when you want to use a utility in general, you must tailor your application to fit in with what the utility can provide. For example, if you want to use electricty to power your device, then you better make sure you can plug it into the electricity supply provided by the power company. But that’s not what I’m talking about here.

Let’s take a couple of concrete examples, which will explain my point. Let’s say you’re making a computer animated movie. You can load your software onto a utility computing service easily. You can probably even upload your input data onto the system. However, the output from your software is high-quality, uncompressed video data. And a characteristic of high-quality uncompressed video data is - it’s BIG (possibly be pushing into the petabyte range). Transmitting this volume data across the Internet may be pretty challenging and expensive (it’s actually a bit challenging [and expensive] to move these volumes of data rapidly even around a LAN). In fact, it’s very possibly cost-prohibitive to to that.

The reverse is also the case: that is, if you need to analyse large volumes of data, you then have the challenge of uploading the data to the grid. This is precisely the problem with a project we’re currently working on - developing sophisticated computer vision algorithms for analysis of large (petabyte) volumes of video data. We’re exploring utility computing for this application, but a straight CPU utility simply doesn’t cut it. Why? Because as well as CPU grids (such as the one Sun is offering), you need to address the problem of data. And for that you need data grids. What we’re doing is exploring ways to tap into high-bandwith international data grids, that are integrated into super-computing CPU grids. At the entry point to the data grid, we’ll almost certainly be using a semi-automated process e.g. uploading from tape.

The bottom line is: unless your super-computing problem is one where you don’t need to input or output large volumes of data (which is a subset of super-computing problems), a straight CPU grid utility is unlikely to meet your needs. As Jonathan Schwartz says, Sun’s strapline of The Network Is The Computer is becoming only increasingly true with the passage of time. The problem is that today, the networks most people have access to don’t meet the general needs of super computing.

I really want to see Sun succeed with their grid offering. Personally, I’d like to see them evolve this away from straight super-computing applications (such as movie production and biotech super computing), into an offering that could serve the needs of Internet start-ups; thereby lowering the barrier to entry for entrepreneurs to create the next Google or Ebay. Having a low-cost, scalable, shared compute utlity, offered as a total solution, that can host and run software services such as those needed to run Google, Ebay, Flickr or Riya would, in my opinion, be a simply massive success.