Tag: Amazon

Does Metcalfe’s Law apply to Cloud Platforms and Big Data?

2012-June-11

Cloud Platforms and Big Data – Have we reached a tipping point?  To think about this, I want to take a look back in history.

Metcalfe’s Law was named for Robert Metcalfe, one of the true internet pioneers, by George Gilder,  in an article that appeared in a 1993 issue of Forbes Magazine, it states that the value of a network increases with the square of the number of nodes.  It was named in the spirit of “Moore’s Law” – the popular aphorism attributed to Gordon Moore that stated that the density of transistors on a chip roughly doubles every 18 months. Moore’s Law succinctly captured why computers grew more powerful by the day.

With the success of “Moore’s Law”, people looked for other “Laws” to guide their thinking about a technology industry that seemed to grow exponentially and evolve chaotically, and “Metcalfe’s Law” was one of them.  That these “laws” were not really laws at all, but really just arguments, predictions, and opinions, was easily forgotten. People grabbed hold of them.

Generalizing a Specific Argument

Gilder’s full name for the “law” was “Metcalfe’s Law of the Telecosm”, and in naming it, he was thinking specifically of the competition between telecommunications network standards, ATM (Asynchronous Transfer Mode) and Ethernet.  Many people were convinced that ATM would eventually “win”, because of its superior switching performance, for applications like voice, video, and data.  Gilder did not agree. He thought ethernet would win, because of the massive momentum behind it.

Gilder was right about that, and for the right reasons. And so Metcalfe’s Law was right!  Since then though, people have argued that Metcalfe’s Law applies equally well to any network.  For example, a network of business partners, a network of retail stores, a network of television broadcast affiliates, a “network” of tools and partners surrounding a platform.  But generalizing Gilder’s specific argument this way is sloppy.

A 2006 article in IEEE Spectrum on Metcalfe’s Law says flatly that  the law is “Wrong”, and explains why:  not all “connections” in a network contribute equally to the value of the network.  Think of Twitter – most subscribers publish very little information, and to very limited circles of friends and family.  Twitter is valuable, and it grows in value as more people sign up, but Metcalfe’s Law does not provide the metaphor for valuing it. Or think of a telephone network: most people spend most of their time on the phone with around 10 people. Adding more people to that network does not increase the value of the network, for those people. Adding more people does not cause revenue to rise according to the  O(n2) metric implicit in Metcalfe’s Law.

Clearly the direction of the “law” is correct – as a network grows, its value grows faster.  We all feel that to be implicitly true, and so we latch on to Gilder’s aphorism as a quick way to describe it. But clearly also, the law is wrong generally.

Alternative “Laws” also Fail

The IEEE article tries to offer other valuation formulae, suggesting that the true value is not O(n2), but instead O(n*log(n)), and specifically suggests this as a basis for valuation of markets, companies, and startups.

That suggestion is arbitrary.  I find the mathematical argument presented in the IEEE article to be hand-wavey and unpersuasive. The bottom line is that networks are different, and there is not one law – not Metcalfe’s, nor Reed’s nor Zipf’s as suggested by the authors of that IEEE article – that applies generally to all of them. Metcalfe’s Law applied specifically but loosely to the economics of ethernet, just as Moore’s Law applied specifically to transistor density. Moore’s Law was not general to any manufacturing process, nor is Metcalfe’s Law general to any network.

Sorry, there is no “law”; One needs to understand the economic costs and potential benefits of a network, and the actual conditions in the market, in order to apply a value to that network.

Prescription: Practical Analysis

Ethernet enjoyed economic advantages in terms of cost of production and generalization of R&D development. Ethernet reached an economic tipping point, and beyond that other factors like improved switching performance of alternatives, were simply not enough to overcome the existing investment in tools, technology, and understanding of ethernet.

We all need to apply that sort of practical thinking to computing platform technology. For some time now, people have been saying that Cloud Platform technology is the Next Big Thing. There have been some skeptics, notably Larry Ellison, but even he has come around and is investing.

Cloud Platforms will “win” over existing on-premises platform options, when it makes economic sense to do so. In practice, this means, when tools for building, deploying, and managing systems in the cloud become widely available, and just as good as those that exist and are in wide use for on-premises platforms.

Likewise “Big Data” will win when it is simply better than using traditional data analysis, for mainstream data analysis workloads. Sure, Facebook and Yahoo use MapReduce for analysis, but, news flash: Unless you have 100m users, your company is not like Facebook or Yahoo. You do not have the same analysis needs. You might want to analyze lots of data, even terabytes of it. But the big boys are doing petabytes. Chances are, you’re not like them.

This is why Microsoft’s Azure is so critical to the evolution of Cloud offerings  Microsoft brought computing to the masses, and the company understands the network effects of partners, tools providers, and developers. It’s true that Amazon has a lead in cloud-hosted platforms, and it’s true that even today, startups prefer cloud to on-premises. But EC2 and S3 are still not commonly considered as general options by conservative businesses. Most banks, when revising their loan processing systems, are not putting EC2 on their short list. Microsoft’s work in bringing cloud platforms to the masses will make a huge difference in the marketplace.

I don’t mean to predict that Microsoft will “win” over Amazon in cloud platforms; I mean only to say that Microsoft’s expanded presence in the space will legitimize Cloud and make it much more accessible. Mainstream. It remains to be seen how soon or how strongly Microsoft will push on Big Data, and whether we should expect to see the same effect there.

The Bottom Line

Robert Metcalfe, the internet pioneer, himself apparently went so far as to predict that by 2013, ATM would “prevail” in the battle of the network standards. Gilder did not subscribe to such views. He felt that Ethernet would win, and Metcalfe’s Law was why. He was right.

But applying Gilder’s reasoning blindly makes no sense. Cloud and Big Data will ultimately “win” when they mature as platforms, and deliver better economic value over the existing alternatives.

 

Is Amazon DynamoDB the look of the future?

2012-May-21

Amazon is now offering a key/value data store that relies on Solid state disc for storage. DynamoDB is the name, and it is intended to complement S3 as a lower-latency store. It’s higher cost, but offers better performance for those customers that need it.

Two things on this.

  1. The HighScalability blog calls Amazon’s use of SSD as a “radical step.”  That view may become antiquated rapidly.The one outlier in the datacenter of today is the use of spinning mechanical platters as a basis to store data. Think about that. There’s one kind of moving part in a datacenter – the disk.  It consumes much of the power and causes much of the heat. We will see SSD replace mag disk as a mainstream storage technology, sooner than most of us think.  Companies like Pure Storage will lead the way but EMC and the other big guys are not waiting to get beaten.Depending on manufacturing ramp-up, this could happen in 3-5 years. It won’t be radical. The presence of spinning platters in a datacenter will be quaint in 8 years.
  2. The exposure of the underlying storage mechanism to the developer is a distraction.  I don’t want to have to choose my data programming model based on my latency requirements.  I don’t want to know, or care, that the storage relies on an SSD. That Amazon exposes it today is a temporary condition, I think. The use of the lower-latency store ought to be dynamically determined by the cloud platform itself, based on provisioning configuration provided by the application owner.  Amazon will just fold this into its generic store. Other cloud platform providers will follow.The flexibility and intelligence allowed in provisioning the store is the kind of thing that will provide the key differentiation among cloud platforms.

What does it mean? Microsoft upping efforts on Infrastructure-as-a-service

2012-May-21

Wired is reporting a rumor that Microsoft will soon launch a new Infrastructure-as-a-service offering to compete with Amazon EC2, in June.

What Does it Mean?

I have no idea whether the “rumor” is true, or even what it really means. I speculate that the bottom line is that we’ll be able to upload arbitrary VHDs to Azure. Right now Microsoft allows people to upload VHDs that run Windows Server 2008.  With this change they may support “anything”.  Because it’s a virtual hard drive, and the creator of that hard drive has full control over what goes into it, that means an Azure customer will be able to provision VMs in the Microsoft cloud that run any OS, including Linux. This would also represent a departure from the stateless model that Windows Azure currently supports for the VM role. It means that VHDs running in the Windows Azure cloud will be able to save local state across stop/restart.

Should we be Surprised?

Is this revolutionary?  Windows Azure already offers compute nodes; it’s beta today but it’s there, and billable.  So there is some degree of Infrastructure-as-a-service capability today.

For my purposes “infrastructure as a service”  implies raw compute and storage, which is something like Amazon’s EC2 and S3. A “platform as a service” walks up the stack a little, and offers some additional facilities for use in applications. This might include application management and monitoring, enhancements to the storage model or service, messaging, access control, and so on. All of those are general-purpose things, usable in a large variety of applications, and we’d say they are “higher level” than storage and compute. In fact those services are built upon the compute+storage baseline.

For generations in the software business, Microsoft has been a major provider of platforms. With its launch in 1990, Windows itself was arguably the first broadly adopted “application platform”.  Since the early 90’s, specialization and evolution have resulted in an proliferation of platforms in the industry – we have client platforms, server platforms (expanding to include the Hypervisor), web platforms (IIS+ASP.NET, Apache+PHP), data platforms, mobile platforms and so on. And beyond app platforms, since Dynamics Microsoft has also beein in the business of offering applications as well, and it’s here we see the fractal nature of the space.  The applications can act as platforms for a particular set of extensions.  In any case, it’s clear that Microsoft has offerings in all those spaces, and more.

Beneath the applications like Dynamics, and beneath the traditional application platforms like Windows + SQL Server + IIS + .NET, Microsoft has continued to deliver the foundational infrastructure, specifically to enable other alternative platforms. Oracle RDBMS and Tomcat running on Windows is a great example of what I mean here. Sure, Microsoft would like to entice customers to adopt the entirety of their higher-level platforms, but the company is willing to make money by supplying lower-level infrastructure for alternative platforms.

Considering that history, the rumor that Microsoft is “upping efforts on infrastructure as a service” should not be surprising.  Microsoft has long provided offerings at many levels of “the stack”.  I expect that customers have clearly told Microsoft they want to run VHDs, just like on EC2, and Microsoft is responding to that.  Not everyone will want this; most people who want this will also want higher-level services.  I still believe strongly in the value of higher-level cloud-based platforms.

Platform differentiation in the Age of Clouds

It used to be that differentiation in server platforms was dominated by the hardware. There were real, though fluctuating and short-lived, performance differences between Sun’s Sparc, HP’s PA-RISC, IBM’s RIOS and Intel’s x86. But for the moment, the industry has found an answer to the hardware question; servers use x64.

With standard high volume servers, the next dominant factor for differentiation was on the application programming model.  We had a parade of players like CORBA, COM, Java, EJB, J2EE, .NET. More recently we have PHP, node.js, Ruby, and Python. The competition in that space has not settled on a single, decisive winner, and in my judgment, that is not likely to happen. Multiple viable options will remain, and the options that enjoy relatively more success do share some common attributes: ease of programming (eg, building an ASPNET service or a PHP page) is favored over raw performance (building an ISAPI or an Apache module in C/C++).  Also, flexibility of the model (JSP/Tomcat/RESTlets) is favored over more heavily prescriptive metaphors (J2EE). I expect the many options in server platform space to continue; the low-cost to develop and extend these platform alternatives means there is no natural economic value of convergence, as there was in server hardware where the R&D costs are relatively high.

Every option in the space will offer its own unique combination of strengths, and enterprises will choose among them. One company might prefer strong support for running REST services, while another might prefer the application metaphor  of Ruby on Rails.  Competition will continue.

But programmer-oriented features will not be the key differentiator in the world of cloud-hosted platforms. Instead, I expect to see operational and deployment issues to dominate.

  • How Reliable is the service?
  • How difficult is it to provision a new batch of servers?
  • How flexible is the hosting model? Sometimes I want raw VMs, sometimes I want higher-level abstractions. I might want to manage a “farm” of servers at a time, or even better, I might want to manage my application without regard for how many VMs back it.
  • How extensive are the complementary services, like access control, messaging, data, analysis, and so on.
  • What kind of operational data do I get out of that farm of servers? I want to see usage statistics and patterns of user activity.

It won’t be ease of development that wins the day.

Amazon has been very disruptive with its AWS, and Microsoft is warming to the competition. This is all good news for the industry. It means more choices, better options, and lower costs, all of which promotes innovation.