Aehso’s Output

ESBs and REST.

Wonderful post (and followup) by Steve Vinoski, spawning lots of other discussions.  I should be surprised by some of the comments but… I’m not - I suspect agendas, pride and reputations may be at play.  Imho, and I stress humble opinion, his views are completely valid. Can enterprise architects disregard the architecture, constraints and protocols that helped the Web scale and adapt to be the global platform it is today?  If they want to build something that will adapt and scale then I also think the answer has to be No.

Platforms like ESBs were built to solve the problem of interconnecting and integrating enterprise systems. The disconnect between how an ESB-based or RESTful solution approach the same problem can mostly be explained by the fact that they both evolved from very different starting points, and therefore have very different principals at their core. 

ESBs, having evolved from DCE-RPC, CORBA based RPC architectures, have always required static definition of non-uniform remote interfaces.  The modern WS-Deathstar-type ESBs are still based on this concept, with WSDL interface typically backed by statically compiled implementations (or if you are exotic XML based mediation languages).  The remote interface semantics can be augmented by a variety (cluster f&@k?) of ‘enterprisey’ features like transactions, routing and reliability defined by a collection of related specifications but it turns out that implementations based on some of those specifications do not interoperate or scale.  This is primarily a function of questionable specification development processes and over complexity.

In the end the non-uniform interfaces at the heart of these solutions tightly couple the clients and servers and this coupling is the underlying limiting factor on the adaptability and extensibility of these systems.  I still really struggle to find a successful real world internet-available IDL or WSDL based web service that has been used in a series of completely unexpected contexts (i.e. mashups) or that has evolved beyond more than one or two revisions without completely breaking backward compatibility for old clients.

In parallel the RESTful HTTP standard emerged and provided the platform for the Web that has scaled and adapted to truly global proportions.  This wasn’t an accident - the Web succeeded because of the RESTful principals upon it was built - stateless, client server, layered systems that support caching and of course the true use of Hypertext As The Engine Of Application State (thankfully now referable to as ‘the hypertext constraint‘).  To me HATEOAS is the most critical feature of RESTful solutions. It is the one feature that negates the need for non uniform interfaces and it is here where the two approaches diverge dramatically.  It also isn’t an easy concept to grasp for folks who are used to traditional RPC oriented systems. 

Regardless, the recent emergence of dynamic languages like Python and Ruby, and their respective web application frameworks, Django and Rails, is now making it economically efficient for anyone to produce RESTful web services that can scale and that are reusable.  Steve is just expressing his opinion that this approach works better.

Don’t get me wrong - the ESB approach has been proven to succeed and
scale but primarily in limited deployment environments and usually only when considerable resources are thrown at the solution.  To put it another way, the inherent limits in non-uniform interface based systems can be pushed through use of sophisticated tools and language bindings but this only be achievable when considerable resources are thrown at the solution.  I think this is what Steve is getting at - I don’t think Steve is saying ESBs are “bad”, just that they are not the best platform for building internet scale services/resources

(Hey, check it out, I didn’t mention SOA once!)

BTW, I’m reading The Future Of Ideas at the moment, hope to write more about that soon, but it does relate to the notion of control mentioned in Steve’s latest followup post.

Seattle Conference on Scalability : YouTube’s Growing Pains.

Cuong Do, one of the original YouTube engineers, gave a great talk (50+ minutes but well worth it) at Google’s recent Seattle Conference on Scalability.  Some takeway thoughts:

• Pre Google, YouTube had 2 sysadmins, 2 scalability software engineers, 2 feature developers, 2 network enginers and 1 DBA.  Yowza! talk about a productive team!
• Python is fast enough.  In their small team, development speed was as important as execution speed.  I’m confident the same applies for Ruby on Rails too (this was a primary consideration for our choosing RoR instead of Java for our new nooked platform).
• MySQL replication works, to the point when replication of writes starts to significantly impact reads from the replicas.  Once you hit that point start working at night to develop an alternative middle tier that relies on partitioning the data into multiple shards.  You work on the alternative middle tier at night because you’ll spend all day trying to keep the replicas up to speed with the master. Map incoming requests to the correct shard by doing a user id lookup on a dedicated table (in a heavily cached local database) to redirect requests to the correct shard.

Of course they now get to play with all of Google’s in-house toys (MapReduce, GFS etc) so doubt if he gives another presentation next year it’ll probably be a very different one.

Speaking of MapReduce, I see that Ruby has an implementation - Starfish.  Cool.