Aehso’s Output

Seattle Conference on Scalability : YouTube’s Growing Pains.

Cuong Do, one of the original YouTube engineers, gave a great talk (50+ minutes but well worth it) at Google’s recent Seattle Conference on Scalability.  Some takeway thoughts:

• Pre Google, YouTube had 2 sysadmins, 2 scalability software engineers, 2 feature developers, 2 network enginers and 1 DBA.  Yowza! talk about a productive team!
• Python is fast enough.  In their small team, development speed was as important as execution speed.  I’m confident the same applies for Ruby on Rails too (this was a primary consideration for our choosing RoR instead of Java for our new nooked platform).
• MySQL replication works, to the point when replication of writes starts to significantly impact reads from the replicas.  Once you hit that point start working at night to develop an alternative middle tier that relies on partitioning the data into multiple shards.  You work on the alternative middle tier at night because you’ll spend all day trying to keep the replicas up to speed with the master. Map incoming requests to the correct shard by doing a user id lookup on a dedicated table (in a heavily cached local database) to redirect requests to the correct shard.

Of course they now get to play with all of Google’s in-house toys (MapReduce, GFS etc) so doubt if he gives another presentation next year it’ll probably be a very different one.

Speaking of MapReduce, I see that Ruby has an implementation - Starfish.  Cool.