Scale or die » Uncategorized

At CMG’09 today

alq — Wed, 09 Dec 2009 17:35:26 +0000

On paper it looked like a scientific approach to performance management, born in the mainframe days when computers were expensive. Now it’s cloud-scale that matters (and an ailing world economy if you’re not a bank) so managing capacity rigorously (and in an automated fashion) makes sense.

So far no breakthrough though, it’s a bit too applied to my taste. Let’s see what the next sessions hold in store.

Netflix describes its culture

alq — Tue, 04 Aug 2009 13:33:21 +0000

I love Amazon Web Services open pricing

alq — Wed, 17 Jun 2009 03:34:05 +0000

I’ve just spent 2 hours crafting a spreadsheet to compare how much it would cost to set up a decent platform to deliver the kind of data services I manage, vs. the same on EC2. Easy access to pricing is a key variable that’s often hard to get from vendors without being subjected to the “custom solution” time-waste. Technology vendors, your customers, more often than not, know what they want. When I ask for a price list, don’t try to second-guess whether I’ve done my homework, just give me the price list. If I have questions regarding the “solution” I’ll be more than happy to ask.

Tokyo train map meets Internet powerhouses

alq — Thu, 16 Apr 2009 21:59:31 +0000

Web Trend Map 4 Final Beta

Originally uploaded by formforce

Cute.

Robots

alq — Sun, 05 Apr 2009 15:00:02 +0000

Somewhat unrelated to the topic of data crunching and computing I wanted to mention an eye-opening book about robots: Wired for War by P.W. Singer.

Posting from emacs

alq — Mon, 30 Mar 2009 20:59:49 +0000

Tiny post about using weblogger.el, I already feel so l33t.

Interesting data growth factoid

alq — Fri, 20 Mar 2009 01:30:48 +0000

From http://aws.amazon.com/publicdatasets/

“United States demographic data from the 1980 (approximately 2 GB), 1990 (approximately 50 GB), and 2000 US Censuses (approximately 200GB)”

Should we expect a 400GB volume for the 2010 census, or 2TB in 2020 and 200TB in 2040? Probably the latter once you start adding biodata.

Thinking about IT Operations and Kanban

alq — Wed, 18 Mar 2009 15:48:43 +0000

As our developers are transitioning to an agile methodology, we have been figuring out how to adapt our operational processes to a more regular schedule that fixed-length, 1-month-long sprints are going to entail. So far we have worked in a more waterfall approach with a high-level of interrupts, taking on projects, doing an upfront analysis to break work into small chunks and piping such chunks through FogBugz to track progress. Recently the team has been reading about kanban as a way to formalize flow and make under-capacity visible. While I believe we have adopted an informal pull-driven process, now is the time to formalize all this so as to properly communicate whether and when infrastructure projects can be delivered.

The first round of experiments is taking shape, more to follow shortly:

Velocity: Panel, a survival guide

alq — Tue, 24 Jun 2008 20:48:15 +0000

Panelists: presented by Adam Jacob (HJK Solutions), Shayan Zadeh (Zoosk, Inc. ), Brian Moon (dealnews.com), Don MacAskill (SmugMug), John Allspaw (Flickr (Yahoo!)), Michael Halligan (BitPusher, LLC) and a gentleman (Fotolog)

Don McAskill: Rafael Nadal started to win Roland-Garros and his fanclub was there. He won the Open, which created a huge spike. Comments had to be turned off for the site to survive. The next year, he won again and stats had to be turned off. For his third victory servers did not collapse. This year he won and we did not even register.

John Allspaw: code gets pushed 20 to 30 times a day… Major events triggered traffic spikes.

Don would love to not operate a data center anymore, despite their expertise.

John: DB problems are hard [everyone in agreement, myself included]

[Discussion follows on scalablity: do not optimize for scale too early]

Don: EC2 is not worth it for servers that run around the clock, but if you’re good at shutting down instances that you don’t need.

Velocity: Sean Quilan @google, Storage at scale

alq — Tue, 24 Jun 2008 19:09:54 +0000

Strategy: buy lots of commodity hardware, because problems tend to be too big for their problem space. Hardware reliability is not that useful as well because it’s expensive.

[Showing the same pictures over and over again, someone from Google PR, please authorize the release of newer pictures]

[A GFS description follows, nothing new so far, read the papers on the topic]

[A BigTable description follows, same deal]

I wish this talk had some new information…