Structure08, my impressions so far

25 06 2008

It’s a day packed with keynotes, panels and shmoozing, with some topics overlapping with Velocity; yet at a much higher level. We’ve alternated between interesting panels (”Harnessing explosive growth”) where the key points are:

  1. a proper architecture lets you scale [much like in traditional building]
  2. build kill switches in all your features
  3. get operations and development on a symbiotic relationship [salesforce and amazon do it]

Some other panels are clearly more about pushing your product (”The race to the next database”). The topic of processing data (possibly in the cloud) is of course crucial yet very few concerns around switching costs, security and privacy are addressed. My take on this is that if you need to run analytics on your data sets and said data sets are huge, you need compute to be close (from a network distance perspective) to your data. Which means that your data must be in the cloud. While I’m reluctant to go down that route right now, Greg Papadopoulos @sun made the compelling analogy that money storage is delegated to reputable third-parties (called banks) so data are likely to follow the same treatment, i.e. the cloud is likely to become the most secure place to store data (or most resilient with an acceptable security policy). Sun’s interesting take on cloud computing is Project Caroline, where infrastructure, including network bits, is driven by code, in a way, that’s presumable a bit cleaner than EC2 (which is quite bare).

Dr. Vogel’s presentation @amzn, was inspiring despite containing basically little new information but fits well into this type of conference, which act as reinforcement devices to jumpstart a new industry.

Live coverage is at gigaom.





Velocity: John Allpaw @flickr, Capacity Planning

24 06 2008

What can cause downtime:

  1. bugs
  2. edge cases
  3. security incidents
  4. real capacity problems

Deployment and management tricks from the HPC world: ganglia, System Imager

Gather metrics of course, and build models, ideally out of live data, rather than artificial benchmarks.

fityk can be used to replace excel to do curve fitting. [My guess is that R would work great for that too]

Some flickr stats: 12,629 nagios checks, 1314 hosts, 6 data centers, 4 photo farms, 3.5-4.5 TB consumed per day.

[So flickr uses nagios + ganglia]

One key trick is to build kill switches in all the features so as to turn things off when load increases.





Velocity: Adam Bechtel @yahoo, Performance plumbing

24 06 2008

When building a global network, you start building out knobs (usually implemented as routing policies):

cost, packet loss, latency, maintenance, diversity, isolation, “special”

[Really funny analogy between anycast and toilets, caching and water supply]

After having developed routing policies, you start looking into anycast. One of the first services to be anycast is DNS.

Anycast scaling: vip, ecmp

Anycast considerations: how to monitor services? how to control users? how to handle transient network events?





Velocity: Panel, a survival guide

24 06 2008

Panelists: presented by Adam Jacob (HJK Solutions), Shayan Zadeh (Zoosk, Inc. ), Brian Moon (dealnews.com), Don MacAskill (SmugMug), John Allspaw (Flickr (Yahoo!)), Michael Halligan (BitPusher, LLC) and a gentleman (Fotolog)

Don McAskill: Rafael Nadal started to win Roland-Garros and his fanclub was there. He won the Open, which created a huge spike. Comments had to be turned off for the site to survive. The next year, he won again and stats had to be turned off. For his third victory servers did not collapse. This year he won and we did not even register.

John Allspaw: code gets pushed 20 to 30 times a day… Major events triggered traffic spikes.

Don would love to not operate a data center anymore, despite their expertise.

John: DB problems are hard [everyone in agreement, myself included]

[Discussion follows on scalablity: do not optimize for scale too early]

Don: EC2 is not worth it for servers that run around the clock, but if you’re good at shutting down instances that you don’t need.





Velocity: Sean Quilan @google, Storage at scale

24 06 2008

Strategy: buy lots of commodity hardware, because problems tend to be too big for their problem space. Hardware reliability is not that useful as well because it’s expensive.

[Showing the same pictures over and over again, someone from Google PR, please authorize the release of newer pictures]

[A GFS description follows, nothing new so far, read the papers on the topic]

[A BigTable description follows, same deal]

I wish this talk had some new information…





Velocity: Rich Wolski @ucsb, EUCALYPTUS

24 06 2008

Eucalyptus is an open-source implementation (not production-ready) of a compute cloud API-compatible with EC2. In academia sysadmin time is very expensive so the roll-out has to be really simple. Eucalyptus currently uses xen and includes a security layer that replaces Amazon’s use of the credit card authentication/authorization system.

Mention of ROCKS, a cluster deployment system.





Velocity: Brent Chapman @great circle, what can IT professionals learn from emergency services?

23 06 2008

Example: a car hits a fire hydrant. Lots of agencies involved (fire dpt, ambo, police, electrical company). How do they coördinate all that?

Incident Command System is the protocol used in pretty much all emergency situations (courses available here).

I’ll put a pointer to slides, the example used in the talk is good. The wikipedia article is supposedly good and this article from ham radio operators is a good introduction.





Velocity: some perf. tools used at Google

23 06 2008

Grinder, jMeter and some Windows tool, whose name I did not catch.





Velocity: Luiz Barroso @google, efficient energy ops

23 06 2008

Hypothetical energy cost extrapolations, 5 years from now, hardware could be only 20-50% of the total energy costs.

Efficiency defined as computing speed divided by power. Can be broken down further (computing speed / power provided to chip x power provided to chip / power provided to server x power provided to server / power provided to data center).

  • Data center efficiency, PUE around 1.83, worse if data center is underutilized
  • Server energy efficiency, 25% dissipated by power supply

From uptime institute, 10-year energy costs, $9/W for consumption, $10-22/W for data center build out.

Rough cost breakdown: 50% on hardware, 22% on energy, 28% on  data center (assumptions, dual socket x86, 4 year depreciation, 70% load at peak).

How to be more efficient:

  1. consolidate workloads
  2. measure actual power usage rather than rely on nameplates
  3. investigate oversubscription

Oversubscription potential rises as the number of machines grows so oversubscribe at the data center level. Also mix workloads and be ready to kill instances if you get close to the limit.

Source: Energy-proportional computing

Consider a data center as a device (5,000 machines), distribution with 2 peaks, one at 5% utilization, another around 30%.

Typical power efficiency of a typical server, a machine running at a load of 0.3 is at 60% power efficiency, while a fully loaded machine is at 100% power efficency, and sadly data center are very rarely at 100% as seen before.

The idea behind energy-proportional computing: a generally proportional relation between work and power. Idleness in a server is scarce. It should happen at the electronics because in software it’s much harder (think of kernel getting interrupts all the time).

If you breakdown power by component, you find out that the CPU is much-more proportional than the rest of the components so even powering down the cpu the total savings are still between 10% and 20% of power gains.

Still CPUs have 2 important power-usage features:

  1. wide dynamic power range (ram, disks and network devices remain in a much closer power range)
  2. active low-power modes, where the cpu can do things

People, which average around 120W, have a 20x dynamic power range, compared to a 2x of a PC.

In conclusion, write fast code (biggest contribution to energy efficiency), consider reduction of all energy-related costs (provisioning), and demand energy-proportionality from equipment manufacturers.

Plug: http://climatesaverscomputing.org





Velocity: John Fowler (Sun), Innovation That Drives Opportunity for the Web Infrastructure

23 06 2008

John is responsible for hardware @Sun.

Web is built on a new software stack (varnish, rails, memcache, hadoop, etc.)

Trends:

  1. 16 cores per socket for 2009, Sun, AMD and Intel on the same track. Clock rates will remain the same.
  2. Application memory capacity increasing, working to get 1TB of RAM at commodity prices
  3. ZFS and SSD, enterprise SSD, $0.08 per iops to compare to $2.43 per ios for HDD

[Sun is clearly attacking the storage market by pushing for commoditization of software, as opposed to proprietary systems such as 3PAR, EMC, etc.] Sun is building something like x4500, using an x4450 with 1 32GB ZIL SSD, 1 80GB SSD ad 5 slow SATA drives, same capex, 3 times the throughput.