Field notes: It isn’t always about virtualization, except when it is

Talking recently with some customers, we discussed the fallacy of always trying to solve a new problem with the same method that found the solution of the proceeding problem. Some might recall the adage, “If you only have a hammer, every problem looks like a nail.”

hammer_1

I am often asked to help comprehend complex performance problems our customers encounter. I don’t always have the right answer, and usually by the time the problem gets to me, a lot of good folks who are trained in solving problems have spent a lot of time trying to sort things out. I can generally be counted upon for a big-picture perspective, some non-product ideas and a few other names of folks to ask once I’ve proved to be of no help.

A recent problem appeared to have no clear solution. The problem was easy to repeat and thus demonstrable. Logs didn’t look out of the ordinary. Servers didn’t appear to be under load. Yet transactions that should be fast, say well under 10 seconds, were taking on the order of minutes. Some hands-on testing had determined that slowness decreased proportionally with the number of users attempting to do work (a single user executing a task took 30-60 seconds, two users at the same time took 1 to 90 seconds, three users took 2-3 minutes, etc.).

So I asked whether the environment used virtualized infrastructure, and if so, could we take a peek at the settings.

Yes, the environment was virtualized. No, they hadn’t looked into that yet. But yes, they would. It would take a day or too to reach the folks who could answer those questions and explain to them why we were asking.

But we never did get to ask them those questions. Their virtualization folks took a peek at the environment and discovered that the entire configuration of five servers was sharing the processing power customarily allocated to a single server. All five servers were sharing 4 GHz of processing power. They increased the resource pool to 60 GHz and the problem evaporated.

I can’t take credit for solving that one. It was simply a matter of time before someone else would have stepped back and asked the same questions. However, I did write it up for the deployment wiki. And I got to mention it here.

Be even smarter with virtualization

It took a bit of unplanned procrastination, but we finally got to the second and third parts of our in-depth investigation of virtualization as it relates to IBM Rational products.

casestudy3snip

Part two is now published here as Be smart with virtualization: Part 2. Best practices with IBM Rational Software.

Part three lives on the deployment wiki here as Troubleshooting problems in virtualized environments.

Part two presents two further case studies and a recap of the principles explored in Part 1. We took a stab at presenting the tradeoffs between different virtualization configurations. Virtualization is becoming more prevalent because it’s a powerful way to manage resources and squeeze efficiencies from hardware. Of course there are balances and Part two goes a bit deeper.

Part three moves to the deployment wiki and offers some specific situations we’ve solved in our labs and with customers. There are also screen shots of one of the main vendor’s tools which can guide you to identify your own settings.

Virtualization demystified

Read about Rational’s perspective on virtualization over at IBM developerWorks

For the IBM Innovate 2011 conference, the Rational Performance Engineering team presented some of its research on virtualization. We had an accompanying slide deck too, and called it the Rational Virtualization Handbook.

It’s taken a bit of time, but we have finally fleshed out the slides and written a proper article.

Actually, the article has stretched into two parts, the first of which lives at Be smart with virtualization. Part 1. Best practices with IBM Rational software. Part 2 is in progress and will contain further examples and some troubleshooting suggestions. I can’t say for sure, but we have a topic lined up which would make a third part, but there’s a lot of work ahead.

I’m tempted to repost large excerpts because I’m proud of the work the team did. And it took a bit longer than expected to convert slideware into a real article, and so the article took a lot of work. I won’t give away the secrets here…. You’ll have to check out IBM developerWorks yourself. However, let me kickstart things with a history sidebar:

A brief history of virtualization

Despite its emergence as a compelling, necessary technology in the past few years, server virtualization has actually been around for quite some time. In the 1970s, IBM introduced hypervisor technology in the System z and System i® product lines. Logical partitions (LPARs) became possible on System p® in 2000. The advent of virtual machines on System x and Intel-based x86 hardware was possible as early as 1999. In just the last few years, virtualization has become essential and inevitable in Microsoft Windows and Linux environments.

What products are supported in virtualized environments?

Very often we explain that asking whether a particular Rational product is supported with virtualization isn’t actually the right question. Yes, we’ve run on Power hardware and lpars for several years now. Admittedly KVM and VMware are newer to the scene. Some may recall how clock drift could really mess things up, but those problems seem to be behind us.

The question isn’t whether Rational products are supported on a particular flavor of virtualization: If we support a particular Windows OS or Linux OS, then we support that OS whether it’s physical or virtualized.

Virtualization is everywhere

Starting in 2010 at Innovate and other events, we routinely asked folks in the audience whether they were aware their organizations were using virtualization (the platform didn’t matter). In 2010 and 2011 we got a few hands, maybe two or three in a room of 20. Folks were asking us if virtualization was supported. Was it safe? Could they use it? What were our suggestions?

Two years later, in 2012, that ratio was reversed: Nearly every hand in the audience shot up. We got knowing looks from folks who had disaster stories of badly managed VMs. There were quite a few people who had figured out how to manage virtualization successfully. There were questions from folks looking for evidence and our suggestions to take back to their IT folks.

Well, finally, we have something a bit more detailed in print.