CLM 5.0.1 datasheets

Collaborative Lifecycle Management (CLM) 5.0.1 was announced and released this week. For Rational Team Concert (RTC) 5.0.1 there were substantial improvements made to the Plan Loading capability which are outlined in Plan Loading Performance Improvements in RTC 5.0.1.

Other 5.0.1 datasheets include:

 

For the CLM 5.0.1 release, performance testing validated that there were no regressions for RTC and RQM between the 5.0 and 5.0.1 releases, therefore there has not been a need to create 5.0.1 datasheets for those products. For 5.0.1 performance information for those products please consult the 5.0 datasheets.

Two new CLM 5.0 datasheets on the jazz.net Deployment wiki

A quick note about two recent performance datasheets and sizing guidelines published on the Jazz.net Deployment wiki:

Just posted is the Rational Engineering Lifecycle Manager (RELM) performance report 5.0 by Aanjan Hari. This article presents the results of the team’s performance testing for the RELM 5.0 release and deployment suggestions derived from the tests.

snap1099

Also recently posted: Sizing and tuning guide for Rational DOORS Next Generation 5.0 by Balakrishna Kolla. This article offers guidance based upon the results of performance tests that the team ran on several hardware and software configurations.

 

CLM 5.0 performance documents and datasheets

CLM 5.0 was announced June 2, 2015, at the IBM Innovate 2014 conference in Orlando, FL. Lots of good things made it into the release. You can get all the details here.

Over on the deployment wiki we have published 11 — yes eleven! — datasheets detailing performance aspects of the CLM 5.0 release. Find them all here.

First the product-specific reports: Collaborative Lifecycle Management performance report: RTC 5.0 release compares RTC 5.0 against the prior release 4.0.6 to verify that there are no performance regressions in 5.0.

50webimprovwide

RTC 5.0 provides new Web caching technology that can improve application performance and scalability. The new technology stores cachable information locally on the client. Web caching performance improvements in Rational Team Concert 5.0 details the changes, and demonstrates the response time improvements from 8% to 2x improvement.

The Collaborative Lifecycle Management performance report: RDNG 5.0 compares RDNG 5.0 against the prior release 4.0.6 to verify that there are no performance regressions. Additionally, several user actions, such as opening a project, have become faster. Note that the RDNG 5.0 architecture is different from prior releases in that the RDNG repository is no separate from the JTS server.

Similarly, Collaborative Lifecycle Management performance report: Rational Quality Manager 5.0 release compares RQM 5.0 to the prior release 4.0.6 to verify that there are no performance regressions in 5.0. The results show that in some cases, page response times are slower.

The CLM reliability report: CLM 5.0 release puts the CLM applications together and runs them under load for seven days to evaluate their performance.

50zcollectingbuildablefiles

Rational Team Concert for z/OS Performance Improvement of Source Code Data Collection and Query in 5.0 shows improvements in source code data collection service and source code data query.

Since release 4.0.1 there have been gradual improvements in releases of RTC for z/OS. Rational Team Concert For z/OS Performance Comparison Between Releases details the improvements which include enterprise build time improving 45% from the 4.0.1 to 5.0 release.

Rational Team Concert Enterprise Extension zIIP Offload Performance in 5.0 documents how zIIP can offload application workload saving time and expense.

Enterprise Extensions promotion improvements in Rational Team Concert version 5.0 on z/OS compares ‘finalize build maps’ activity with the ‘publish build map links’ option selected between releases 4.0.6 and 5.0 where an improvement of 70% was observed.

dcc5

In the reporting space, Collaborative Lifecycle Management performance report: Export Transform Load (ETL) 5.0 release shows that DM and Java ETL for 5.0 have similar throughput as 4.0.6. The RM part of DM and Java ETL have an approximate 8% improvement due to the performance optimization of the RM publish service.

CLM 5.0 introduces a new reporting technology and Collaborative Lifecycle Management performance report: Data Collection Component Performance (DCC) 5.0 release compares the old technology with the new.

For the particular topology and dataset tested:

  1. DCC performance has a significant improvement per the comparison of JAVA and DCC ETL based on performance test data. The duration reduced from 48 hours to 16 hours. For the duration of specific applications the DCC also has a significant improvement. It improved about 30% on QM loading, 20% on RM loading, 60% on CCM loading and 90% on Star job.
  2. DCC duration also has a significant improvement per the comarison of DM and DCC ETL based on performance test data. The duration reduced from 139 hours to 16 hours. The major improvements are the RRC ETL and RQM ETL. RQM loading improved about 60% and RRC loading improved about 85%.

I’m sure you’ll agree with me that a lot of good work went into CLM 5.0. If you have comments or questions on the reports, please use the comments box on the actual reports.

 

Software sizing isn’t easy

I’m going to quote pretty much the entirety of an introduction I wrote to an article just posted at the jazz.net Deployment wiki on CLM Sizing (https://jazz.net/wiki/bin/view/Deployment/CLMSizingStrategy):

Whether new users or seasoned experts, customers using IBM Jazz products all want the same thing: They want to use the Jazz products without worrying that their deployment implementation will slow them down, that it will keep up with them as they add users and grow. A frequent question we hear, whether it’s from a new administrator setting up Collaborative Lifecycle Management (CLM) for the first time, or an experienced administrator tuning their Systems and Software Engineering (SSE) toolset, is “How many users will my environment support?”

Back when Rational Team Concert (RTC) was in its infancy we built a comprehensive performance test environment based on what we thought was a representative workload. It was in fact based upon the workload the RTC and Jazz teams itself used to develop the product. We published what we learned in our first Sizing Guide. Later sizing guides include: Collaborative Lifecycle Management 2011 Sizing Guide and Collaborative Lifecycle Management 2012 Sizing Report (Standard Topology E1). As features were added and the release grew, we started to hear about what folks were doing in the field. The Jazz products, RTC especially, are so flexible that customers were using them with wonderfully different workloads than we had anticipated.

Consequently, we stepped back from proclaiming a one-size fits all approach, and moved to presenting case studies and specific test reports about the user workload simulations and the loads we tested. We have published these reports on the jazz.net Deployment wiki at Performance datasheets. We have tried to make a distinction between performance reports and sizing guides. Performance reports document a specific test with defined hardware, datashape and workload, whereas sizing guides suggest patterns or categories of hardware, datashape and workload. Sizing reports are not specific and general descriptions of topologies and estimations of workloads they may support.

Throughout the many 4.0.x release cycles, we were still asked “How many users will my environment support?” Our reluctance to answer this apparently straightforward question frustrated customers new and old. Everyone thinks that as the Jazz experts we should know how to size our products. Finally, after some analysis and messing up countless whiteboards, we would like to present some sizing strategies and advice for the front-end applications in the Jazz platform: Rational Team Concert (RTC), Rational Requirements Composer (RRC)/Rational DOORS Next Generation (DNG) and Rational Quality Manager (RQM). These recommendations are based upon our product testing and analysis of customer deployments.

sizingestact_wide

The article talks about how complex estimating a software sizing can be. Besides the obligatory disclaimer, there’s a pointer to the CLM and SEE recommended topologies and a discussion of basic definitions. There’s also a table listing many of the non-product (or non-functional) factors which can wreck havoc with the ideal performance of a software deployment.

Most importantly, the article provides some user sizing basics for Rational Team Concert (RTC), Rational Requirements Composer (RRC)/Rational DOORS Next Generation (DNG) and Rational Quality Manager (RQM). Eventually we’ll talk a bit more about the strategies / concepts needed to determine whether you may need two CCMs or multiple application servers in your environment.

For now, I hope we’re taking a good step towards answering the perennial question: “How many users will my environment support,” and explaining why it’s so hard to answer that question accurately.

As always, comments and questions are appreciated.

CLM 4.0.6 performance reports

A fresh batch of 4.0.6 datasheets was posted to the deployment wiki coincident with the 4.0.6 release. 4.0.6 was released February 28, 2014. Yes, that was a month or so ago, and so I’m late with mentioning the timely performance reports, our largest batch yet. The team worked hard to get the data and compile the reports.

For those keen on migrating data from ClearCase to Rational Team Concert, the ClearCase Version Importer performance report: Rational Team Concert 4.0.6 release report shows basic data on how long an import may take.

The Collaborative Lifecycle Management performance report: RTC 4.0.6 release shows that 4.0.6 RTC performance is comparable to 4.0.5.

zosbuildnos

For the RTC for z/OS 4.0.6 release, the Rational Team Concert for z/OS Performance in 4.0.6 report  shows that 4.0.6 performance is similar to 4.0.5. For 4.0.6 There were enhancements made to RTC for z/OS queries so that they use the Java JFS API: “In the scenario of the incremental build (request build after changing all the copybook files), the “Collecting buildable files” activity in preprocessing time improved about 25%, and result in an about 5% improvement in total run time.”

The Collaborative Lifecycle Management performance report: RRC 4.0.6 release report shows that RRC 4.0.6 performance is comparable to 4.0.5.

Similarly, the Collaborative Lifecycle Management performance report: Rational Quality Manager 4.0.6 release shows that RQM 4.0.6 performance is comparable to 4.0.5.

The CLM Reliability report for the 4.0.6 release demonstrates the capability of a Standard Topology (E1) configuration under sustained 7-day load.

The Collaborative Lifecycle Management performance report: Export Transform Load (ETL) 4.0.6 release report demonstrates that there are no performance regressions in 4.0.6 ETL performance compared to 4.0.5. The 4.0.6 ETL functionality is more comprehensive than it was for 4.0.5, and so there are situations where ETL times may have increased although the data now indexed is more complete and accurate.

Comments, questions? Use the comments box on the actual reports themselves.

 

Field notes: It isn’t always about virtualization, except when it is

Talking recently with some customers, we discussed the fallacy of always trying to solve a new problem with the same method that found the solution of the proceeding problem. Some might recall the adage, “If you only have a hammer, every problem looks like a nail.”

hammer_1

I am often asked to help comprehend complex performance problems our customers encounter. I don’t always have the right answer, and usually by the time the problem gets to me, a lot of good folks who are trained in solving problems have spent a lot of time trying to sort things out. I can generally be counted upon for a big-picture perspective, some non-product ideas and a few other names of folks to ask once I’ve proved to be of no help.

A recent problem appeared to have no clear solution. The problem was easy to repeat and thus demonstrable. Logs didn’t look out of the ordinary. Servers didn’t appear to be under load. Yet transactions that should be fast, say well under 10 seconds, were taking on the order of minutes. Some hands-on testing had determined that slowness decreased proportionally with the number of users attempting to do work (a single user executing a task took 30-60 seconds, two users at the same time took 1 to 90 seconds, three users took 2-3 minutes, etc.).

So I asked whether the environment used virtualized infrastructure, and if so, could we take a peek at the settings.

Yes, the environment was virtualized. No, they hadn’t looked into that yet. But yes, they would. It would take a day or too to reach the folks who could answer those questions and explain to them why we were asking.

But we never did get to ask them those questions. Their virtualization folks took a peek at the environment and discovered that the entire configuration of five servers was sharing the processing power customarily allocated to a single server. All five servers were sharing 4 GHz of processing power. They increased the resource pool to 60 GHz and the problem evaporated.

I can’t take credit for solving that one. It was simply a matter of time before someone else would have stepped back and asked the same questions. However, I did write it up for the deployment wiki. And I got to mention it here.

CLM 4.0.5 performance reports

To coincide with the CLM 4.0.5 release, the performance team has produced six — that’s six! — reports.

Collaborative Lifecycle Management performance report: RTC 4.0.5 release compares the performance of an unclustered Rational Team Concert version 4.0.5 deployment to the previous 4.0.4 release. With the workload as described in the report, we found no performance regressions between current release and prior release.

RTC for z/OS recently introduced a Java API to access JFS (Jazz Foundation Services) instead of using the HTTP APIs which provides the potential for significant performance improvements. The Rational Team Concert for z/OS performance impact of Java JFS API adoption in 4.0.5 report compares the performance before and after RTC for z/OS adopted the Java JFS API in part of the resource operations and queries. Comparison is made between the 4.0.5 RC1 development version and the previous 4.0.4 release.

Collaborative Lifecycle Management performance report: RRC 4.0.5 release compares the performance of an unclustered Rational Requirements Composer version 4.0.5 deployment to the previous 4.0.4 release.

Similarly, Collaborative Lifecycle Management performance report: Rational Quality Manager 4.0.5 release compares the performance of an unclustered Rational Quality Manager version 4.0.5 deployment to the previous 4.0.4 release.

There were design changes to ETL functionality for RM which are highlighted in Collaborative Lifecycle Management performance report: Export Transform Load (ETL) 4.0.5 release. This report presents the results of “Extract, Transform, and Load” (ETL) performance testing for CLM. The ETL type includes Java ETL and DM ETL. Data load includes full load and delta load. The article focusses on ETL performance comparison between the 4.0.5 release and the 4.0.4 release.

Finally, the CLM Reliability report: CLM 4.0.5 release presents a sample of the results from a CLM 405 Reliability test. Reliability testing is about exercising the CLM applications so that failures are discovered and removed before the system is deployed. There are many different combinations of pathways through the complex CLM application, this test scenario exercises the most likely use cases. The use cases are put under constant load for a seven day period to validate that the CLM application provides the expected level of service, without any downtime or degradation in overall system performance.

 

 

Two new performance reports for CLM 4.0.4

Timed with the release of CLM 4.0.4, two new performance reports are posted on the Deployment wiki. These are part of Rational development and the Rational Performance Engineering team’s plan to release relevant and useful reports more frequently.

Collaborative Lifecycle Management performance report: RTC 4.0.4 release compares the 4.0.4 release to the previous 4.0.3 release. The performance goals are to verify that there are no performance regressions between the 4.0.4 and 4.0.3 releases when running tests using the 1100-1200 concurrent user workload as described. The report shows similar throughput for the 4.0.4 release and nmon data comparing 4.0.3 and 4.0.4 show similar CPU, memory and disk utilization on application server. The database shows similar CPU and disk utilization, but higher memory utilization.

Collaborative Lifecycle Management performance report: RRC 4.0.4 release compares the 4.0.4 release with the prior 4.0.3 release to verify there are no performance regressions using the 400-user workload as described. The report shows that there are several actions in 4.0.4 which are faster than in 4.0.3.

More reports are coming, so keep an eye on the Performance datasheets and sizing guidelines page.

New performance material at the jazz.net Deployment wiki

As promised, we have started to publish some datasheets and reports on the jazz.net Deployment wiki. In between the necessary work of qualifying and testing new releases, the team has explored some more complex scenarios. Some of these explorations are responses to customer requests, so keep letting us know what’s important to you. Others are topics which have sat dormant in our backlog and we’ve only just recently been able to align resources to achieve them.

* * *

percentage

Plan loading improvements in RTC 4.0.3

Everybody works with plans in RTC, whether tiny team plans to multi-year large plans with thousands of items and complex team/owner/plan item relationships. When working with larger, more complex plans, some clients noted sub-optimal performance of plan loading. They sought techniques to optimize their plan usage, and also asked IBM to improve plan load response times.

For the Rational Team Concert 4.x releases, the development team made significant improvements to plan loading behavior. Significant changes were made to 4.0.3. This case study compared identically structured plans of varying sizes with the goal of determining the difference in plan load time between RTC 3.0.1.3 and 4.0.3.

“Using the 3.0.1.3 release the larger plans took more than a minute to load while in 4.0.3 all plans, regardless of size or browser used, took less than a minute to load. In this study plans of all sizes loaded faster in 4.0.3 than in 3.0.1.3. Notably, plans with larger numbers of work items loaded proportionally faster in 4.0.3.”

See Case study: Comparing RTC 3.0.1.3 and 4.0.3 plan loading performance: https://jazz.net/wiki/bin/view/Deployment/CaseStudyRTCPlanLoadingPerformance

* * *

Sharing a JTS server between RTC, RQM and RRC

Key to a successful CLM implementation is separate application servers (Rational Team Concert (RTC), Rational Quality Manager (RQM) and Rational Requirements Composer (RRC)) sharing the same JTS server. Folks have asked about the breaking point of a shared JTS server. From the report:

“Overall, for this set of machines and these workloads, the JTS never became a bottleneck. There was only a small amount of degradation in maximum throughputs (5-10%) even when all 5 CLM servers were at maximum load.”

Throughput was measured in transactions-per-second and graphs show the different combinations of servers connected to the single JTS and the relative loads and transaction rates.

Visit Performance impact of sharing a Jazz Team Server: https://jazz.net/wiki/bin/view/Deployment/PerformanceImpactOfSharingAJazzTeamServer

* * *

Sizing VMware

Everyone is using virtualization, and VMware’s ESX is popular for deploying Linux and Windows OSes. We offer stated minimum suggested CPU sizes which are applicable to both physical and virtual servers. This particular report looks at the performance impact of varying vCPU sizes of VMware VMs which are serving Rational Team Concert.

“In this study, we were using dual 6-core physical hyper-threaded CPUs that were not able to be translated to 12 or 24 vCPUs within the virtual environment. We found better performance using 16 vCPUs in our Virtual Machines.”

Look at Performance impact of different vCPU sizes within a VMWare hypervisor: https://jazz.net/wiki/bin/view/Deployment/PerformanceImpactOfvCPUSizes

* * *

RRDI 2.0.x sizing guidelines

One of my Jumpstart colleagues wrote a note about sizing the Rational Reporting for Development Intelligence (RRDI), an essential ingredient in most CLM deployments. To properly size RRDI requires understanding the approximate number of concurrent users and estimating how many of them might interact with reports.

Take a look at RRDI 2.0.x sizing guidelines: https://jazz.net/wiki/bin/view/Deployment/RRDISizingGuidelines

* * *

The future

We plan to post more reports as they are completed on the wiki here: https://jazz.net/wiki/bin/view/Deployment/PerformanceDatasheetsAndSizingGuidelines. As always, let us know what you think is missing or what you’re interesting in hearing more about. You can ask here or on the wiki itself. Thanks.

 

 

Wet Paint: Behind the Jazz.net Deployment wiki

The Deployment wiki

As previously alluded, we’re trying to change the model for how we communicate CLM and Rational product information specific to deployment. Presumably we could discuss what we mean by deployment, but the jazz.net Deployment wiki says it best:

“… a single repository for Rational deployment guidance and best practices for customers, partners, and Rational staff beyond the basic product and support documentation. The scope of the wiki is the full range of Rational products and solutions, but it focuses on advice for those that are most commonly deployed. The wiki assembles deployment information from many locations into a single community-developed and community-managed knowledge base.”

But really, what’s it for?

Having spent some time behind the curtain helping to get the wiki in order, I can speak to some of the behaviors we’re trying to improve:

  1. Bring accurate, relevant information to customers and practitioners
  2. Create a community where qualified experts can share their knowledge

We’re trying to consolidate the many different ways customers can find Rational product information, improve the shelf life of that information, and find a way to incorporate practitioner experience.

What’s wrong with the old model? (Well, nothing really…)

Back when, you’d open the box with the product diskettes and get a thick tome or two explaining how to install and use the software. Printed documentation seems to have gone the way of cassettes and eight-track tapes. Online documentation seems to be the current trend. Yes, trees are saved and cycle time reduced. I’m not completely sure that wikis represent the state of the art for documentation and knowledge management but this isn’t the place for that discussion.

Going about this the right way

No surprise that those of us working on the wiki are some of the most articulate of the bunch. However, we are a very self-conscious crowd even if we’re trying to impose our view of the world. We’re constantly asking each other to review what we write and collate. We want to get the facts right. We want to help our customers.

As we keep our goals in mind, there are a few things we worry about:

  1. Too much frequent, spurious change
  2. Readers misunderstanding or misusing what we say
  3. Making statements which are just plain wrong

Readers arrive at wikis well-aware that the content may be in flux. Still, we want to minimize trivial changes. At the deployment wiki there’s a wrapper of legal statements and terms of use which protect all of us. Much of what we publish is driven by common sense. We’re also constantly keeping an eye on what each of us do, not to police, but to be sure we’re doing our best work everywhere we can.

Humble beginnings

My work on the wiki started with the Troubleshooting Performance section. We assembled an excellent International team straddling development and customer support. At the end of February 2013, the Performance Troubleshooting Team locked itself in a room for one week. That we’d effectively take a clean sheet of paper and embark on a conceptually difficult topic was perhaps over-reaching, but we got an immense amount of work done. Supplied with generous servings of coffee, donuts and bagels, the team defined wiki structure, authored dozens of pages and figured out several necessary behaviors useful for the wiki’s more straightforward pages.

You might have thought we’d start on more direct material. That is, begin with “How to make a peanut butter and jelly sandwich,” rather than jump right into “Let’s figure out why you’re having trouble making a sandwich.”

Very often, the best instructions are sequential:

  1. Desire a specific result
  2. Assemble ingredients
  3. Follow sequential steps
  4. Arrive at result

Inventing theory

Troubleshooting presumes something, anything, may have gone wrong, and requires a diagnostic vocabulary. We had to write pages on how to solve problems, before we even picked the problems we wanted to solve. We had to agree on how to frame issues so that readers could use our pages to comprehend their own concerns and use our pages for help.

  1. Desire a specific result
  2. Assemble ingredients
    1. How do you know if you don’t have all the ingredients?
    2. Might some of the ingredients be faulty?
  3. Follow sequential steps
    1. How do you know if you missed a step?
  4. Arrive at result
    1. How might you determine you haven’t achieved the result?

We settled upon situations as the name for the things we were trying to help solve and write about. We tried to come up with reasonably narrow situations, and applied some of our Agile techniques to frame situations:

“As a <user> trying to do <X action>, I encounter <Y performance problem>.”

We discovered that situations fell into two categories: situations which administrators were most likely to encounter (not necessarily product specific, more likely to be environmental) and situations which users were most likely to encounter (more likely to be product specific, not necessarily environmental).

Given performance problems often appear to end-users as a culmination of several problems, we decided that the situations we would discuss had to be discrete and specific: “How to troubleshooting performance of <feature>,” or “How to diagnose performance problems in <non-product component within an environment>.”

Because the pages are on an easily discoverable public wiki, the material had to be reasonably self-contained with clearly defined scope. We also had to add hints for where readers would go next should our pages not actually prove to be helpful.

We thought a lot about who our audience might be: The administrator situations and user situations seem obvious now, but they weren’t our original design.

Admittedly, maybe I worried more about theory than most. Or at least, the team let me worry about it while they were doing lots of research and writing.

Actual practice

Many of the situations required researching and writing background pages to explain key concepts even before troubleshooting started. A page on performance troubleshooting ETLs required explaining how to read ETL log files. A page on slow performance in web browsers required material about how to measure browser performance independently of CLM. A discussion of troubleshooting virtualized configurations required an introduction to virtualization concepts.

Each situation proposes an initial assessment to identify symptoms, impact/scope, timing/frequency and any other environmental changes. These are the basics of troubleshooting. Then there might suggestions about how to collect data, use tools, or complete an analysis. Finally there are the paragraphs readers expect where possible causes and solutions are itemized. Not every cause and solution applies to everyone. We didn’t want to create a greater burden for support by suggesting things that were wrong, irrelevant or confusing.

As mentioned above we also wanted to make sure readers could find their way to support, other IBM and generally available material. As much as we worried about a top-down structure (entry pages, secondary pages, etc.), we are well-aware that folks might arrive at a page from an Internet search.

For consistency, we created a template for writing our pages. At the end of our troubleshooting template are Ground Rules. They look simple, but they are easily internalized and effective.

Ready right now, and preparing for the future

We’re desperate for feedback. Each page has a comments section which we monitor. We’re considering updating the wiki platform with some “like” capability, as well as tie it closer to jazz.net’s forum.

We created a page where in-progress articles can be found, and suggestions for potential troubleshooting topics can be made.  So please visit the wiki and let us know what you think.