The Mythological Transactional Business Process Design of Experiments

This is part one of a series of articles on how to apply conjoint analysis to a transactional business process management. Part one deals with Delphi’s application of a screening conjoint Design of Experiments.

Over the last two years, Delphi’s Master Black Belt community developed Six Sigma Transactional Black Belt training materials in an effort to improve its business processes. The Master Black Belts felt strongly that Design of Experiments (DOEs) should be included in the transactional curriculum, but couldn’t find suitable non-manufacturing examples to include in instructional materials. So the Master Black Belts debated about whether to use manufacturing examples or create generic factors and levels. Neither one of these options was optimal, and could be a "turn off" for the transactional candidates. In the end, generic factors and levels were chosen.

The 2008 IQPC Lean Six Sigma and Process Improvement conference provided Delphi with the opportunity to query Lean Six Sigma consultants about the application of Design of Experiments to transactional or business examples. No one had good examples, although most consultants stated that Design of Experiments was widely used in a business environment. The examples were geared towards retail improvements (e.g. how best to configure a retail environment to maximize sales), which are more closely aligned with a typical conjoint Design of Experiments application. In his book, Lean Six Sigma for the Service Industry, Michael L. George states, "Lean tools aren’t the only ones that are often overlooked. Another under-used tool is Design of Experiments (DOE). Design of Experiments is a method for simultaneously investigating anywhere from a handful to dozens of potential causes of variation in a process. Design of Experiments used to be solely the domain of the statistician, but simple software tools have made it accessible to many." So why is Design of Experiments so under utilized in a business environment and why are true business examples so hard to come by?

Delphi Introduces Conjoint Analysis

This last year, Delphi has realized that a marketing analysis tool—conjoint analysis—could be adopted and applied to business processes. Conjoint analysis could be used to quantify and to optimize business process options and decisions which were previously thought of as unquantifiable. Delphi has found that conjoint analysis is an excellent tool to quantify the Voice of the Customer and is an effective tool to link business strategies to research and data.

Conjoint analysis is a relatively new and evolving methodology which was first introduced to the marketing community in 1971 by Paul E. Green, Professor Emeritus of Marketing, University of Pennsylvania, Wharton and Vithala R. Rao, Professor of Management, Professor of Marketing and Quantitative Methods Johnson Graduate School of Management, Cornell University. Professor Green first coined the phrase "Conjoint Analysis" in 1978 and developed conjoint analysis as a marketing tool. It grew from psychometrics (the alleged ability to obtain information about a person or event by touching an object related to that person or event) and discrete choice modeling. Traditionally, conjoint analysis is a human behavioral model (Design of Experiments) that focuses on human reactions to potential products/services. The method forces respondents to consider all attributes together (jointly) and forces respondents to trade-off competing values and needs. This captures the essential dilemma of market choice: the perfect product is seldom available, but lesser alternatives are. Typically, conjoint analysis is used to configure a product or service to maximize customer desirability and sales.

Conjoint analysis forces a respondent to consider combinations of attributes (inputs) and level settings as a combined set, which is often referred to as a card or profile (this is analogous to a run in a standard Design of Experiments). The respondent then is forced to assign a numeric rating to each profile for overall desirability (0 = least desirable, and 100 = most desirable). (Click on diagram to enlarge.)


Figure 1: First and last cards used for the conjoint DOE and the respondent’s thought process

Delphi's Area For Improvement: Its Change Management Process

Over the years, Delphi has continued to focus on technology development, which has evolved to meet market needs. The same focus was not applied to Delphi’s business processes. It is well known that a broken business process can prevent companies from being best in class. Delphi’s Change Management process is a business process that touches all aspects of the development and production value streams. At its conception, the Change Management process was designed to comply with the standard Configuration Management II process. Over the years, it was modified to prevent defects from passing through the process. Changes made were designed to adapt to the lowest common denominator; in other words, checks in balances were added for every issue that arose. Soon, the process became very complicated, unmanageable, and the most disliked process within Delphi. As a result, the Change Management process was selected as an area for improvement.

Preliminary Step: Gathering Voice of the Customer

To better understand the major issues with the process, a survey was designed and delivered to the users of the Change Management process to gather the Voice of the Customer. Affinity diagrams were used to create a Pareto of issues. The top issues identified were 1) Too many approvals, 2) The Change Management organizational structure, 3) Process rules / procedures, 4) Cost of the process and 5) Time to process a change request. These five areas became the focus of the improvement investigation. In-person interviews quickly revealed that users had their own very strong opinions of how the Change Management process should be changed. Unfortunately, these opinions were not aligned. The MBB’s working on the project needed an analytical approach to quantify these opinions to understand their effect on the response variables, which in this case was the rank assigned to each group of attributes at specific levels. Conjoint analysis was selected as the tool to quantify and statistically evaluate the effect of various changes to the process.

Employing Conjoint Analysis to Improve the Change Management Process

The CM process example had five attributes: time (4 levels), cost (4 levels), approvals (3 levels), organizational structure (4 levels), and change process rules (4 levels). A full factorial Design of Experiments would involve 768 cards (Design of Experiments runs; 4 x 4 x 3 x 4 x 4 = 768). 768 cards are too many for a single respondent to rank reliably. Respondent fatigue is a major issue in conjoint analysis and a typical conjoint study should be limited to less than 30 cards. This had a large impact on the Design of Experiments design, resulting in a highly fractionated Design of Experiments design where only main effects and certain selected interactions could be obtained. Other assumptions may be required with aliasing, such as assuming that all effects are due to main effects and not interactions.

Table 1 details the conjoint analysis and their respective levels. The levels for each attribute represent the expected performance. The respondents will have opinions regarding each attribute, some of which may be offered as an available option and others that are not. (Click on diagram to enlarge.)


Table 1: Attributes and levels (CR = Change Request, CO = Change Owner, PDP = Product Development Process)

Table 2 represents the conjoint analysis (Design of Experiments) layout. The fractional factorial design contains 21 of the 768 combinations. The combination of attributes/levels was created by the statistical software program. The response variable is the rank that each respondent assigns to the combination of five attribute levels represented in each row of the table. (Click on diagram to enlarge.)


Table 2: Conjoint Analysis (DOE) Layout for Change Management Project

Figure 2 shows a card which represents the first combination of attributes levels in the Design of Experiments; 20 other cards were created to represent the other combinations. Respondents must consider the combination of attribute levels on the card and rank that combination as it compares to the other 20 combinations. This ranking process forces the respondents to make tradeoffs which represent their preferences as to how the Change Management process should be improved. The rankings of the respondents can be compared using statistical software to determine the statistical significance of each attribute and its corresponding preferred level setting. (Click on diagram to enlarge.)


Figure 2: First card used for the conjoint DOE

Conjoint Analysis Results on the Change Management Process

The results of this screening conjoint analysis on the Change Management process revealed that respondents ranked time, approvals, process rules, cost and organizational structure in importance order (most significant to least significant). They did not have a strong (significant) preference regarding the support staff organizational structure. This is important because it allows flexibility with regard to how the Change Management organization is configured. Time, approval level, process rules (procedures) and cost were statistically significant at alpha of 0.25 (since conjoint analysis is inherently "noisy" a lower alpha was chosen for this screening Design of Experiments). Analysis results are shown in Figure 3 and Table 3. Figure 3 shows the main effects plots; the preferred levels are indicated by the points with the highest means or ranks. Table 3 shows the Analysis of Variance for the Design of Experiments. Attributes time, cost, approvals, and process rules are statistically significant. (Click on diagram to enlarge.)


Figure 3: Minitab main effects plot for the conjoint DOE


Table 3: Analysis output for the screening conjoint analysis

While the analysis provided much insight into the employees’ change process preferences, a recurring concern expressed was that none of the combinations contained the optimal combination that would have been selected by the respondent. The respondent repeatedly wanted to create a new combination; one that contained their individual preferences for each attribute. However, Delphi found that conjoint analysis is an effective tool because it forces the respondents to make trade-offs. If respondents were allowed to express every preference, the researchers would simply be collecting opinions and not able to devise an appropriate course of action!

Additional resources:

• Yoram (Jerry) Wind & Paul E. Green (2004), Marketing Research and Modeling: Progress and Prospects, Norwell Massachusetts: Kluwer Academic Publishers.
• Bryan K. Orme (2006), Getting Started with Conjoint Analysis: Strategies for Product Design and Pricing Research, Madison, WI, Research Publishers LLC.
• Joseph F. Hair, Jr., William C. Black, Barry J. Babin, Rolph E. Anderson, Ronald L. Tatham (2006), Multivariate Data Analysis 6th Ed., Upper Saddle River, NJ: Pearson Prentice Hall.
• Mary Loye (2000), Preference Structure Measurement: Conjoint Analysis and Related Techniques, Intelliquest: New York, NY
• Edward F. McQuarrie (1996), The Market Research Tool Box: A Concise Guide for Beginners, SAGE Publications, Inc: Thousand Oaks, CA