How Fraunhofer Judges BPM Suites

It has been over three months since the Fraunhofer IESE’s BPM Analysis 2014 results were revealed, which saw Bizagi heralded ‘Highest Ranked BPM Product’.

Bizagi’s CMO, Jolanta Pilecka, caught up with Dr Sebastian Adam, Head of Business Process Management Study at the Fraunhofer Institute, to find out more about the study and how it will evolve.

JP: What was the overall purpose of the study? What were you investigating, and why?

SA: The goal of the study was to assess a selection of BPM Suites with regard to their product capabilities and their ease of use. This means we wanted to find out what functionalities today’s BPM Suites can offer, and how easily these can be used. Hence, we compared around 20 products from the perspective BPM experts as well as BPM users in eight evaluation categories.

JP: You’ve run the study before. What was different about it this time?

SA: The main difference to the previous study in 2013 is the number of participating vendors. Last time we had investigated nine BPM Suites, this time it was the double amount. We are very happy about this increase of interest in our study, because it allows us to present a more representative picture of the market. Our intention has been to keep the requirements on the products mostly the same as in the first run of the study. However, to avoid making it too easy for the companies participating for the second time, we created a new scenario for the product presentations and modified the individual tasks which needed to be accomplished during the workshops.

JP: How does this study compare to similar analyses of BPM Suites?

SA: A significant difference to some other studies is the focus on what we call "BPM Suites in action". We did not simply check the existence of certain features, but we mainly assessed non-functional aspects such as simplicity, flexibility, integration requirements, and ease of use in an everyday context. In addition, we did not gather the data for our evaluation via online surveys or interviews, but in one-day workshops where we could have a very detailed look at all the BPM Suites.

JP: Can you talk us through your evaluation process and why did you take this approach?

SA: As a very first step we established a set requirement on BPM Suites. We ensured the practical relevance of the requirements by eliciting them from industrial partners of Fraunhofer IESE. Based on the set of requirements, we created a test scenario which gave the structure for the evaluation workshops. During these workshops, the representatives of the vendors were asked to perform certain tasks, for instance initiating a new process, executing a process step on a mobile device, or implementing a parallel execution of process activities. The performance of all the tasks has been rated by BPM experts, including practitioners as well as academics. These ratings have been aggregated and formed the basis for the detailed evaluation we present in the study report. We chose this systematic approach along with well-defined metrics to have a sound foundation for this study, minimizing potential threats to validity. This way we can ensure an objective and meaningful analysis of BPM Suites.

JP: What were your key findings?

SA: In a nutshell we claim that there is currently no perfect BPM Suite, but on the other hand we did not encounter a complete failure. In all the investigated products we identified strengths and weaknesses. In consequence we cannot give a blanket recommendation, but depending on the demands of potential customers, the study is able to guide towards suitable and promising solutions.

JP: You say that take-up of BPM Suites is on the increase. How does your study change or benefit potential buyers of BPM software?

SA: The main benefit of our study is that potential buyers can get access to a neutral and unbiased evaluation of BPM Suites. Since Fraunhofer IESE is not affiliated to any of the participating companies, we have been free to mention the good but also the bad things about each product. We grouped all the requirements into eight categories without prioritizing them. This allows interested readers to assign their own priorities to the aspects of a BPM Suite which are important to the specific situation. We believe that it is not possible to find a single solution fitting to all possible demands. This is the reason why an essential part of our study report is the decision tree. This decision tree allows potential buyers to narrow the set of BPM Suites down to the ones which receive outstanding ratings in the categories which are of high importance.

JP: How does your work impact the business community as a whole?

SA: We hope that the awareness of the importance of providing comfortable solutions to users of BPM Suites increases. In terms of product capabilities almost all the investigated BPM suites provide a high amount of functionalities and are able to fulfil a majority of typical requirements. The more interesting factor is ease of use, where we detect huge differences between the BPM Suites. A challenging task for the future will be to develop ideas how to provide a comfortable way to work with the product without sacrificing flexibility and functionality.

JP: What are your plans for the future of the study?

SA: We are currently concerned with the planning of future studies. We would like to continue our series of product evaluations, because we experience a very positive response to our work so far. Most likely we will shift the focus of future studies a bit towards promising aspects such as adaptive case management as well as the integration of Internet of Things (IoT) and services.

Further reading

Fraunhofer BPMS Study 2014 was first published 30 November 2014 and is available for purchase from the Fraunhofer bookshop. To read the full English version of the Bizagi review, (21 pages, PDF) access it here.