I am happy to announce that a paper of mine will appear at the 11 International Conference on Service Oriented Computing (ICSOC 2013) which will take place in Berlin, in December 2 – 5, 2013 . The paper, titled “Open Source Versus Proprietary Software in Service-Orientation: The Case of BPEL Engines“, is joint work with my fellow PhD student Simon Harrer. Simon will also be the one presenting the paper in Berlin. A preprint is available here.
The paper is one step further in our work on benchmarking process engines. As before, we focus on standard conformance and the paper marks a significant extension to our work presented at last years SOCA. We complement our evaluation of open source engines with an evaluation of proprietary ones and found some surprising (at least for me) results. As before, we use our benchmarking tool betsy, which we continuously work on, to compute the data. For the paper, we extended the test set that is used by betsy significantly. We implemented a test category for benchmarking the pattern support provided by an engine, more precisely the support for workflow control-flow patterns. These patterns are quite well-known in the workflow community and as I have some previous exposure , to them, this was a clear choice. The point of the patterns is to characterize the expressiveness of the language dialect supported by an engine; that is, to characterize how well the engine supports things typically needed in workflows.
Conformance in General
Here, I just want to provide a very high-level overview of the results, all the details can be found in the paper, which I want to encourage you to have a look at. Contrasting the amount of successful conformance tests between proprietary and open source engines, we found that proprietary engines perform better; that is, they successfully pass a significantly higher amount of tests. Given my experience with these engines, I would have expected the reverse result. The difference diminishes to an insignificant level, if we only consider the top open source engines which are on par to the proprietary ones. Two of the open source engines we look at seem to be in a rather non-mature state (although they are declared as mature) and pass only a little amount of tests, dragging the overall rating of open source engines down. The selection of the engines might be counted as a bias in the paper, but lacking usage data we can hardly do anything but consider all engines equally, hence the above result.
Expressiveness (aka Pattern Support)
When it comes to the support for control-flow patterns, we found that all engines have a fair degree of support, even the ones that do not perform well in general. Most control-flow patterns can be implemented by simple combinations of basic control-flow structures, so support is wide-spread. One decisive aspect, however, that needs to be present for pattern support and which is lacking in some engines, is support for truly parallel execution of activities.