Onuralp Cifci

Empirical Evaluations in Software Engineering

Why is it important for software practitioners to conduct empirical evaluations?

Empirical evaluations assists practitioners in answering questions, testing claims and making informed decisions based on the results. It allows Software Managers and practitioners to identify what choices are available and conduct an analysis and a development of arguments for and against usage of that software tool. Without undertaking this assessment, software practitioners would have to rely on their peers for direction and, while the advice may be sound, they would not have sufficient evidence and all the facts to confirm its “suitability, limits, qualities, costs, and inherent risks” to the project (Kitchenham, 2005, p.2).

Despite the advantages, “the software industry frequently adopts technologies without first undertaking a structured evaluation” (Rainer, Beecham, 2008 p.2), preferring to rely on previous knowledge and guidance from others in order to save time. In doing so they may find the technology they financed is outdated, doesn’t meet their requirements or they may even find themselves changing their requirements to fit the technology limitations.

SLRs focus on the collection and appraisal of best evidence from research. How does the EBSE method extend the SLR method?

EBSE extends the SLR method of gathering evidence by integrating the results from Step 1 – 3 with experiences, values, system constraints (Step 4), and then assessing this performance in order to improve the evaluation (Step 5). Where SLR is a process of gathering evidence, EBSE enhances it by adding further knowledge and critiquing the completed evaluation. Only by doing this, can Project managers and practitioners have a basis on technology adoption decision.

In Step 4, a way of integrating ‘experiences’ into EBSE is to look for valid practitioner views on the technology and their experiences with it, i.e. the Microsoft Advert that used Phil Pavitt’s experience with Windows Server System and Linux. ‘Values’ are a way of appealing to a personal situation and provoking an idea, possibly by talking about goals values important to the project aims or stakeholders. ‘System Constraints’ are factors that affect the consideration of a technology, be it time, cost, dependencies and conflict.

Step 5 makes the researcher evaluate the work they’ve done in Steps 1 – 4, in order to “encourage the evaluator to reflect on the evaluation they have conducted so that they can improve both their performance during future evaluations and the outcome of (or: the recommendation that results from) those evaluations.” (Rainer, Beecham, 2008, p18)

Sources

Kitchenham, Barbara (1996). DESMET: A method for evaluating Software Engineering Methods and Tools.

Rainer, Austen; Beecham, Sarah (2008) Supplementary Guidelines, Assessment Scheme and Evidence-Based Evaluations of the use of Evidence Based Software Engineering.