Evaluation of the effectiveness of a tool to support novice auditors

The Accessibility Evaluation Assistant (AEA) is a web accessibility knowledge management tool designed specifically to assist novice auditors in conducting an accessibility evaluation. The software incorporates a bespoke structured walkthrough approach designed to guide the auditor through a series of checks based on established accessibility principles with the goal of identifying accessibility barriers. A previous trial examined the effectiveness of the AEA and explored the pedagogical potential of the tool when incorporated into the undergraduate computing curriculum. The results of the evaluations carried out by the novices yielded promising levels of validity and reliability. This paper presents the results of a second experiment designed to test the overall efficacy of the AEA when compared to a WCAG 2.0 conformance review. The results of evaluations produced using both AEA and Conformance Review methods were examined and comparisons made of quality factors such as effectiveness, reliability, efficiency and usefulness. Quantitative and qualitative data from the experiment support continued use of the AEA in an educational context, highlighting the benefits compared to WCAG 2.0 and gives further insight into the complex nature of developing accessibility evaluation skills in novices.