This paper describes the design and development of a web accessibility knowledge management tool, known as the Accessibility Evaluation Assistant (AEA) designed to assist novice auditors in the process of an accessibility evaluation. The software incorporates a structured walkthrough approach to guide the auditor through a series of checks for established accessibility principles with the goal of conducting a comprehensive accessibility evaluation. The tool also offers the functionality to tailor the evaluation and prioritise checks based on the needs of two different user groups, or the specific content features of the website. The tool has recently been trialled with a group of 38 undergraduate computing students studying an Accessibility and Adaptive Technologies module with the aim of assessing its reliability and accuracy to validate the method. Some initial conclusions about the reliability and validity of the method and the pedagogical implications of the tool are presented. The results will help highlight the checks that can easily be verified by novices, and those that require a more detailed understanding of accessibility; require informed judgement; or are open to individual interpretation.