Abstract: Defects in requirements specifications can have severe consequences during the software development lifecycle. Some of them result in time and cost overruns due to incorrect or missing quality characteristics, such as security. This characteristic requires special attention in web applications because they have become a target for manipulating sensible data. Several concerns make security difficult to deal with. For instance, (1) when stakeholders discuss general requirements they are often unaware that they should also discuss security-related aspects, because (2) they typically do not have enough expertise in security. This often leads to unspecified or ill-defined security requirements. These concerns become even more challenging in agile contexts, where lightweight documentation is typically produced. To tackle this problem, we designed an approach for reviewing security-related aspects in agile requirements specifications of web applications. Our proposal considers user stories and security specifications as inputs and relates those user stories to security properties via Natural Language Processing. Based on the related security properties, our approach identifies high-level security requirements from the Open Web Application Security Project (OWASP) to be verified, and generates a reading technique to support reviewers in detecting defects. We evaluate our approach via three experiment trials conducted with 56 novice software engineers, measuring effectiveness, efficiency, usefulness, and ease of use. We compare our approach against using: (1) the OWASP high-level security requirements, and (2) a perspective-based approach as proposed in contemporary state of the art. The results strengthen our confidence that using our approach has a positive impact (with large effect size) on the performance of inspectors in terms of effectiveness and efficiency. The files below contains the following data: 1) Consent form.docx: This document provides to the research subjects information to decide whether to participate in a research study or not based on an explanation of the proposed research and the nature of the participation that is requested of them. 2) Characterization_Questionnaire.docx: This document shows the questions used to characterize the working experience and knowledge of the participants of the controlled experiment. The answers obtained through this questionnaire allowed us to identify some key characteristics about three knowledge areas: Agile Software Development, Software Security and Software Inspection. 3) Experiment_Results.xlsx: This Excel document shows a summary of the results of the experiments by participant ID, trial and technique. It also synthesizes the defects found by the reviewers and the characterization, follow up and TAM questionnaire. 4) FollowUp_Questionnaire_AdHoc_Technique.docx: This document shows the questions used to acquire information about the review conducted by inspectors who used the ad hoc technique. 5) FollowUp_Questionnaire_Reading_Technique.docx: This document shows the questions used to acquire information about the review conducted by inspectors who used the proposed reading technique. 6) Repository of Keywords.docx: This document shows the keywords used by our proposed approach and that indicate some security concern. 7) Task_description_AdHoc_A.docx: This document shows textual instructions on how to follow the ad hoc review for a set of user stories. 8) Task_description_AdHoc_B.docx: This document shows textual instructions on how to follow the ad hoc review for another set of user stories. 9) Task_description_RT_A.docx: This document shows textual instructions on how to follow the proposed reading technique for a set of user stories. 10) Task_description_RT_B.docx: This document shows textual instructions on how to follow the proposed reading technique for another set of user stories. 11) Training_Experiment.pptx: This document contains the material used to explain the topics involved in the experiment. It was used to train the subjects.