Developing secure software is essential for protecting passwords and other sensitive data. Despite the abundance of cryptographic libraries available to developers, prior work has shown that developers often unknowingly misuse the provided Application Programming Interfaces (APIs), resulting in serious security vulnerabilities. Eclipse CogniCrypt is an IDE plugin that aims at helping developers use cryptographic APIs more easily and securely by providing three main functionalities: (1) it provides a use-case oriented view of cryptographic APIs and guides the developer through their configuration, (2) it generates the code needed to accomplish the chosen use case based on the selected choices, and (3) it continuously analyzes the developer’s code to ensure that no API misuses are introduced later. However, so far the effectiveness of CogniCrypt was never empirically evaluated. In this work, we fill this gap through a controlled experiment with 24 Java developers. We evaluate the tool’s effectiveness in reducing API misuses and saving developer time. The results show that CogniCrypt significantly improves code security and also speeds up development for cryptograph-related tasks. The feedback received during the study suggests that developers particularly appreciate CogniCrypt’s code generation. Its static-analysis is valued for keeping the code up-to-date. Yet, the further integration of generated code into a developer’s project still presents a major challenge. Nonetheless, our results show that CogniCrypt effectively helps application developers produce more secure code.