The recent leap in Large Language Models (LLMs) has paved the way for several research ideas. LLMs are employed not only for personal use but also in professional contexts to enhance human productivity at work. A significant area of research is human-robot collaboration (HRC), which focuses on developing methodologies for effective interaction between humans and AI-enabled machines. In this regard, exploitation of LLMs appears to be a practical approach. However, these models are susceptible to several limitations, including context-induced errors, the propagation of misleading information, and hallucinations. Such deficiencies impede the seamless application of LLMs in scenarios where a high degree of accuracy is essential. To address this issue, this study introduces a dual-agent system designed to validate the responses generated by LLMs. This novel system is integrated into a framework called “CogniVera”, which facilitates collaborative tasks involving a collaborative robot (cobot) through vocal interactions. This initiative represents a significant advancement in HRC, enabling robots to communicate vocally with human operators during assembly tasks. To evaluate the feasibility of this approach, a focused case study will be conducted, concentrating on the human-robot collaborative task of box assembly utilizing vocal communication. The outcomes of this study are anticipated to yield valuable insights into the efficacy of the proposed dual-agent system in enhancing the reliability and performance of LLMs in practical applications.