Applications of Artificial Intelligence (AI) are increasingly integrated into the field of modern education, both in-person and remote, as they enhance and enrich the educational experience. AI introduces a new, innovative model of teaching and learning that offers significant benefits to both students and educators. However, the potential risks associated with the use of AI in education must be investigated and assessed. A significant challenge in AI applications within education is the management and protection of educational-personal data. In this context, this article examines the need to protect the privacy of educational data collected through AI applications and explores the ethical dilemmas and concerns raised by their use, as well as the potential for unfair processing or data breaches.