This paper presents a music generation model trained with Bach's chorales and classical music theory rules. Although previous work has shown promising results in generating the four-part harmony, one of the limitations is the frequent appearance of parallel 5th or 8th, which are prohibited in music theory and rarely used in Bach's chorale. To address this issue, we propose an additional loss that minimizes the probability of prohibited patterns, comparing the results with those from inference using a post-hoc probability manipulation to prevent parallel 5th and 8th. The experimental result shows that applying the proposed loss term can help to reduce parallel motion without losing other quality.