Rap battle is a competition in which two rappers improvise rap verses alternately, and a verse is composed of multiple sentences uttered in one turn by a rapper. In this paper, we propose a method for generating response verses that are semantically related and rhyme with the opponent's verse in rap battles. Our approach uses a language generation model BERT2BERT to generate rap sentences and constructs a verse by appropriately arranging them using a BERT model. When generating rap sentences, it is important to include words that rhyme with a specific word in the opponent's verse, but it is difficult to include such words using a conventional sentence generation model that generates sentences in a forward direction from the beginning of the sentence. To address this issue, our proposed method trains the model to generate sentences in a reverse direction from the end of the sentence, which enables the model to generate rap sentences that highly likely have rhymes at the end. To train the model, we constructed our own rap battle corpus consisting of 6,791 verses. Our experimental results demonstrate that our proposed method outperforms a method that uses a conventional model generating sentences in a forward direction.