Malware families often use the Domain Generation Algorithm (DGA) to communicate with the Command and Control (C&C) servers. Although machine learning and deep learning based methods have achieved good accuracy in DGA detection task, it has problems on the new DGA families with limited datasets. In this paper, RL-Gen, a Reinforcement Learning (RL) framework, is proposed to improve the performance of character-level text generation with few input samples. RL-Gen has two modules, W-Generator and Evaluator. W-Generator is an improved generation model based on WGAN-GP, which is regarded as an agent, and Evaluator acts as an environment to evaluate the generated text. Especially in DGA case, Evaluator is an effective DGA detection model (ATT-GRU). The parameters’ updating of W-Generator is optimized by the reward from Evaluator, which promotes the generating abilities on speed and quality. Experiments show that the generated DGAs are sufficiently close to real DGAs, and can play an alternative role of real DGAs in detection model training. And RL-Gen gets better quality of text generation more quickly and smoothly than WGAN-GP.