In this paper, we show that fiction vs non-fiction genre classification can be achieved with very high accuracy using simple readability metrics, which have been extensively studied by linguists for many decades. In addition, we explore the BERT model for this classification and find that, although it can also achieve very high accuracy with the same amount of training data, its results are very hard to understand. We tried many adversarial attacks to break the fine-tuned BERT model but found it to be quite resilient.