Fraud detection in financial systems remains a critical challenge due to highly imbalanced data, evolving fraudulent tactics, and strict privacy constraints that limit the availability of data. Traditionally, tree based models such as random forests, XGBoost, and LightGBM have been the backbone of fraud detection, offering robust performance through extensive feature engineering. However, recent advances in large language models (LLMS), pretrained on massive corpora and endowed with powerful in-context learning capabilities suggest that these models can be leveraged to enhance fraud detection even in low-data regimes. In this study, we explore the applications of LLMs to fraud detection on tabular data by converting structured inputs into natural language through various serialization techniques, including list templates, text templates, and a markdown-based t-table format. This conversion enables LLMs to exploit their pre-trained knowledge for zero-shot and few-shot learning scenarios. We evaluate the impact of different serialization methods on model performance and examine the sample efficiency of LLMs relative to conventional tree-based models. Our experimental results demonstrate that LLMs achieve competitive performance on fraud detection tasks, particularly when data is scarce, and offer a promising alternative to traditional approaches. This work provides valuable insights and guidelines for deploying LLMs in real-world financial applications, paving the way for more efficient, data driven fraud detection systems.