Understanding the sources of cross-sectional variation in asset returns remains central to financial economics. This paper introduces a \textit{Cluster-Based Factor Model} (CBFM) that integrates hierarchical clustering and locally linear embedding (LLE) to construct parsimonious, data-driven factors from characteristics-managed returns. The clustering stage identifies groups of highly correlated characteristics managed returns, while the subsequent LLE stage extracts low-dimensional nonlinear factors within each cluster. Factor loadings are then estimated through a neural network framework, enabling flexible, time-varying relationships between latent factors and asset returns. Empirical evaluation using Fama–French 25 value-weighted portfolios shows that the CBFM achieves nearly twice the explanatory and predictive $R^2$ of traditional economic and categorical groupings. Compared with benchmark models such as RP–PCA and standard PCA, the CBFM yields smaller and statistically insignificant alphas, demonstrating superior pricing accuracy and reduced overfitting. These findings highlight the efficacy of data-driven clustering for uncovering the true structure of systematic risk.