High-capacity machine learning models, such as Gradient Boosting Machines(GBM) and deep Random Forests, often succumb to the bias-variance trade-offwhen training data is scarce (N < 500). While they offer low bias, they frequentlyoverfit noise or fail to approximate smooth functions due to discrete partitioning.Conversely, classical linear models (OLS) offer stability but lack the capacity tomodel complex dynamics. This paper introduces Influence-Weighted BaggedSplines (IWBS), a specialized ensemble architecture designed for high-complexity,small-data regimes. IWBS combines the flexibility of randomized additive splineswith a novel Out-of-Bag (OOB) Stability Weighting mechanism. Unlike stan-dard bagging, which averages learners uniformly, IWBS penalizes ensemble mem-bers that exhibit high prediction instability on held-out data. We benchmark IWBSagainst fully tuned Tree Ensembles (GBM, Random Forest) and specialized small-data solvers (Gaussian Processes, GAMs, Kernel Ridge Regression) across physical,economic, and biological domains. Results demonstrate that IWBS achieves state-of-the-art performance in signal-rich tasks (Concrete, Moneyball), outperformingboth tree-based methods and kernel smoothers by capturing high-frequency non-linearities without overfitting. Furthermore, we establish the method’s boundaryconditions, showing that in high-noise regimes (Diabetes), global smoothers likeKernel Ridge Regression remain superior to structure-discovery approaches.