A very simple safe-Bayesian random forest

Article English OPEN
Quadrianto, Novi ; Ghahramani, Zoubin (2015)
  • Publisher: Institute of Electrical and Electronics Engineers (IEEE)
  • Related identifiers: doi: 10.1109/TPAMI.2014.2362751
  • Subject: QA273
    arxiv: Statistics::Computation
    acm: ComputingMethodologies_PATTERNRECOGNITION

Random forests works by averaging several predictions of de-correlated trees. We show a conceptually radical approach to generate a random forest: random sampling of many trees from a prior distribution, and subsequently performing a weighted ensemble of predictive probabilities. Our approach uses priors that allow sampling of decision trees even before looking at the data, and a power likelihood that explores the space spanned by combination of decision trees. While each tree performs Bayesian inference to compute its predictions, our aggregation procedure uses the power likelihood rather than the likelihood and is therefore strictly speaking not Bayesian. Nonetheless, we refer to it as a Bayesian random forest but with a built-in safety. The safeness comes as it has good predictive performance even if the underlying probabilistic model is wrong. We demonstrate empirically that our Safe-Bayesian random forest outperforms MCMC or SMC based Bayesian decision trees in term of speed and accuracy, and achieves competitive performance to entropy or Gini optimised random forest, yet is very simple to construct.
  • References (32)
    32 references, page 1 of 4

    [1] Leo Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth, 1984.

    [2] J. R. Quinlan. Induction of decision trees. Machine Learning, pages 81-106, 1986.

    [3] Sebastian Nowozin. Improved information gain estimates for decision tree induction. In International Conference on Machine Learning (ICML), 2012.

    [4] L. Breiman. Random forests. Technical Report TR567, UC Berkeley, 1999.

    [5] Jamie Shotton, Andrew W. Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. Real-time human pose recognition in parts from single depth images. In Computer Vision and Pattern Recognition (CVPR), 2011.

    [6] Gabriele Fanelli, Matthias Dantone, Juergen Gall, Andrea Fossati, and Luc Gool. Random forests for real time 3d face analysis. International Journal of Computer Vision, pages 1-22, 2012.

    15.59±3.46 27.03±1.80 Israeli-Images dataset 15.88±0.86 35.21±2.28 34.54±1.32 33.97±1.91 Animals with Attributes dataset 27.39±2.66 41.96±0.91 41.86±1.38 41.57±0.92 34.93±2.05 38.17±0.92

    [7] Antonio Criminisi, Jamie Shotton, and Ender Konukoglu. Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semisupervised learning. Foundations and Trends in Computer Graphics and Vision, 7(2-3):81-227, 2012.

    [8] Tin Kam Ho. The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8):832-844, 1998.

    [9] Wray L. Buntine. Learning classification trees. Statistics and Computing, 2:63-73, 1992.

  • Metrics
    views in OpenAIRE
    views in local repository
    downloads in local repository

    The information is available from the following content providers:

    From Number Of Views Number Of Downloads
    Sussex Research Online - IRUS-UK 0 289
Share - Bookmark