In this thesis we aim to gain better understanding on the working of the belief propagation algorithm designed for graphical models in other computational frameworks like the neural systems, as well as the error associated with loopy belief propagation in Bayesian networks. In the first part, we examine a few recent neural computational models of belief propagation and highlight the significance of these models that demonstrate the viability of performing belief propagation using neural computations by transforming it into a dynamical system. We also propose the idea of implementing the belief propagation in computational models like the Hopfield network through free energy minimisation. It is widely known that exact inference in loopy graphs is computationally difficult and thus there has been a lot of effort spent in the area of developing practical approximate inference algorithms. Loopy belief propagation is a widely used approximate inference algorithm for graphical models. In the second part of this thesis, we analyse the loopy error and propose two exact inference algorithms using belief propagation with loop correction for Bayesian networks with generic loops. We also propose a new approximate inference method called the 2-Pass loopy belief propagation and demonstrate empirically its potential for use as a fast approximate inference algorithm with comparable accuracy to standard loopy belief propagation. We also discuss issues related to its application as an approximate inference method.