Reconfigurable hardware such as FPGAs are being increasingly employed for application acceleration due to their high degree of parallelism, flexibility and power efficiency — factors which are key in the rapidly evolving field of embedded real-time vision. While recent advances in technology have increased the capacity of FPGAs, lack of standard models for developing custom accelerators creates issues with scalability and compatibility. In this paper, we describe a model for designing streaming hardware accelerators with run-time configurability. This model provides a generic interface for each hardware module, a modular and hierarchical structure for parallelism at multiple levels and a run-time reconfiguration framework for increased flexibility. We present case studies to accelerate sample neu-romorphic vision algorithms which are inspired by models of the mammalian visual cortex. Experimental results show speedups of several factors over comparable CPU implementations and higher performance-per-watt over relevant GPU implementations.