Bilinear interpolation is widely used in computer vision for extracting pixel values for positions that lie off the pixel grid in an image. For each sub-pixel, the values of four neighbors are used to compute the interpolated value. This presents a challenge since four pixels must be read from the source image memory for each output pixel. This paper presents an architecture, for implementation within FPGA-based vision systems, that takes advantage of the heterogeneous resources available on modern devices to parallelize these memory accesses through efficient distribution of the source image in embedded memories. We show how intrinsic information in the sub-pixel addresses can be used to implement bilinear interpolation efficiently. We then suggest modifications to the architecture for larger image sizes which exceed the memory capabilities of modern FPGAs. The architecture is shown to achieve performance of 250Msamples per second in a modern device.