The emergence of machine learning applications has brought the new demands for high performance computing in cloud environment. Besides accelerators, such as GPU or TPU, fast interconnection among and within computers becomes more and more important to achieve efficient training and learning. One of the high-bandwidth, low-latency interconnection architectures is InfiniBand. However, software based virtualization of InfiniBand in QEMU/KVM suffers large performance degradation owing to virtualization overhead and memory allocation problem. In this paper, two techniques, doorbell mapping and memlink, are proposed to optimize the performance the InfiniBand virtualization on QEMU/KVM. Doorbell mapping allows the applications in guest user-space to access the doorbell memory page directly so that the virtualization overhead is minimized. Memlink ensures memory contiguity after virtualization, which is a critical requirement for zero-copy between guest and host. Experiments show that the virtualized InfiniBand with mmap and memlink can achieve near native performance for large data transmissions. Comparing to the previous InfiniBand virtualization on QEMU/KVM, our implementation obtains over 3.5 times performance improvement in various benchmarks.