As the scale of cloud data centers becomes larger and larger, the energy consumption of data centers also grows rapidly. Dynamic consolidation of Virtual Machines (VMs) presents a significant opportunity to save energy by turning off idle or under-utilized Physical Machines (PMs) in data centers. In this paper, we present a multi-agent based architecture for performing dynamic VM consolidation task. The architecture uses a local agent in each PM to decide when a PM becomes overloaded using reinforcement learning approach. Moreover, a global agent is proposed as a supervisor to dynamically optimize the VM placement based on the local agents' decisions. Therefore, agents cooperate together to minimize the number of active PMs according to the current resource requirements. Experimental results on the real workload traces from more than a thousand Planet Lab virtual machines show that the proposed architecture can reduce the energy consumption and maintains the required performance level in a large-scale data center.