This paper studies the physical layer security in a multi-user downlink, where a single user is selected for secret transmission during each time frame. Current works usually assume a worst case where all unselected users are curious and act as eavesdroppers, and conclude that no multiuser diversity is achievable for secrecy when the number of users is pretty large. However, the worst case may happen rarely in practice. A general scenario is that several (maybe all) of the unselected users act as eavesdroppers. In this case, selecting the user with the largest SNR (i.e., signal to noise ratio) does not necessarily achieve the maximum secrecy rate. For the general case, we derive the new closed-form expression of the secrecy outage probability, which increases with the number of curious users, and tends to converge in the high-SNR and large-user-number regime. When the number of curious users is supposed to be small, the secrecy outage probability could be any value smaller than one even in the large-user-number regime. These results provide additional insights into the system performance.