An ecological community network generally has a clustering structure formed by evolutionary and ecological processes. Because females and males of a single visitor species often differ in their evolutionary and ecological relationship with flowering plants, visitor sex should affect the clustering structure of flower–visitor networks. Two related metrics are used to evaluate clustering structures: compartmentation and modularity. Compartmentation refers to the number of clearly separate subgroups, whereas modularity describes subgroups according to their number of aggregating links. Thus, compartmentation is a measure of network fragmentation and modularity is a measure of network heterogeneity. Because male visitors tend to give priority to search for mates, we hypothesized that male visitors increased compartmentation and decreased modularity compared with female visitors. By using museum specimens of flower–visitor insects together with the plant species that they were visiting, recorded at the time of collection, we constructed 11 networks for each research site and collection year, separated each into male and female subnetworks, and then compared the two metrics between them. Results showed that compartmentation was not different between them, while male subnetworks had lower modularity than female subnetworks, and strengthened modularity of species networks together with female subnetworks. These structural characteristics of male subnetworks likely reflect less choosy and less mutualistic visits of males compared with females. This might imply that the role of male visitors in maintaining flower diversity has been overlooked.