This building was first established as the Church of Our Lady in the second half of the sixteenth century, while Crete was under Venetian rule. During this period, the church belonged to the Augustinian monks and had an associated monastery. After the Ottoman conquest of Rethymno in AD 1646, the church was converted into the Gazi Hüseyin Paşa Mosque, named for the commander of the Ottoman army in Crete from 1646-1656. It was later nicknamed Camii Neratze, meaning "Mosque of Bitter Oranges" in Greek. The mosque is a rectangular structure oriented on the east-west axis of the original church, and features three domes added after the conversion to a mosque. A carved entryway that is likely from the Venetian period is located on the northern wall of the mosque, and a türbe (mausoleum) is located to the south of the mosque. Hüseyin Pasha created a vakf to support the mosque that consisted of the revenue from several settlements near Rethymno, Kissamos, Agios Vasileios, and Chania. The complex included an imaret (soup kitchen), library (converted from the chapel of Corpus Christi), and school, in addition to the mosque (converted from the church of the Augustinian monastery). The conversion of the church into a mosque entailed rebuilding the south wall, adding domes to the roof, and adding a new internal support system for the domes. According to Evliya Celebi, an Ottoman traveller who visited Crete between 1668 and 1671, the mosque complex featured a soup kitchen (imaret) that fed both the rich and the poor. The current minaret was built in 1890/1891 by Giorgios Daskalakis, who also built the bell tower of a nearby church. The mosque was restored in the 1980s, and the mosque is currently used as a concert and lecture hall.