Let ${\bf F}(n)$ be a family of partitions of $n$ and let ${\bf F}(n,d)$ denote the set of partitions in ${\bf F}(n)$ with Durfee square of size $d$. We define the Durfee polynomial of ${\bf F}(n)$ to be the polynomial $P_{{\bf F},n}= \sum |{\bf F}(n,d)|y^d$, where $ 0 \leq d \leq \lfloor \sqrt{n} \rfloor.$ The work in this paper is motivated by empirical evidence which suggests that for several families ${\bf F}$, all roots of the Durfee polynomial are real. Such a result would imply that the corresponding sequence of coefficients $\{|{\bf F}(n,d)|\}$ is log-concave and unimodal and that, over all partitions in ${\bf F}(n)$ for fixed $n$, the average size of the Durfee square, $a_{{\bf F}}(n)$, and the most likely size of the Durfee square, $m_{{\bf F}}(n)$, differ by less than 1. In this paper, we prove results in support of the conjecture that for the family of ordinary partitions, ${\bf P}(n)$, the Durfee polynomial has all roots real. Specifically, we find an asymptotic formula for $|{\bf P}(n,d)|$, deriving in the process a simple upper bound on the number of partitions of $n$ with at most $k$ parts which generalizes the upper bound of Erdös for $|{\bf P}(n)|$. We show that as $n$ tends to infinity, the sequence $\{|{\bf P}(n,d)|\},\ 1 \leq d \leq \sqrt{n},$ is asymptotically normal, unimodal, and log concave; in addition, formulas are found for $a_{{\bf P}}(n)$ and $m_{{\bf P}}(n)$ which differ asymptotically by at most 1. Experimental evidence also suggests that for several families ${\bf F}(n)$ which satisfy a recurrence of a certain form, $m_{{\bf F}}(n)$ grows as $c \sqrt{n}$, for an appropriate constant $c=c_{{\bf F}}$. We prove this under an assumption about the asymptotic form of $|{\bf F}(n,d)|$ and show how to produce, from recurrences for the $|{\bf F}(n,d)|$, analytical expressions for the constants $c_{{\bf F}}$ which agree numerically with the observed values.