Let $P$ be a set of $n$ points in $\mathbb{R}^d$, so that each point is colored by one of $C$ given colors. We present algorithms for preprocessing $P$ into a data structure that efficiently supports queries of the following form: Given an axis-parallel box $Q$, count the number of distinct colors of the points of $P\cap Q$. We present a general and relatively simple solution that has a polylogarithmic query time and worst-case storage about $O(n^d)$. It is based on several interesting structural properties of the problem, which we establish here. We also show that for random inputs, the data structure requires almost linear expected storage. We then present several techniques for achieving space-time tradeoff. In $\mathbb{R}^2$, the most efficient solution uses fast matrix multiplication in the preprocessing stage. In higher dimensions we use simpler tradeoff mechanisms, which behave just as well. We give a reduction from matrix multiplication to the off-line version of problem, which shows that in $\mathbb{R}^2$ our time-space tradeoffs are reasonably sharp, in the sense that improving them substantially would improve the best exponent of matrix multiplication. Finally, we present a generalized matrix multiplication problem and show its intimate relation to counting colors in boxes in higher dimension.