Understanding how the visual system processes and categorizes objects is a fundamental question in neuroscience. This study investigated whether early visual areas encode semantic category information independently of low-level visual features. Using fMRI data from the Kay Natural Images dataset, we focused on V1–V4 and LOC. We explored three binary distinctions: animate vs. inanimate, natural vs. human-made, and face-present vs face-absent. We also included an extended face category encompassing partial and animal faces. To address these questions, we used Representational Similarity Analysis (RSA) and controlled for low-level structure with features from AlexNet's Conv2 layer. Our results revealed that both animacy and the extended face category are represented across early, intermediate, and higher visual areas, independent of low- level visual features. Significant encoding emerged in V2–V4 and LOC, highlighting an early prioritization of evolutionarily relevant information within the visual system.