Human adenoviruses are the most widely used vectors in gene medicine, with applications ranging from oncolytic therapies to vaccinations, but adenovirus vectors are not without side effects. In addition, natural adenoviruses pose severe risks for immunocompromised people, yet infections are usually mild and self-limiting in immunocompetent individuals. Here we describe how adenoviruses are recognized by the host innate defense system during entry and replication in immune and nonimmune cells. Innate defense protects the host and represents a major barrier to using adenoviruses as therapeutic interventions in humans. Innate response against adenoviruses involves intrinsic factors present at constant levels, and innate factors mounted by the host cell upon viral challenge. These factors exert antiviral effects by directly binding to viruses or viral components, or shield the virus, for example, soluble factors, such as blood clotting components, the complement system, preexisting immunoglobulins, or defensins. In addition, Toll-like receptors and lectins in the plasma membrane and endosomes are intrinsic factors against adenoviruses. Important innate factors restricting adenovirus in the cytosol are tripartite motif-containing proteins, nucleotide-binding oligomerization domain-like inflammatory receptors, and DNA sensors triggering interferon, such as DEAD (Asp-Glu-Ala-Asp) box polypeptide 41 and cyclic guanosine monophosphate-adenosine monophosphate synthase. Adenovirus tunes the function of antiviral autophagy, and counters innate defense by virtue of its early proteins E1A, E1B, E3, and E4 and two virus-associated noncoding RNAs VA-I and VA-II. We conclude by discussing strategies to engineer adenovirus vectors with attenuated innate responses and enhanced delivery features.