The immune system contributes to maintaining the body’s functional integrity through its two main functions: recognizing and destroying foreign external agents (invading microorganisms) and identifying and eliminating senescent cells and damaged or abnormal endogenous entities (such as cellular debris or misfolded/degraded proteins). Accordingly, the immune system can detect molecular and cellular structures with a spatial resolution of a few nm, which allows for detecting molecular patterns expressed in a great variety of pathogens, including viral and bacterial proteins and bacterial nucleic acid sequences. Such patterns are also expressed in abnormal cells. In this context, it is expected that nanostructured materials in the size range of proteins, protein aggregates, and viruses with different molecular coatings can engage in a sophisticated interaction with the immune system. Nanoparticles can be recognized or passed undetected by the immune system. Once detected, they can be tolerated or induce defensive (inflammatory) or anti-inflammatory responses. This paper describes the different modes of interaction between nanoparticles, especially inorganic nanoparticles, and the immune system, especially the innate immune system. This perspective should help to propose a set of selection rules for nanosafety-by-design and medical nanoparticle design.