This is the challenge design document for the "Anatomical Brain Barriers to Cancer Spread: Segmentation from CT and MR Images" Challenge, accepted for MICCAI 2020. The goal of this challenge is to identify best methods to segment brain structures that serve as barriers to cancer spread, for use in computer assisted target definition for radiotherapy plan optimization. Target delineation is a major task in the radiotherapy workflow since it influences the overall outcome of the treatment. Accurate delineation of structures for treatment planning is associated with better local tumor control and reduced radiation dose to non-target tissues leading to improved therapeutic index. Accuracy can be improved through automation of target definition by identifying natural anatomical barriers to tumor spread. The barriers are brain structures that are not routinely outlined during the treatment planning for radiotherapy plan optimization. Advanced expertise and time is required to identify anatomy on computed tomography images used for treatment planning. Automation of the barrier delineation will make the treatment planning workflow more efficient leading to improved scheduling and potentially increasing the size of patient panels. With improving precision of radiation dose delivery and increasing number of patients treated in the cancer centers worldwide, algorithmic assistance for the target definition is becoming a necessity. Creating standardized definition of the target and normal anatomy will also create an excellent educational resource for clinicians working at centers with low patient volume, and for medical residents. Segmentation of the brain structures is a challenging task as different structures can be appreciated more or less favorably on different imaging modalities. For example, the skull is typically segmented using a CT image, whereas falx cerebri is better seen on an MR T1-weighted image. Furthermore, as multi-modality images are usually acquired at different time points they could present with subtle differences even for brain imaging. This presents a unique technological challenge as information from multi-modality imaging is used by the radiation oncologist to define the clinical target volume for each individual patient's disease. The proposed challenge will provide participants with 60 cases, each consisting of a planning CT scan and two MR scans (post-contrast T1-weighted and T2-weighted FLAIR volumes) with 12 normal structures previously contoured using co-registered CT and MR at a single institution using an imaging protocol defined for glioma radiotherapy treatment planning. This dataset is valuable to the medical imaging and radiation oncology communities as it could advance algorithm development for automated segmentation of these structures, which in turn could be used for automated radiotherapy target definition.