Many future decision support systems will be human-centric, i.e., require substantial human oversight and control. Because these systems often provide critical services, high assurance will be needed that they satisfy their requirements. How to develop “high assurance human-centric decision systems” is unknown: while significant research has been conducted in areas such as agents, cognitive science, and formal methods, how to apply and integrate the design principles and disparate models in each area is unclear. This paper proposes a novel process for developing human-centric decision systems where AI (artificial intelligence) methods-namely, cognitive models to predict human behavior and agents to assist the human-are used to achieve adequate system performance, and software engineering methods, namely, formal modeling and analysis, to obtain high assurance. To support this process, the paper introduces a software engineering technique-formal model synthesis from scenarios-and two AI techniques-a model for predicting human overload and user model synthesis from participant studies data. To illustrate the process and techniques, the paper describes a decision system controlling unmanned air vehicles.