The performance of machine translation systems has improved significantly in recent years, but they are not immune to errors. This motivates the requirement to identify translations in need of human review and post-edits. Most current research focuses on quality estimation, that is predicting translation quality on a continuous numerical scale, which can be difficult to interpret in applied contexts. In contrast, this project focused on predicting binary labels indicating whether translations contain errors that alter the meaning of the original text, also referred to as critical errors. Such errors are likely of high priority for human review and post-edit in vast majority of use cases.