We are developing a system for patient management in colorectal cancer, in which a difficult case of non-rigid registration, namely of pre- and post-therapy images, arises. Numerous non-rigid registration algorithms have been proposed in medical image analysis, and we have applied several leading algorithms to our non-rigid registration problem; but with unpromising results. The fundamental reason appears to be that they lack with knowledge of the particular application. We propose a graphical representation of anatomical knowledge relevant for colorectal cancer, and of the ways in which this anatomy may be predicted to change as a result of chemo and radiotherapy. We show how we interleave this representation with an adaptive registration algorithm to make the non-rigid registration result both robust and accurate.