Robot surveillance requires, in addition to detection, to make sense of what is happening in a real world scenario, which is what humans do with contexts. This is critical when robots have to interact with people. Thus, the main issue is how to model human-like context to be mapped to robots, so that they can mirror human understanding. We propose a context model formalizing its relation with the robot, organized according to different dimensions of the environment. We then introduce the notions of endurants and perdurants to account for how space and time, respectively, aggregate context for humans, and in turn can help the surveillance from robots. To map real world data, i.e., sensory inputs, to our context model, our idea is a system capable of managing both the robots sensors and interacting with sensors from other devices. A possible use case is a robot, using the system fusing sensory inputs and the context model, patrolling an university building. Our contribution is a modelling of context that, while accounting for how human represent it, can be used in robots for surveillance purposes, coupled with a proposed system for exploiting sensors from both the robot and the environment.