With advances in sensing technology, automatic recognition of human activities has become a popular research topic. Miniaturised wearable devices can now collect and process the data during activities of daily living. Such technologies rely on algorithms that can effectively combine and interpret wearable sensor data to identify different activities. There are four contributions in this thesis based on a novel wearable device - `Smart Glasses'. The device is able to recognise the subjects' activities of daily living (ADLs) using their first-person vision and motion data. This system consists of a series of algorithms and models. Firstly, we develop first-person video feature extraction algorithms for egocentric vision. The method utilises the optical flow principle on consequent images and extracts the informative motion features, Secondly, we propose dynamic motion detection algorithms to automatically extract the `motion' from the first-person view. The algorithms, based on a Bayesian regression framework known as Gaussian Processes (GP), extract the dynamic portion and related motion tracks of the images. Thirdly, we present a Multi-Scale Conditional Random Field model, which can be applied on top of the conventional approaches. This allows the system to obtain multi-scale context from the sequence of activities. Furthermore, it also has the capacity to integrate multiple sensors into the same system, and potentially increase the system functions. Finally, all the designed algorithms are validated and tested on a wide range of populations, including healthy adults, elders and a mixed disabled patient population, aged between 22 to 89 years old. Our approaches achieve an overall accuracy of up to 89.59% and 84.45% over 12 ADLs for adults and the elderly, and up to 77.07% accuracy on 14 activities from the disabled patients.