Maritime autonomy is rapidly gaining interest both in academic and industrial circles. Safe navigation for autonomous surface vehicles requires a robust and reliable tracking system that maintains and estimates the positions and velocities of other vessels. Sensor fusion is expected to play a key part in this task but requires complex detection and estimation systems. This thesis primarily considers the task of multi-sensor, multi-target tracking for autonomous surface vessels (ASVs) in a littoral environment. The contributions are directed both towards single-sensor, single-target tracking as well as towards a larger, heterogeneous sensor fusion system. The core contribution of this thesis is a complete multi-sensor detection and tracking system leveraging radar, lidar, infrared and electro-optical cameras. Due to different operating modalities, these sensors require different and often complex processing pipelines which are developed and described in detail. Detection performance is evaluated on a large, real-world dataset with multiple targets. In addition, a multi-sensor extension of the Joint Integrated Probabilistic Data Association (JIPDA) multi-target tracker is developed. This is motivated by the differing performance characteristics of the various sensors at differing ranges. For the ASV to best make use of the various sensors the tracking system must be able to account for these differences, motivating the development of a multi-sensor JIPDA. A typical sensor suite onboard an ASV will contain a multitude of sensors, both active and passive, for regulatory and performance reasons. Cameras form an essential part of this package, being mandated for the classification of other vessels and the identification of light signals. This has obvious benefits not only for accuracy but also adds additional layers of redundancy in case of sensor failures. A major issue in this regard occurs if the vessel is left without any active sensors such as radar and lidar. Imaging sensors, being passive, do not provide explicit range information, resulting in an observability problem. The development of detection pipelines and tracking methods for camera tracking has therefore been a key issue for this thesis. Experimental data gathering and evaluation have been major drivers for most of the developments presented in this work and have been used to both validate methods and direct further research. Accurate evaluation of tracking system performance does require experimental work in the form of sensor data recording, however, the actual evaluation can be performed at a later date, enabling rapid iteration on already existing data. A complete autonomy system must also include a collision avoidance system that, depending on what the tracking system outputs, could change ownship behavior. This will influence sensor data output, and therefore also the tracking estimates, making offline verification of the complete system difficult. The final part of this thesis proposes a camera-based autonomy system, combining the developments on camera-only tracking with a collision avoidance system. Experimental validation of this system was performed with a single target, demonstrating the viability of camera-only tracking for ASV navigation.