In recent years compressive sampling (CS) has appeared in the signal processing literature as a legitimate contender for processing of sparse signals. Natural signals such as speech, image and video are compressible. In most signal processing systems dealing with these signals the signal is first sampled and later on compressed. The philosophy of CS however is to sample and compress the signal at the same time. CS is finding applications in a wide variety of areas including medical imaging, seismology, cognitive radio, and channel estimation among others. Although CS has been given a great deal of attention in the past few years the theory is still naive and its fullest potential is still to be proven. The research in CS covers a wide span from theory of sampling and recovery algorithms to sampling device design to sparse CS-based signal processing applications. The contributions of this thesis are as follows; (i) The analog-to-information converter (AIC) is the device that is designed to collect compressed samples. It is a replacement for the analog-to-digital converter in a traditional signal processing system. We propose a modified structure for the AIC which leads to reducing the complexity of the current design without sacrificing the recovery performance. (ii) Traditional parameter estimation algorithms such as least mean square (LMS) do not assume any structural information about the system. Motivated by the ideas from CS we introduce a number of modified LMS algorithms for the sparse channel estimation problem. Decimated LMS algorithms for the special case of frequency sparse channels are also given. (iii) At last we consider the problem of CS of two dimensional signals. The most straightforward approach is to first find the vector form of a two dimensional signal and then use traditional CS methods to collect the compressed samples. However, our approach samples all the columns of a two dimensional signal with the same measurement matrix. This leads to simplification of the sampling process and also enables us to perform parallel signal recovery.