Abstract Spiny lobsters are targeted in capture fisheries due the high market demand and this situation has led to overfishing activities. Therefore, a smart monitoring system is imperative to be developed in handling this issue in order to store the distribution of marine spiny lobsters’ information such as habitat preferences, population density and biology aspects of the lobsters. This study focused on separating the mixed underwater acoustic sound. An output of the separation system was used for Passive Acoustic Monitoring (PAM) application. The objective of this study is to provide estimated source signals from a recorded mixed acoustic signal. The feasibility of extracting spiny lobster sound as the target sound was observed during the experiment. In this study, Blind Source Separation (BSS) approach was employed to estimate the target sound (i.e., spiny lobsters) from the mixed underwater acoustic signal consisting of the sound of spiny lobsters and several man-made and natural sounds. In this investigation, two different blind source separation methods namely Fast Fixed Point Independent Component Analysis (FastICA) and Non Negative Matrix Factorization (NMF) were implemented. The mixture of one target sound which was spiny lobster and four interferences signals were used as input in this experiment,. The separated source sound by using FastICA with Negentropy, FastICA with Kurtosis and NMF algorithms were compared and evaluated based on bss_eval_sources toolbox metrics. These metrics consisted of signal to distortion ratio (SDR), signal to interferences ratio (SIR) and signal to artifacts ratio (SAR). In conclusion, the FastICA with Negentropy technique provides the best performance in separating mixed signal based on SIR, SAR and SDR measurement results that showed the FastICA with Negentropy generated the highest average values compared to the FastICA with Kurtosis and NMF.