Background: Rheumatic heart disease affects 33 million people in low and middle income countries and is the leading cause of cardiovascular death among children and young adults. Penicillin prophylaxis prevents progression in asymptomatic disease. Efforts to expand echocardiographic screening are focusing on simplified protocols, non-physician ultrasonographers, and portable ultrasound devices, including handheld ultrasound. Recent advances support the use of single-view screening protocols. With the increasing availability and low cost of handheld devices, studies are needed to evaluate their performance in these settings. Methods: We conducted a retrospective study comparing the rate of screen positive ultrasounds before and after the use of a handheld ultrasound in an RHD screening program in Ethiopia. We also performed a cross-sectional device comparison in 19 at-risk school-children participating in the rheumatic heart disease screening program. Results: Between March of 2019 and Jan of 2022, 6631 children were screened for rheumatic heart disease of whom 4029 were screened after the introduction of a handheld device. Before the use of the handheld ultrasound device 291 (11.2%) children had a screen positive ultrasounds compared with 167 (4.1%) afterwards (p < 0.001). We also compared non-expert to expert interpretation by device and found a significant difference in interpretation for the Lumify (p=0.025). There was a trend towards shorter jet length by color Doppler in the handheld ultrasound device for both expert and non-expert review. Conclusions: Our study highlights that the screen-positive rate in a RHD screening program is influenced by the device being used in the screening process.