Optomechanical accelerometers promise quantum-limited readout, high detection bandwidth, self-calibration, and radiation pressure stabilization. We present a simple, scalable platform that enables these benefits with nano-$g$ sensitivity at acoustic frequencies, based on a pair of vertically integrated Si$_3$N$_4$ membranes with different stiffnesses, forming an optical cavity. As a demonstration, we integrate an ultrahigh-Q ($>10^7$), millimeter-scale Si$_3$N$_4$ trampoline membrane above an unpatterned membrane on the same Si chip, forming a finesse $\mathcal{F}\approx2$ cavity. Using direct photodetection in transmission, we resolve the relative displacement of the membranes with a shot-noise-limited imprecision of 7 fm/$\sqrt{\text{Hz}}$, yielding a thermal-noise-limited acceleration sensitivity of 562 n$g/\sqrt{\text{Hz}}$ over a 1 kHz bandwidth centered on the fundamental trampoline resonance (40 kHz). To illustrate the advantage of radiation pressure stabilization, we cold damp the trampoline to an effective temperature of 4 mK and leverage the reduced energy variance to resolve an applied stochastic acceleration of 50 n$g/\sqrt{\text{Hz}}$ in an integration time of minutes. In the future, we envision a small-scale array of these devices operating in a cryostat to search for fundamental weak forces such as dark matter.