The Deep Fourier Residual (DFR) method is a specific type of variational physics-informed neural networks (VPINNs). It provides a robust neural network-based solution to partial differential equations (PDEs). The DFR strategy is based on approximating the dual norm of the weak residual of a PDE. This is equivalent to minimizing the energy norm of the error. To compute the dual of the weak residual norm, the DFR method employs an orthonormal spectral basis of the test space, which is known for rectangles or cuboids for multiple function spaces. In this work, we extend the DFR method with ideas of traditional domain decomposition (DD). This enables two improvements: (a) to solve problems in more general polygonal domains, and (b) to develop an adaptive refinement technique in the test space using a Dofler marking algorithm. In the former case, we show that under non-restrictive assumptions we retain the desirable equivalence between the employed loss function and the H1-error, numerically demonstrating adherence to explicit bounds in the case of the L-shaped domain problem. In the latter, we show how refinement strategies lead to potentially significant improvements against a reference, classical DFR implementation with a test function space of significantly lower dimensionality, allowing us to better approximate singular solutions at a more reasonable computational cost.