In Time and Wavelength Division Multiplexed (TWDM) Passive Optical Networks (PONs), dynamically (de)activating Optical Subscriber Units (OSUs) and assigning Optical Network Units (ONUs) to the active OSUs helps improving the OLT energy efficiency and balancing the network load.However, energy efficiency improvement is traded with a delay increase due to the finite ONU tuning speed and the potential increase of the normalized load with respect to the number of active OSUs. Moreover, even though load balance potentially assures delay balance, the scheduling and assignment of ONUs to OSUs might cause ONU delay unbalance (i.e., delay unfairness). Indeed, if an ONU is forced to tune at each reconfiguration it suffers a high delay penalty due to its limited tuning time.In this study, the impact of the ONU scheduling and assignment to OSUs upon network reconfiguration on the average frame delay unbalance is evaluated. Then, a method that balances the average frame delay by swapping the tuning ONU upon reconfiguration is proposed, for some critical scenarios. Simulation and experimental results show that the delay unbalance is large when the number of active ONUs is small (e.g., at network start up or when the majority of the ONUs is inactive/sleeping). In this case, the proposed method effectively balances the delay experienced by ONUs without impacting the OLT energy savings.