Abstract Background Oxidative stress (OS) and inflammatory reactions play pivotal roles in secondary brain injury after traumatic brain injury (TBI). Histone deacetylase 3 (HDAC3) controls the acetylation of histones and non-histones, which has a significant impact on the central nervous system’s reaction to damage. This research determined the implications of RGFP966, a new and specific inhibitor of HDAC3, for the antioxidant (AO) systems mediated by nuclear factor erythroid2-related factor 2 (Nrf2) and the Nod-like receptor protein 3 (NLRP3) inflammasome in TBI. The study also studied the underlying mechanisms of RGFP966’s actions. Our objective was to examine the impacts and underlying RGFP966 mechanisms in TBI. Methods In vitro, a rat cortical neuron OS model was induced by H2O2, followed by the addition of RGFP966 to the culture medium. Neurons were collected after 24 h for western blot (WB), terminal deoxynucleotidyl transferase dUTP nick end labeling (TUNEL) and 2′-7′-dichlorodihydrofluorescein diacetate staining. In vivo, RGFP966 (10 mg/kg) was administered post-TBI. Brain tissue water content and modified neurological severity scores were assessed 72 h post-injury. Cortical tissues surrounding the focal injury were subjected to western blot, TUNEL staining, Nissl staining and immunofluorescence/immunohistochemistry staining, and malondialdehyde level, hindered glutathione content and superoxide dismutase activity were measured. Serum was collected for the enzyme-linked immunosorbent assay. Nrf2-specific shRNA lentivirus was injected into the lateral ventricle of rats for 7 days, and cerebral cortex tissue was analyzed by WB and real-time polymerase chain reaction. Results During in vitro and in vivo experiments, RGFP966 suppressed HDAC3 expression, promoted Nrf2 nuclear translocation, activated downstream AO enzymes, mitigated excessive reactive oxygen species production and alleviated nerve cell apoptosis. RGFP966 effectively reduced brain edema and histological damage and enhanced neurological and cognitive function in rats with TBI. RGFP966 markedly inhibited NLRP3 inflammasome activation mediated by high-mobility group box 1 (HMGB1)/toll-like receptor 4 (TLR4). Nrf2 knockdown in TBI rats attenuated the AO and anti-inflammatory, neuroprotective impacts of RGFP966. Conclusions Overall, our findings demonstrate that RGFP966 can mitigate the first brain damage and neurological impairments in TBI. The underlying mechanism involves triggering the Nrf2-mediated AO system and negatively regulating the HMGB1/TLR4-mediated NLRP3 inflammasome pathway.