Driven by global climate goals, an increasing amount of Renewable Energy Sources (RES) is currently being installed worldwide. Especially in the context of offshore wind integration, hybrid AC/DC grids are considered to be the most effective technology to transmit this RES power over long distances. As hybrid AC/DC systems develop, they are expected to become increasingly complex and meshed as the current AC system. Nevertheless, there is still limited literature on how to optimize hybrid AC/DC topologies while minimizing the total power generation cost. For this reason, this paper proposes a methodology to optimize the steady-state switching states of transmission lines and busbar configurations in hybrid AC/DC grids. The proposed optimization model includes optimal transmission switching (OTS) and busbar splitting (BS), which can be applied to both AC and DC parts of hybrid AC/DC grids. To solve the problem, a scalable and exact nonlinear, non-convex model using a big M approach is formulated. In addition, convex relaxations and linear approximations of the model are tested, and their accuracy, feasibility, and optimality are analyzed. The numerical experiments show that a solution to the combined OTS/BS problem can be found in acceptable computation time and that the investigated relaxations and linearisations provide AC feasible results.