Abstract The shear buckling behavior of the web-post in a castellated steel beam with fillet corner hexagonal web openings is studied using the finite element method. The buckling modes and buckling capacity of the web-post in the castellated beam with fillet corner hexagonal web openings are compared with those having circular and elongated circular openings. The web-post in a castellated steel beam with the proposed opening shape can achieve as good structural performance as that with circular openings. The effects of the opening distance, the opening height, and the web thickness on the buckling behavior of the web-post are investigated. The Strut Model is modified for predicating the shear buckling capacity of the web-post in the castellated steel beam with fillet corner openings. The equivalent rectangular opening is also modified to calculate the Vierendeel failure capacity of the castellated steel beam with the fillet corner hexagonal openings at the perforated sections. The load bearing capacity of the castellated steel beam with the proposed opening shape is the minimum value among the web-post buckling capacity, the Vierendeel failure capacity at the perforated section, and the shear rupture capacity of the horizontal weld.