Backdoor attacks compromise deep neural networks by injecting them with covert, malicious behaviors during training, which attackers can later activate at test-time. As backdoors become more sophisticated, defenses struggle to catch up. This paper introduces a simple yet effective Backdoor Attack using Inpainting as a Trigger, dubbed BAIT. The attack's trigger relies on a randomly-drawn polygonal patch, filled via inpainting with an off-the-shelf generative adversarial network. Using BAIT, we show that several defenses, including common test-time input purification methods, can be bypassed by patch-based backdoors. To counter this, we propose four targeted defense strategies.