Extended abstract accepted at the ICRA (International Conference on Robotics and Automation) 2025 Workshop Robot Software Architectures (RSA). Reliable autonomy in robotic systems requires deliberation mechanisms capable of handling dynamic environments robustly. However, despite their potential for improving system robustness, formal verification techniques like model checking remain uncommon in the robotics domain due to usability and accessibility challenges. This work synthesizes insights from two studies on robotic deliberation system development, identifying key challenges in designing and debugging deliberation mechanisms. Based on these insights, solutions to improve development workflows and ensure system robustness are discussed. One proposal is the introduction of High-Level SCXML (HL-SCXML) to model system components including both Robot Operating System 2 (ROS 2) and Behavior Tree (BT) features. AS2FM (Autonomous Systems to Formal Models) is a tool that allows statistical model checking for ROS 2-based systems by translating a system model given in HL-SCXML format into JANI, enabling the verification of system properties at design-time. To enhance usability, we propose integrating AS2FM with model-driven engineering (MDE) tools such as Papyrus for Robotics, automating the generation of HL-SCXML models, reducing manual effort. Additionally, we present an approach leveraging reinforcement learning using scene graphs, enabling autonomous robots to navigate in unknown environments more robustly. Combining the formal verification and learning approaches aims to practically ensure reliable and adaptable autonomy of robotic systems.