There is currently a worldwide effort to develop UAS Traffic Management (UTM) systems that help ensure safe and reliable operation of Unmanned Autonomous Systems (UAS) in urban environments. A large number of factors must be considered in planning such flights, including GIS (roads, topography, etc.), weather (temperature, wind, precipitation), localization and navigation (GPS, V2X communication), infrastructure obstacles (buildings, towers, etc.), excluded zones of operation, etc. We have developed a cloud-based geospatial intelligence system, BRECCIA, which brokers such information among a set of intelligent agents. In this work, an extended version of BRECCIA is proposed as a universal-UTM (U-UTM) which allows the specification of urban airways constrained to be above roadways. In addition, we develop a reinforcement learning approach for the determination of optimal flight policies through such airways, where these policies can take into account a variety of factors (wind, precipitation, communication, etc.) which impact UAV path following capabilities. A novel context-based probabilistic state transition function is introduced. Simulation experiments are performed to demonstrate the performance of the approach.