You have already added 0 works in your ORCID record related to the merged Research product.
You have already added 0 works in your ORCID record related to the merged Research product.
Robust model-based deep reinforcement learning for flow control
Robust model-based deep reinforcement learning for flow control
Active flow control has the potential of achieving remarkable drag reductions in applications for fluid mechanics, when combined with deep reinforcement learning (DRL). The high computational demands for CFD simulations currently limits the applicability of DRL to rather simple cases, such as the flow past a cylinder, as a consequence of the large amount of simulations which have to be carried out throughout the training. One possible approach of reducing the computational requirements is to substitute the simulations partially with models, e.g. deep neural networks; however, model uncertainties and error propagation may lead an unstable training and deteriorated performance compared to the model-free counterpart. The present thesis aims to modify the model-free training routine for controlling the flow past a cylinder towards a model-based one. Therefore, the policy training alternates between the CFD environment and environment models, which are trained successively over the course of the policy optimization. In order to reduce uncertainties and consequently improve the prediction accuracy, the CFD environment is represented by two model ensembles responsible for predicting the states and lift force as well as the aerodynamic drag, respectively. It could have been shown that this approach is able to yield a comparable performance to the model-free training routine at a Reynolds number of Re = 100 while reducing the overall runtime by up to 68.91%. The model-based training, however, showed a high dependency of the performance and stability on the initialization, which needs to be investigated further.
model-based deep reinforcement learning, computational fluid dynamics (CFD), active flow control, deep reinforcement learning (DRL)
model-based deep reinforcement learning, computational fluid dynamics (CFD), active flow control, deep reinforcement learning (DRL)
3914
5.2 Generalization of the training routine . . . . . . . . . . . . . . . . . . . . . . . . 46 5.2.1 Early stopping of the model-training . . . . . . . . . . . . . . . . . . . . . 46 5.2.2 Comparison of the prediction accuracy . . . . . . . . . . . . . . . . . . . . 47 5.2.3 Influence of the number of models within the ensemble . . . . . . . . . . . 49 5.2.4 Alternation between model-based and model-free episodes . . . . . . . . . 51 5.2.5 Improvements of the training routine and migrating to AWS . . . . . . . 54 5.3 Comparison of the final results for Re = 100 . . . . . . . . . . . . . . . . . . . . . 59 5.4 Model-based training at higher Reynolds numbers . . . . . . . . . . . . . . . . . 61 5.5 Comparison of the final results for Re = 500 . . . . . . . . . . . . . . . . . . . . . 64
[1] Abbeel, P., Quigley, M., and Ng, A. Y.: “Using inaccurate models in reinforcement learning”. In: Proceedings of the 23rd international conference on Machine learning. Ed. by Cohen, W. ACM Other conferences. New York, NY: ACM, 2006, pp. 1-8. doi: 10.1145/ 1143844.1143845.
[2] Beck, N., Landa, T., Seitz, A., et al.: “Drag Reduction by Laminar Flow Control”. In: energies 11(1) (2018), pp. 1-28. doi: 10.3390/en11010252. [OpenAIRE]
[3] Belousov, B., Abdulsamad, H., Klink, P., et al., eds.: Reinforcement learning algorithms: Analysis and applications. Vol. 883. Studies in computational intelligence. Cham, Switzerland: Springer Nature, 2021. doi: 10.1007/978-3-030-41188-6.
[4] Chua, K., Calandra, R., McAllister, R., et al.: “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models”. In: arXiv (2018).
[5] Clavera, I., Rothfuss, J., Schulman, J., et al.: “Model-Based Reinforcement Learning via Meta-Policy Optimization”. In: arXiv (2018).
[6] Darshan Thummar: “Active flow control in simulations of fluid flows based on deep reinforcement learning”. MA thesis. Braunschweig: TU Braunschweig, 2021. doi: 10.5281/ ZENODO.4897961. [OpenAIRE]
[7] Dong, H., Ding, Z., and Zhang, S., eds.: Deep Reinforcement Learning: Fundamentals, Research and Applications. Singapore: Springer Singapore Pte. Limited, 2020.
[8] Elhawary, M. A.: “Deep Reinforcement Learning for Active Flow Control around a Circular Cylinder Using Unsteady-mode Plasma Actuators”. In: arXiv (2020). [OpenAIRE]
[9] Fabian Gabriel: “Aktive Regelung einer Zylinderumströmung bei variierender Reynoldszahl durch bestärkendes Lernen”. MA thesis. Braunschweig: TU Braunschweig, 2021. doi: 10.5281/ZENODO.5634050.
citations This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).0 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average citations This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).0 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average
Citations provided by BIP!Popularity provided by BIP!Active flow control has the potential of achieving remarkable drag reductions in applications for fluid mechanics, when combined with deep reinforcement learning (DRL). The high computational demands for CFD simulations currently limits the applicability of DRL to rather simple cases, such as the flow past a cylinder, as a consequence of the large amount of simulations which have to be carried out throughout the training. One possible approach of reducing the computational requirements is to substitute the simulations partially with models, e.g. deep neural networks; however, model uncertainties and error propagation may lead an unstable training and deteriorated performance compared to the model-free counterpart. The present thesis aims to modify the model-free training routine for controlling the flow past a cylinder towards a model-based one. Therefore, the policy training alternates between the CFD environment and environment models, which are trained successively over the course of the policy optimization. In order to reduce uncertainties and consequently improve the prediction accuracy, the CFD environment is represented by two model ensembles responsible for predicting the states and lift force as well as the aerodynamic drag, respectively. It could have been shown that this approach is able to yield a comparable performance to the model-free training routine at a Reynolds number of Re = 100 while reducing the overall runtime by up to 68.91%. The model-based training, however, showed a high dependency of the performance and stability on the initialization, which needs to be investigated further.