Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Article . 2024
License: CC BY
Data sources: ZENODO
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Article . 2024
License: CC BY
Data sources: ZENODO
ZENODO
Article . 2024
License: CC BY
Data sources: Datacite
ZENODO
Article . 2024
License: CC BY
Data sources: Datacite
ZENODO
Article . 2024
License: CC BY
Data sources: Datacite
ZENODO
Article . 2024
License: CC BY
Data sources: Datacite
versions View all 4 versions
addClaim

Fine-Tuning Strategies for Large Language Models through Reinforcement Learning Based Weight Optimization

Authors: Sai Sukesh Reddy Tummuri;

Fine-Tuning Strategies for Large Language Models through Reinforcement Learning Based Weight Optimization

Abstract

Large language model (LLM) fine-tuning based on reinforcement learning has emerged as a crucial strategy for improving response quality, coherence, and safety as well as matching model outputs with human preferences. In order to enhance LLM performance across several objectives at once, this study suggests a novel framework for weight optimization using reinforcement learning. Experiments were carried out in a simulated human-preference environment that closely resembles the statistical features of actual RLHF datasets in order to assess the method\\\'s reproducibility and reliability without the need for external datasets. Key performance metrics Accuracy, Precision, Recall, and F1-Score were used to evaluate the suggested method. These metrics varied realistically between 94% and 97%, indicating the optimization strategy\\\'s robustness. Several visualizations, such as reward improvement over training steps, policy loss reduction over 18 epochs, multi-objective reward contributions, and comparisons with traditional fine-tuning strategies, were used to further analyze training dynamics. The findings show that the suggested strategy maintains stable training and balanced optimization across various objectives in addition to achieving high performance metrics. A comparative analysis demonstrates that the AMORL-WO approach performs better at matching model outputs with human preferences than conventional supervised fine-tuning (SFT), RLHF, and PPO-based techniques. Overall, this study shows that weight optimization based on reinforcement learning is a useful, effective, and multi-objective method for LLM fine-tuning that can result in responses that are safer, more coherent, and more in line with preferences. These results demonstrate the potential of reinforcement learning in large-scale model optimization and offer a promising basis for future development of human-aligned AI systems

  • BIP!
    Impact byBIP!
    citations
    The total number of citations received by all articles of the researcher of interest.
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
citations
The total number of citations received by all articles of the researcher of interest.
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green