There is increasing evidence that innovation offers perspectives for poverty alleviation in small producers’ contexts in developing countries. However, innovations sometimes imply additional harmful environmental and social consequences, which are not in line with broader poverty alleviation and sustainable development notions. In this paper, we explore the question how small producers in northern Vietnam, as innovators in a cluster context, innovate and take broader societal considerations into account in the innovation process. We found that traditional institutional theories show a number of shortcomings for our analysis into the multifaceted meta-textual societal process towards responsible innovation. Instead, we applied Actor‐Network Theory (ANT) which describes the shaping of the human interaction in detail providing a better understanding why in some networks innovatorsbehave opportunistically while in others they acknowledge responsibility for harmful innovation outcomes. The ANT lens reveals the role of informal institutions in innovations systems and the role of materiality in network development. Lastly, ANT demonstrates how the creation and falling apart of actor networks is essential in the dynamics of a five‐stage model of the responsible innovation societal process. We conclude with factors and conditions steering the societal process and emerging questions concerning power, allowing human and non-human actants into the network, and the issue whether a network in the responsible innovation zone is also desirable in terms of economic competitiveness.