Project website: minedojo.org Paper: arxiv.org/abs/2206.08853 GitHub: github.com/MineDojo/MineDojo Minecraft is among the most streamed games on YouTube. Human players have demonstrated a stunning range of creative activities and sophisticated missions that take hours to complete. We collect 730K+ narrated Minecraft videos, which add up to 33 years of duration and 2.2B words in English transcripts. The time-aligned transcripts enable the agent to ground free-form natural language in video pixels and learn the semantics of diverse activities without laborious human labeling. There are two files in our YouTube knowledge base. youtube_tutorial.json (tutorial videos): Minecraft tutorial videos include step-by-step demonstrations and sometimes detailed verbal explanations. They also serve as a rich source of creative missions that humans find interesting. We harvest thousands of tasks from these videos in our benchmarking suite. youtube_full.json (general gameplay videos): Unlike tutorials, general gameplay videos do not necessarily provide guidance on particular tasks. Instead, they capture the “in-the-wild” human experiences that are much larger in quantity, diverse in contents, and rich in learning signals. Data Structure list[ { "id": str, # video id "title": str, # video title "link": str, # video link "view_count": int # number of times the video has been viewed "like_count": int # number of users who have indicated that they liked the video "duration": float # video duration in seconds "fps": float, # video FPS } ] Check out our paper! @article{fan2022minedojo, title = {MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge}, author = {Linxi Fan and Guanzhi Wang and Yunfan Jiang and Ajay Mandlekar and Yuncong Yang and Haoyi Zhu and Andrew Tang and De-An Huang and Yuke Zhu and Anima Anandkumar}, year = {2022}, journal = {arXiv preprint arXiv: Arxiv-2206.08853} }