We introduce a large-scale dataset of human actions and eye movements while playing Atari videos games. Human subjects play games in a frame-by-frame manner to allow enough decision time in order to obtain near-optimal decisions. For every game frame, its corresponding image frame, the human keystroke action, the reaction time to make that action, the gaze positions, and immediate reward returned by the environment were recorded. The gaze data was recorded using an EyeLink 1000 eye tracker at 1000Hz. The human subjects are amateur players who are familiar with the games. The human subjects were only allowed to play for 15 minutes and were required to rest for at least 15 minutes before the next trial. We collected data from 4 subjects, 16 games, 175 15-minute trials, and a total of 2.97 million frames/demonstrations. 1. meta_data.csv: meta data for the dataset. Data fields: Game: String. Game name. TrialNumber: Integer. One can use this number to locate the associated .tar.bz2 file and label file. SubjID: Char. Human subject identifiers. Load: Integer. 0 indicates that the game starts from scratch. If this field is non-zero, it means that the current trial continues from a saved trial. The number indicates the trial number to look for. FrameAveraging: Boolean. The game engine allows one to turn this on or off. When turning on (TRUE), two consecutive frames are averaged, this alleviates screen flickering in some games. FPS: Integer. Frame per second when an action key is held down. NumberOfFrames: Number of image frames in the .tar.bz2 repository. AverageValError: Float. Eye-tracking validation error at the end of each trial in visual degree (1 visual degree = 1.44 cm in our experiment). BestScore: Integer. The highest game score obtained from this trial. 2. *.tar.bz2 files: contains game image frames. The filename indicates its trial number. 3. *.txt files: label file for each trial. Data fields include: frame_id: String. The ID of a frame, can be used to locate the corresponding image frame in .tar.bz2 file. episode_id: Integer (not available for some trials). Episode number, starting from 0 for each trial. A trial could contain a single trial or multiple trials. score: Integer (not available for some trials). Current game score for that frame. duration(ms): Integer. Time elapsed until the human player made a decision. unclipped_reward: Integer. Immediate reward returned by the game engine. action: Integer. See action_enums.txt for the mapping. This is consistent with the Arcade Learning Environment setup. gaze_positions: Null/A list of integers: x0,y0,x1,y1,...,xn,yn. Gaze positions for the current frame. Could be null if no gaze. (0,0) is the top-left corner. x: horizontal axis. y: vertical. If you use the Atari-HEAD in your research, we ask that you please cite the following: Zhang, Ruohan, Zhuode Liu, Luxin Zhang, Jake A. Whritner, Karl S. Muller, Mary M. Hayhoe, and Dana H. Ballard. "AGIL: Learning attention from human for visuomotor tasks." In Proceedings of the European Conference on Computer Vision (ECCV), pp. 663-679. 2018. @inproceedings{zhang2018agil, title={AGIL: Learning attention from human for visuomotor tasks}, author={Zhang, Ruohan and Liu, Zhuode and Zhang, Luxin and Whritner, Jake A and Muller, Karl S and Hayhoe, Mary M and Ballard, Dana H}, booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, pages={663--679}, year={2018} }