publication . Preprint . 2020

TREC CAsT 2019: The Conversational Assistance Track Overview

Dalton, Jeffrey; Xiong, Chenyan; Callan, Jamie;
Open Access English
  • Published: 30 Mar 2020
Abstract
The Conversational Assistance Track (CAsT) is a new track for TREC 2019 to facilitate Conversational Information Seeking (CIS) research and to create a large-scale reusable test collection for conversational search systems. The document corpus is 38,426,252 passages from the TREC Complex Answer Retrieval (CAR) and Microsoft MAchine Reading COmprehension (MARCO) datasets. Eighty information seeking dialogues (30 train, 50 test) are an average of 9 to 10 questions long. Relevance assessments are provided for 30 training topics and 20 test topics. This year 21 groups submitted a total of 65 runs using varying methods for conversational query understanding and ranki...
Subjects
free text keywords: Computer Science - Information Retrieval, Computer Science - Computation and Language, Computer Science - Machine Learning
Download from

[1] Carterette, B., Kanoulas, E., Hall, M., Clough, P.: Overview of the trec 2014 session track. In: The Twenty-Third Text REtrieval Conference Proceedings (TREC 2014). NIST Special Publication 500-308 (2015)

[2] Culpepper, J.S., Diaz, F., Smucker, M.: Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in Lorne (SWIRL 2018). SIGIR Forum 52(1), 34-90 (2018)

[3] Dietz, L., Gamari, B., Dalton, J., Craswell, N.: Trec complex answer retrieval overview. In: The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018). NIST Special Publication 500-331 (2019)

[4] J.Belkin, N., Cool, C., Stein, A., Thiel, U.: Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems. Expert Systems with Applications 9(3), 379-395 (1995)

[5] Krovetz, R.: Viewing morphology as an inference process. In: Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 191-202. ACM (1993)

[6] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K.N., Jones, L., Chang, M.W., Dai, A., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics (2019), https://tomkwiat.users.x20web.corp.google. com/papers/natural-questions/main-1455-kwiatkowski.pdf

[7] Radlinski, F., Craswell, N.: A theoretical framework for conversational search. In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval. pp. 117-126. ACM (2017) [OpenAIRE]

[8] Solomon, P.: Conversation in information-seeking contexts: A test of an analytical framework. Library & Information Science Research 19(3), 217-248 (1997)

[9] Zhang, H., Song, X., Xiong, C., Rosset, C., Bennett, P.N., Craswell, N., Tiwary, S.: Generic intent representation in web search. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 65-74. ACM (2019)

Abstract
The Conversational Assistance Track (CAsT) is a new track for TREC 2019 to facilitate Conversational Information Seeking (CIS) research and to create a large-scale reusable test collection for conversational search systems. The document corpus is 38,426,252 passages from the TREC Complex Answer Retrieval (CAR) and Microsoft MAchine Reading COmprehension (MARCO) datasets. Eighty information seeking dialogues (30 train, 50 test) are an average of 9 to 10 questions long. Relevance assessments are provided for 30 training topics and 20 test topics. This year 21 groups submitted a total of 65 runs using varying methods for conversational query understanding and ranki...
Subjects
free text keywords: Computer Science - Information Retrieval, Computer Science - Computation and Language, Computer Science - Machine Learning
Download from

[1] Carterette, B., Kanoulas, E., Hall, M., Clough, P.: Overview of the trec 2014 session track. In: The Twenty-Third Text REtrieval Conference Proceedings (TREC 2014). NIST Special Publication 500-308 (2015)

[2] Culpepper, J.S., Diaz, F., Smucker, M.: Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in Lorne (SWIRL 2018). SIGIR Forum 52(1), 34-90 (2018)

[3] Dietz, L., Gamari, B., Dalton, J., Craswell, N.: Trec complex answer retrieval overview. In: The Twenty-Seventh Text REtrieval Conference Proceedings (TREC 2018). NIST Special Publication 500-331 (2019)

[4] J.Belkin, N., Cool, C., Stein, A., Thiel, U.: Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems. Expert Systems with Applications 9(3), 379-395 (1995)

[5] Krovetz, R.: Viewing morphology as an inference process. In: Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 191-202. ACM (1993)

[6] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K.N., Jones, L., Chang, M.W., Dai, A., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics (2019), https://tomkwiat.users.x20web.corp.google. com/papers/natural-questions/main-1455-kwiatkowski.pdf

[7] Radlinski, F., Craswell, N.: A theoretical framework for conversational search. In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval. pp. 117-126. ACM (2017) [OpenAIRE]

[8] Solomon, P.: Conversation in information-seeking contexts: A test of an analytical framework. Library & Information Science Research 19(3), 217-248 (1997)

[9] Zhang, H., Song, X., Xiong, C., Rosset, C., Bennett, P.N., Craswell, N., Tiwary, S.: Generic intent representation in web search. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 65-74. ACM (2019)

Powered by OpenAIRE Research Graph
Any information missing or wrong?Report an Issue