Open Universiteit

Please use this identifier to cite or link to this item:
Title: Scoring Summaries Using Recurrent Neural Networks
Authors: Ruseti, Stefan
Dascalu, Mihai
Johnson, Amy M.
McNamara, Danielle S.
Balyan, Renu
McCarthy, Kathryn S.
Trausan-Matu, Stefan
Keywords: Automated summary evaluation
Recurrent neural network
Semantic models
Word embeddings
Issue Date: Jun-2018
Publisher: Springer
Citation: Ruseti, S., Dascalu, M., Johnson, A. M., McNamara, D. S., Balyan, R., McCarthy, K., & Trausan-Matu, S. (2018). Scoring Summaries using Recurrent Neural Networks. In R. Nkambou, R. Azevedo & J. Vassileva (Eds.), 14th Int. Conf. on Intelligent Tutoring Systems (ITS 2018) (pp. 191–201). Montreal, Canada: Springer.
Abstract: Summarization enhances comprehension and is considered an effective strategy to promote and enhance learning and deep understanding of texts. However, summarization is seldom implemented by teachers in classrooms because the manual evaluation requires a lot of effort and time. Although the need for automated support is stringent, there are only a few shallow systems available, most of which rely on basic word/n-gram overlaps. In this paper, we introduce a hybrid model that uses state-of-the-art recurrent neural networks and textual complexity indices to score summaries. Our best model achieves over 55% accuracy for a 3-way classification that measures the degree to which the main ideas from the original text are covered by the summary . Our experiments show that the writing style, represented by the textual complexity indices, together with the semantic content grasped within the summary are the best predictors, when combined. To the best of our knowledge, this is the first work of its kind that uses RNNs for scoring and evaluating summaries.
Appears in Collections:1. RAGE Publications

Files in This Item:
File Description SizeFormat 
ITS Stefan 2018.pdf279.41 kBAdobe PDFView/Open

This item is licensed under a Creative Commons License Creative Commons