Saku Sugawara

菅原 朔 (ja)
Assistant Professor at National Institute of Informatics, Tokyo, Japan
Ph.D. in Computer Science (University of Tokyo, 2020)

Email: saku(at)nii.ac.jp

GitHub / Google Scholar

研究テーマのチュートリアル的スライド（2022年版、日本語）
RA を募集しています

News

• One paper (on evaluating rationale understanding in logical reading comprehension) has been accepted to EMNLP 2023 (main) and one paper (on evaluating the projectivity of presuppositions) to CoNLL 2023. (Oct 2023)
• Two papers (on counter-commonsense physical reasoning and an opinion piece about defining and testing NLU) have been accepted to ACL 2023 (main and Findings). (May 2023)
• One paper (on analysis of underlying reasoning tasks in multi-hop QA) has been accepted to EACL 2023 (Findings). (Jan 2023)
• One paper (on analysis of shortcut preference in QA) has been accepted to AAAI 2023 and one paper (on negative results for improving perturbation methods in QA) to the KnowledgeNLP workshop at AAAI 2023. (Nov 2022)
• One paper (on analysis of relative position bias in extractive QA) has been accepted to the BlackboxNLP 2022 workshop. (Oct 2022)
• I have a seminar talk (in Japanese) on the evaluation and explanation of NLU systems. [link] [slide] (Oct 2022)
• Two papers (on debiasing method for NLU and cross-modal curriculum learning) have been accepted to EMNLP 2022. (Oct 2022)
• We present the Possible Stories dataset at COLING 2022. [poster] [slide] (Oct 2022)
• One paper (on a new dataset for reasoning over date information) has been accepted to AACL-IJCNLP 2022. (Sep 2022)
• I started writing news that keep updated only if I'm not lazy.

Research Area

• Computational linguistics, natural language processing
• Natural language understanding, machine reading comprehension
• Task design, dataset construction

Publications

2023

• Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
  Akira Kawabata, Saku Sugawara
  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)EMNLP 2023
  [paper] [arxiv] to appear.
• PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments
  Daiki Asami, Saku Sugawara
  In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL 2023)CoNLL 2023
  [paper] [arxiv] to appear.
• Probing Physical Reasoning with Counter-Commonsense Context
  Kazushi Kondo, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023 short paper), Jul 2023.ACL 2023 short paper
  [paper] [arxiv]
• On Degrees of Freedom in Defining and Testing Natural Language Understanding
  Saku Sugawara, Shun Tsugita
  In Findings of the Association for Computational Linguistics: ACL 2023, Jul 2023.Findings of ACL: ACL 2023
  [paper] [arxiv]
• Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
  Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa
  In Findings of the Association for Computational Linguistics: EACL 2023, May 2023.Findings of ACL: EACL 2023
  [paper] [arxiv]
• Which Shortcut Solution Do Question Answering Models Prefer to Learn?
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23), Feb 2023.AAAI 2023
  [paper] [arxiv] [slide] [poster]
• Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Workshop on Knowledge Augmented Methods for NLP (KnowledgeNLP) at AAAI 2023, Feb 2023.KnowledgeNLP Workshop at AAAI 2023
  [paper] [arxiv] [poster]

2022

• Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP 2022) at EMNLP 2022, pp.418-425, Dec 2022.BlackboxNLP Workshop at EMNLP 2022
  [paper] [arxiv] [poster]
• Debiasing Masks: A New Framework for Shortcut Mitigation in NLU
  Johannes Mario Meissner, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022 short paper), pp.7607-7613, Dec 2022.EMNLP 2022 short paper
  [paper] [arxiv]
• Cross-Modal Similarity-Based Curriculum Learning for Image Captioning
  Hongkuan Zhang, Saku Sugawara, Akiko Aizawa, Lei Zhou, Ryohei Sasano, Koichi Takeda
  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022 short paper), pp.7599-7606, Dec 2022.EMNLP 2022 short paper
  [paper] [arxiv]
• How Well Do Multi-hop Reading Comprehension Models Understand Date Information?
  Xanh Ho, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP 2022 short paper), pp.470-479, Nov 2022.AACL-IJCNLP 2022 short paper
  [paper] [arxiv] [bib] [poster]
• Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios
  Mana Ashida, Saku Sugawara
  In Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022), pp.3606-3630, Oct 2022.COLING 2022
  [paper] [arxiv] [bib] [data] [slide] [poster]
• A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension
  Xanh Ho*, Johannes Mario Meissner*, Saku Sugawara, Akiko Aizawa
  Preprint.
  [arxiv] *: equal contribution
• What Makes Reading Comprehension Questions Difficult?
  Saku Sugawara, Nikita Nangia, Alex Warstadt, Samuel R. Bowman
  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), pp.6951-6971, May 2022.ACL 2022
  [paper] [arxiv] [bib] [data] [slide] [poster]

2021

• Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 3rd Workshop on Machine Reading for Question Answering (MRQA) at EMNLP 2021, pp.63-72, Nov 2021.MRQA Workshop at EMNLP 2021
  [paper] [arxiv] [bib]
• Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (ACL-IJCNLP 2021 SRW), pp.197-214, Aug 2021.ACL-IJCNLP 2021 Student Research Workshop
  [paper] [arxiv] [bib]
• What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
  Nikita Nangia*, Saku Sugawara*, Harsh Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), pp.1221-1235, Aug 2021.ACL-IJCNLP 2021
  [paper] [arxiv] [bib] [data/code] *: equal contribution
• Embracing Ambiguity: Shifting the Training Target of NLI Models
  Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021 short paper), pp.862-869, Aug 2021.ACL-IJCNLP 2021 short paper
  [paper] [arxiv] [bib]
• Benchmarking Machine Reading Comprehension: A Psychological Perspective
  Saku Sugawara, Pontus Stenetorp, Akiko Aizawa
  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), pp.1592-1612, Apr 2021.EACL 2021
  [paper] [arxiv] [bib] [slide] [poster]

2020

• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
  Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020), pp.6609-6625, Dec 2020.COLING 2020
  [paper] [arxiv] [bib]
• Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
  Saku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko Aizawa
  In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-20), pp.8918-8927, Feb 2020.AAAI 2020
  [paper] [arxiv] [bib] [poster] [spotlight] [data]

2018

• What Makes Reading Comprehension Questions Easier?
  Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa
  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp.4208-4219, Nov 2018.EMNLP 2018
  [paper] [arxiv] [bib] [slide] [data]

2017

• Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability
  Saku Sugawara, Yusuke Kido, Hikaru Yokono, Akiko Aizawa
  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), pp.806-817, Aug 2017.ACL 2017
  [paper] [bib] [slide] [data]
  
• Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems
  Saku Sugawara, Hikaru Yokono, Akiko Aizawa
  In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI-17), pp.3089-3096, Feb 2017.AAAI 2017
  [paper] [bib] [slide] [data]

2016

• Annotation and Analysis of Discourse Relations, Temporal Relations and Multi-Layered Situational Relations in Japanese Texts
  Kimi Kaneko, Saku Sugawara, Koji Mineshima, Daisuke Bekki
  In Proceedings of COLING 2016 Workshop on Asian Language Resources 12 (ALR12), pp.10-19, Dec 2016.
  [paper] [bib] [web]
• An Analysis of Prerequisite Skills for Reading Comprehension
  Saku Sugawara, Akiko Aizawa
  In Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing (UBLP), pp.1-5, Nov 2016.
  [paper] [bib] [poster] [web] [data]
• On Dialogue Breakdown: Annotation and Detection - dialogue breakdown detection challenge
  Kotaro Funakoshi, Ryuichiro Higashinaka, Michimasa Inaba, Yuka Kobayashi, Saku Sugawara, Katsuya Takanashi, Hiroko Otsuka, Hanae Koiso, Mayumi Bono
  In IVA 2016 Workshop on Chatbots and Conversational Agent Technologies (WOCHAT), Sep 2016.
  [paper] [web]

Collaborators

• Research Assistants
• Yoko Kayano (Jul 2023-)
• Miyu Oba (Nara Institute of Science and Technology, Apr 2023-)
• Akari Haga (Nara Institute of Science and Technology, Dec 2022-)
• Rei Emura (Tohoku University, May 2022-)
• Junpei Suzuki (University of Tokyo, Apr 2021-)
• Visiting Student
• Daiki Asami (University of Delaware, Apr 2022-; formerly Research Assitant Oct 2021 -- Mar 2022)
• Former Collaborators
• Viktor Schlegel (Visiting Researcher from University of Manchester, Jul 2022 -- Sep 2022)
• Akira Kawabata (Research Assistant from Nara Institute of Science and Technology, Oct 2021 -- Jun 2022)
• Mana Ashida (Research Assistant from Tokyo Metropolitan University, Jun 2021 -- Mar 2022)
• Chihiro Taguchi (Research Assistant from Nara Institute of Science and Technology, Apr 2021 -- Sep 2021)

Education

• Apr 2017 -- Mar 2020, Ph.D. in Computer Science
  • Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
  • Ph.D. thesis: Evaluating Natural Language Understanding in Machine Reading Comprehension
    [paper] [slide]
• Apr 2015 -- Mar 2017, M.Sc. in Computer Science
  • Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
  • Master thesis: Prerequisite Skills of Natural Language Understanding for the Analysis and Development of Reading Comprehension
• Apr 2013 -- Mar 2015, B.A. in Letters (Philosophy)
  • Philosophy Course, Faculty of Letters, University of Tokyo
  • Graduation thesis: Language Understanding Based on the Concept of Proper Function (On the biological/teleo- semantics by Ruth G. Millikan)
• Apr 2011 -- Mar 2013
  • Natural Sciences I, College of Arts and Sciences, University of Tokyo

Experience

• Apr 2020 -- Present, Assistant Professor (five years), National Institute of Informatics, Japan
• Apr 2020 -- Present, Visiting Researcher, RIKEN AIP, Japan
• Jun 2020 -- Dec 2021, Visiting Researcher, New York University, US
• May 2017 -- Mar 2020, Research Assistant, RIKEN AIP, Japan
• Nov 2018 -- Mar 2019, Visiting Student, University College London, UK
• Jun 2018 -- Sep 2018, Research Intern, Microsoft Research Montreal (Maluuba), Canada
• Apr 2016 -- Mar 2018, Part-time Engineer, Preferred Networks, Japan
• May 2016 -- Apr 2017, Research Assistant, AIST AIRC, Japan
• Jun 2015 -- Mar 2016, Part-time Researcher, Honda Research Institute, Japan

Funding

• Nov 2023, Research Grant, Google (30K USD)
• Apr 2022 -- Mar 2025, Grant-in-Aid for Early-Career Scientists, JSPS (3.5M JPY) [22K17954]
• Oct 2021, Research Grant, Google (30K USD)
• Dec 2020 -- Mar 2024, PRESTO, JST (39+2.9+4M JPY) [JPMJPR20C4]
• Sep 2020 -- Mar 2022, Grant-in-Aid for Research Activity Start-up, JSPS (2.2M JPY) [20K23335]
• Oct 2019 -- Mar 2021, ACT-X, JST (4.5M JPY) [JPMJAX190G]
• Apr 2018 -- Mar 2020, DC2, JSPS (1.9M JPY)

Activities

• Area chair / Editor
• 2024: EACL (Action Editor for ARR Oct 2023)
• 2023: EACL (Question Answering), ARR (June 2023), ANLP (domestic journal)
• 2022: ACL (Action Editor for ARR Nov 2021), NAACL (Action Editor for ARR Jan 2022), EMNLP (Question Answering)
• 2021: NAACL (Language Resources and Evaluation), ACL-IJCNLP (Resource and Evaluation)
• Reviewer
• 2024: ICLR
• 2023: ACL, ICML
• 2022: AAAI, ICLR, ICML, COLING, NeurIPS
• 2021: ICLR, IJCAI (senior), EMNLP, NeurIPS, ARR, ACM CSUR
• 2020: EMNLP, COLING

Awards

• Mar 2020, Dean's Award, Graduate School of Information Science and Technology, University of Tokyo
• Mar 2019, Excellent Student Award, National Institute of Informatics

Talks

• Evaluation and Explanation of Natural Language Understanding Systems.
  Lecture (online), NLP Seminar by Association for NLP, Japan, Oct 2022.
• Constructing an Explainable Benchmark of Natural Language Understanding.
  Invited talk (online), the 45th IBISML meeting, Japan, Mar 2022.
• Benchmarking Natural Language Understanding: A Psychological and Philosophical Perspective?
  NLP Colloquium (online), Japan, Feb 2022.
• Towards An Explainable Benchmark of Natural Language Understanding.
  STAIR Lab AI Seminar (online), Japan, Nov 2021.
• Evaluation of Natural Language Understanding and Construction of Benchmark.
  AFSA Seminar (online), Japan, Aug 2021.
• Evaluating Natural Language Understanding in Machine Reading Comprehension.
  Invited talk, AIST AIRC, Japan, Aug 2019.
• Evaluation Metrics for the Machine Reading Comprehension Task.
  MiCS, Tohoku University, Japan, Nov 2017.
• Evaluation Metrics for the Machine Reading Comprehension Task.
  Techtalk, Google Tokyo, Japan, Oct 2017.
• An Overview and Analysis of Reading Comprehension Tasks.
  Nagoya NLP seminar, Nagoya University, Japan, Jun 2017.