Saku Sugawara

菅原 朔 (ja)
Assistant Professor at National Institute of Informatics, Tokyo, Japan
Ph.D. in Computer Science (University of Tokyo, 2020)

Email: saku(at)nii.ac.jp

GitHub / Google Scholar

RA を募集しています

かわさき市民セミナー 生成AI講座 スライド（2024年6月）
機械学会 IIP 部門 AI 講習会スライド（2024年3月）
さきがけ「信頼されるAI」成果報告会スライド（2023年11月）
研究テーマのチュートリアル的スライド（2022年10月）

News

• Two papers (on language acquisition and rationale evaluation) have been accepted to EMNLP 2024 (main). (Sep 2024)
• I had an invited poster presentation at YANS 2024. (Sep 2024)
• My proposal on investigating the nature of language models is accepted by Google Research Grant (30K USD). (Aug 2024)
• We present two ACL 2024 Findings papers (Asami et al. and Haga et al.) at the CMCL 2024 Workshop. (Aug 2024)
• My proposal on the evaluation of linguistic intelligence is accepted by JST FOREST Grant (50M JPY in seven years). (Jun 2024)
• previous news

Research Interests

Computational linguistics, natural language processing, natural language understanding, evaluation methods, psychometrics, psycholinguistics, language acquisition, philosophy of science

Publications

2024

• Can Language Models Induce Grammatical Knowledge from Indirect Evidence?
  Miyu Oba, Yohei Oseki, Akiyo Fukatsu, Akari Haga, Hiroki Ouchi, Taro Watanabe, Saku Sugawara
  In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Nov 2024.EMNLP 2024
  [paper] to appear.
• Rationale-Aware Answer Verification by Pairwise Self-Evaluation
  Akira Kawabata, Saku Sugawara
  In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Nov 2024.EMNLP 2024
  [paper] to appear.
• A Corpus Analysis of Noun Phrases with Processing Difficulty in Japanese Legal Language
  Rei Emura, Saku Sugawara, Masatoshi Koizumi
  In the 30th Conference of Architectures and Mechanisms for Language Processing (AMLaP 2024), Sep 2024.AMLaP 2024
  [abstract] [poster]
• What Makes Language Models Good-enough?
  Daiki Asami, Saku Sugawara
  In Findings of the Association for Computational Linguistics: ACL 2024, Aug 2024.Findings of ACL: ACL 2024
  [paper] [poster]
• Modeling Overregularization in Children with Small Language Models
  Akari Haga, Saku Sugawara, Akiyo Fukatsu, Miyu Oba, Hiroki Ouchi, Taro Watanabe, Yohei Oseki
  In Findings of the Association for Computational Linguistics: ACL 2024, Aug 2024.Findings of ACL: ACL 2024
  [paper] [poster]
• Automatic Feedback Generation for Short Answer Questions Using Answer Diagnostic Graphs
  Momoka Furuhashi, Hiroaki Funayama, Yuya Iwase, Yuichiroh Matsubayashi, Yoriko Isobe, Toru Nagahama, Saku Sugawara, Kentaro Inui
  In Proceedings of the 16th annual International Conference on Education and New Learning Technologies (EDULEARN24), Jul 2024.EDULEARN 2024
  [paper] [poster]

2023

• Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension
  Akira Kawabata, Saku Sugawara
  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pp.116-143, Dec 2023.EMNLP 2023
  [paper] [arxiv] [poster] [data]
• PROPRES: Investigating the Projectivity of Presupposition with Various Triggers and Environments
  Daiki Asami, Saku Sugawara
  In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL 2023), pp.122-137, Dec 2023.CoNLL 2023
  [paper] [arxiv] [poster] [data]
• Probing Physical Reasoning with Counter-Commonsense Context
  Kazushi Kondo, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023 short paper), pp.603-612, Jul 2023.ACL 2023 short paper
  [paper] [arxiv]
• On Degrees of Freedom in Defining and Testing Natural Language Understanding
  Saku Sugawara, Shun Tsugita
  In Findings of the Association for Computational Linguistics: ACL 2023, pp.13625-13649, Jul 2023.Findings of ACL: ACL 2023
  [paper] [arxiv]
• Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
  Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa
  In Findings of the Association for Computational Linguistics: EACL 2023, pp.1163-1180, May 2023.Findings of ACL: EACL 2023
  [paper] [arxiv]
• Which Shortcut Solution Do Question Answering Models Prefer to Learn?
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI-23), pp.13564-13572, Feb 2023.AAAI 2023
  [paper] [arxiv] [slide] [poster]
• Penalizing Confident Predictions on Largely Perturbed Inputs Does Not Improve Out-of-Distribution Generalization in Question Answering
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Workshop on Knowledge Augmented Methods for NLP (KnowledgeNLP) at AAAI 2023, Feb 2023.KnowledgeNLP Workshop at AAAI 2023
  [paper] [arxiv] [poster]

2022

• Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP 2022) at EMNLP 2022, pp.418-425, Dec 2022.BlackboxNLP Workshop at EMNLP 2022
  [paper] [arxiv] [poster]
• Debiasing Masks: A New Framework for Shortcut Mitigation in NLU
  Johannes Mario Meissner, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022 short paper), pp.7607-7613, Dec 2022.EMNLP 2022 short paper
  [paper] [arxiv]
• Cross-Modal Similarity-Based Curriculum Learning for Image Captioning
  Hongkuan Zhang, Saku Sugawara, Akiko Aizawa, Lei Zhou, Ryohei Sasano, Koichi Takeda
  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022 short paper), pp.7599-7606, Dec 2022.EMNLP 2022 short paper
  [paper] [arxiv]
• How Well Do Multi-hop Reading Comprehension Models Understand Date Information?
  Xanh Ho, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP 2022 short paper), pp.470-479, Nov 2022.AACL-IJCNLP 2022 short paper
  [paper] [arxiv] [bib] [poster]
• Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios
  Mana Ashida, Saku Sugawara
  In Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022), pp.3606-3630, Oct 2022.COLING 2022
  [paper] [arxiv] [bib] [data] [slide] [poster]
• A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension
  Xanh Ho*, Johannes Mario Meissner*, Saku Sugawara, Akiko Aizawa
  Preprint.
  [arxiv] *: equal contribution
• What Makes Reading Comprehension Questions Difficult?
  Saku Sugawara, Nikita Nangia, Alex Warstadt, Samuel R. Bowman
  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), pp.6951-6971, May 2022.ACL 2022
  [paper] [arxiv] [bib] [data] [slide] [poster]

2021

• Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 3rd Workshop on Machine Reading for Question Answering (MRQA) at EMNLP 2021, pp.63-72, Nov 2021.MRQA Workshop at EMNLP 2021
  [paper] [arxiv] [bib]
• Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (ACL-IJCNLP 2021 SRW), pp.197-214, Aug 2021.ACL-IJCNLP 2021 Student Research Workshop
  [paper] [arxiv] [bib]
• What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
  Nikita Nangia*, Saku Sugawara*, Harsh Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), pp.1221-1235, Aug 2021.ACL-IJCNLP 2021
  [paper] [arxiv] [bib] [data/code] *: equal contribution
• Embracing Ambiguity: Shifting the Training Target of NLI Models
  Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021 short paper), pp.862-869, Aug 2021.ACL-IJCNLP 2021 short paper
  [paper] [arxiv] [bib]
• Benchmarking Machine Reading Comprehension: A Psychological Perspective
  Saku Sugawara, Pontus Stenetorp, Akiko Aizawa
  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), pp.1592-1612, Apr 2021.EACL 2021
  [paper] [arxiv] [bib] [slide] [poster]

2020

• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
  Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020), pp.6609-6625, Dec 2020.COLING 2020
  [paper] [arxiv] [bib]
• Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
  Saku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko Aizawa
  In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-20), pp.8918-8927, Feb 2020.AAAI 2020
  [paper] [arxiv] [bib] [poster] [spotlight] [data]

2018

• What Makes Reading Comprehension Questions Easier?
  Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa
  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp.4208-4219, Nov 2018.EMNLP 2018
  [paper] [arxiv] [bib] [slide] [data]

2017

• Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability
  Saku Sugawara, Yusuke Kido, Hikaru Yokono, Akiko Aizawa
  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), pp.806-817, Aug 2017.ACL 2017
  [paper] [bib] [slide] [data]
  
• Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems
  Saku Sugawara, Hikaru Yokono, Akiko Aizawa
  In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI-17), pp.3089-3096, Feb 2017.AAAI 2017
  [paper] [bib] [slide] [data]

2016

• Annotation and Analysis of Discourse Relations, Temporal Relations and Multi-Layered Situational Relations in Japanese Texts
  Kimi Kaneko, Saku Sugawara, Koji Mineshima, Daisuke Bekki
  In Proceedings of COLING 2016 Workshop on Asian Language Resources 12 (ALR12), pp.10-19, Dec 2016.
  [paper] [bib] [web]
• An Analysis of Prerequisite Skills for Reading Comprehension
  Saku Sugawara, Akiko Aizawa
  In Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing (UBLP), pp.1-5, Nov 2016.
  [paper] [bib] [poster] [web] [data]
• On Dialogue Breakdown: Annotation and Detection - dialogue breakdown detection challenge
  Kotaro Funakoshi, Ryuichiro Higashinaka, Michimasa Inaba, Yuka Kobayashi, Saku Sugawara, Katsuya Takanashi, Hiroko Otsuka, Hanae Koiso, Mayumi Bono
  In IVA 2016 Workshop on Chatbots and Conversational Agent Technologies (WOCHAT), Sep 2016.
  [paper] [web]

Collaborators

• Research Assistants
• Takuma Sato (Nara Institute of Science and Technology, Oct 2024-)
• Taro Hiraoka (Hokkaido University, Oct 2024-)
• Haruto Suzuki (Keio University, Sep 2024-)
• Shogo Tsuboi (Hitotsubashi University, Sep 2024-)
• Momoka Furuhashi (Tohoku University, Dec 2023-Apr 2024; May 2025- in NII LLMC)
• Yoko Kayano (Jul 2023-)
• Miyu Oba (Nara Institute of Science and Technology, Apr 2023-)
• Akari Haga (Nara Institute of Science and Technology, Dec 2022-)
• Rei Emura (Tohoku University, May 2022-)
• Visiting Student
• Daiki Asami (University of Delaware, Apr 2022-; formerly Research Assitant Oct 2021 -- Mar 2022)
• Internship Students
• Yuhang Hu (Technical University of Munich, Oct 2024-)
• Yiwei Liu (École Polytechnique Fédérale de Lausanne, Sep 2024-)
• Former Collaborators
• Junpei Suzuki (Research Assistant from University of Tokyo, Apr 2021 -- Mar 2024)
• Viktor Schlegel (Postdoc Researcher from University of Manchester, Jul 2022 -- Sep 2022)
• Akira Kawabata (Research Assistant from Nara Institute of Science and Technology, Oct 2021 -- Jun 2022)
• Mana Ashida (Research Assistant from Tokyo Metropolitan University, Jun 2021 -- Mar 2022)
• Chihiro Taguchi (Research Assistant from Nara Institute of Science and Technology, Apr 2021 -- Sep 2021)
• Visitors
• Elvira Amador-Domínguez (Universidad Politécnica de Madrid, Jul 2024 -- Aug 2024)
• Yao Lu (University College London, Mar 2024)
• Jiayi Wang (University College London, Mar 2024)
• Eduardo Sánchez González (University College London / Meta, Dec 2023)

Education

• Apr 2017 -- Mar 2020, Ph.D. in Computer Science
  • Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
  • Ph.D. thesis: Evaluating Natural Language Understanding in Machine Reading Comprehension
    [paper] [slide]
• Apr 2015 -- Mar 2017, M.Sc. in Computer Science
  • Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
  • Master thesis: Prerequisite Skills of Natural Language Understanding for the Analysis and Development of Reading Comprehension
• Apr 2013 -- Mar 2015, B.A. in Letters (Philosophy)
  • Philosophy Course, Faculty of Letters, University of Tokyo
  • Graduation thesis: Language Understanding Based on the Concept of Proper Functions (On the biological/teleo- semantics by Ruth G. Millikan)
• Apr 2011 -- Mar 2013
  • Natural Sciences I, College of Arts and Sciences, University of Tokyo

Experience

• Apr 2020 -- Present, Assistant Professor (five years), National Institute of Informatics, Japan
• Apr 2020 -- Present, Visiting Researcher, RIKEN AIP, Japan
• Aug 2024 -- Present, Advisor, Asahi Shimbun, Japan
• Jun 2020 -- Dec 2021, Visiting Researcher, New York University, US
• May 2017 -- Mar 2020, Research Assistant, RIKEN AIP, Japan
• Nov 2018 -- Mar 2019, Visiting Student, University College London, UK
• Jun 2018 -- Sep 2018, Research Intern, Microsoft Research Montreal (Maluuba), Canada
• Apr 2016 -- Mar 2018, Part-time Engineer, Preferred Networks, Japan
• May 2016 -- Apr 2017, Research Assistant, AIST AIRC, Japan
• Jun 2015 -- Mar 2016, Part-time Researcher, Honda Research Institute, Japan

Funding

• Oct 2024 -- Mar 2032, FOREST, JST (50M JPY) [JPMJFR232R]
• Aug 2024, Research Grant, Google (30K USD)
• Nov 2023, Research Grant, Google (30K USD)
• Apr 2022 -- Mar 2025, Grant-in-Aid for Early-Career Scientists, JSPS (3.5M JPY) [22K17954]
• Oct 2021, Research Grant, Google (30K USD)
• Dec 2020 -- Mar 2024, PRESTO, JST (39+2.9+4M JPY) [JPMJPR20C4]
• Sep 2020 -- Mar 2022, Grant-in-Aid for Research Activity Start-up, JSPS (2.2M JPY) [20K23335]
• Oct 2019 -- Mar 2021, ACT-X, JST (4.5M JPY) [JPMJAX190G]
• Apr 2018 -- Mar 2020, DC2, JSPS (1.9M JPY)

Activities

• Area chair / Editor
• 2024: ARR (Action Editor for EACL, NAACL, ACL, EMNLP), COLM
• 2023: EACL (Question Answering), ARR (June 2023), ANLP (domestic journal)
• 2022: ACL (Action Editor for ARR Nov 2021), NAACL (Action Editor for ARR Jan 2022), EMNLP (Question Answering)
• 2021: NAACL (Language Resources and Evaluation), ACL-IJCNLP (Resource and Evaluation)
• Reviewer
• 2025: COLING
• 2024: ICLR, LREC-COLING
• 2023: ACL, ICML
• 2022: AAAI, ICLR, ICML, COLING, NeurIPS
• 2021: ICLR, IJCAI (senior), EMNLP, NeurIPS, ARR, ACM CSUR
• 2020: EMNLP, COLING

Awards

• Mar 2020, Dean's Award, Graduate School of Information Science and Technology, University of Tokyo
• Mar 2019, Excellent Student Award, National Institute of Informatics

Talks

• On the Difficulties of Evaluating Linguistic Generative AI
  Lecture (online), AI Lecture in the Information, Intelligence, and Precision Equipment Division of the Japan Society of Mechanical Engineers, Japan, Mar 2024. [web]
• Evaluation and Explanation of Natural Language Understanding Systems.
  Lecture (online), NLP Seminar by Association for NLP, Japan, Oct 2022. [web]
• Constructing an Explainable Benchmark of Natural Language Understanding.
  Invited talk (online), the 45th IBISML meeting, Japan, Mar 2022.
• Benchmarking Natural Language Understanding: A Psychological and Philosophical Perspective?
  NLP Colloquium (online), Japan, Feb 2022.
• Towards An Explainable Benchmark of Natural Language Understanding.
  STAIR Lab AI Seminar (online), Japan, Nov 2021.
• Evaluation of Natural Language Understanding and Construction of Benchmark.
  AFSA Seminar (online), Japan, Aug 2021.
• Evaluating Natural Language Understanding in Machine Reading Comprehension.
  Invited talk, AIST AIRC, Japan, Aug 2019.
• Evaluation Metrics for the Machine Reading Comprehension Task.
  MiCS, Tohoku University, Japan, Nov 2017.
• Evaluation Metrics for the Machine Reading Comprehension Task.
  Techtalk, Google Tokyo, Japan, Oct 2017.
• An Overview and Analysis of Reading Comprehension Tasks.
  Nagoya NLP seminar, Nagoya University, Japan, Jun 2017.