Saku Sugawara

菅原 朔 (ja)
Assistant Professor at National Institute of Informatics, Tokyo, Japan
Ph.D. in Computer Science (University of Tokyo, 2020)

Email: saku(at)nii.ac.jp

CV / GitHub / Google Scholar

RA を募集しています

Research Area

• Computational linguistics, natural language processing
• Natural language understanding, machine reading comprehension
• Task design, dataset construction

Publications

2022

• How Well Do Multi-hop Reading Comprehension Models Understand Date Information?
  Xanh Ho, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP 2022 short paper), Nov 2022.AACL-IJCNLP 2022 short paper
  To appear.
• Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios
  Mana Ashida, Saku Sugawara
  In Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022), Oct 2022.COLING 2022
  [paper] [arxiv] [bib] [data] [slide] [poster]
• A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension
  Xanh Ho*, Johannes Mario Meissner*, Saku Sugawara, Akiko Aizawa
  Preprint.
  [arxiv] *: equal contribution
• What Makes Reading Comprehension Questions Difficult?
  Saku Sugawara, Nikita Nangia, Alex Warstadt, Samuel R. Bowman
  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), May 2022, pp.6951-6971.ACL 2022
  [paper] [arxiv] [bib] [data] [slide] [poster]

2021

• Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 3rd Workshop on Machine Reading for Question Answering (MRQA) at EMNLP 2021, Nov 2021, pp.63-72.EMNLP 2021 MRQA Workshop
  [paper] [arxiv] [bib]
• Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
  Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (ACL-IJCNLP 2021 SRW), Aug 2021, pp.197-214.ACL-IJCNLP 2021 Student Research Workshop
  [paper] [arxiv] [bib]
• What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
  Nikita Nangia*, Saku Sugawara*, Harsh Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), Aug 2021, pp.1221-1235.ACL-IJCNLP 2021
  [paper] [arxiv] [bib] [data/code] *: equal contribution
• Embracing Ambiguity: Shifting the Training Target of NLI Models
  Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, Akiko Aizawa
  In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021 short paper), Aug 2021, pp.862-869.ACL-IJCNLP 2021 short paper
  [paper] [arxiv] [bib]
• Benchmarking Machine Reading Comprehension: A Psychological Perspective
  Saku Sugawara, Pontus Stenetorp, Akiko Aizawa
  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), Apr 2021, pp.1592-1612.EACL 2021
  [paper] [arxiv] [bib] [slide] [poster]

2020

• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
  Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa
  In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020), Dec 2020, pp.6609-6625.COLING 2020
  [paper] [arxiv] [bib]
• Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
  Saku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko Aizawa
  In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-20), Feb 2020, pp.8918-8927.AAAI 2020
  [paper] [arxiv] [bib] [poster] [spotlight] [data]

2018

• What Makes Reading Comprehension Questions Easier?
  Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa
  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), Nov 2018, pp.4208-4219.EMNLP 2018
  [paper] [arxiv] [bib] [slide] [data]

2017

• Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability
  Saku Sugawara, Yusuke Kido, Hikaru Yokono, Akiko Aizawa
  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Aug 2017, pp.806-817.ACL 2017
  [paper] [bib] [slide] [data]
  
• Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems
  Saku Sugawara, Hikaru Yokono, Akiko Aizawa
  In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI-17), Feb 2017, pp.3089-3096.AAAI 2017
  [paper] [bib] [slide] [data]

2016

• Annotation and Analysis of Discourse Relations, Temporal Relations and Multi-Layered Situational Relations in Japanese Texts
  Kimi Kaneko, Saku Sugawara, Koji Mineshima, Daisuke Bekki
  In Proceedings of COLING 2016 Workshop on Asian Language Resources 12 (ALR12), Dec 2016, pp.10-19.
  [paper] [bib] [web]
• An Analysis of Prerequisite Skills for Reading Comprehension
  Saku Sugawara, Akiko Aizawa
  In Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing (UBLP), Nov 2016, pp.1-5.
  [paper] [bib] [poster] [web] [data]
• On Dialogue Breakdown: Annotation and Detection - dialogue breakdown detection challenge
  Kotaro Funakoshi, Ryuichiro Higashinaka, Michimasa Inaba, Yuka Kobayashi, Saku Sugawara, Katsuya Takanashi, Hiroko Otsuka, Hanae Koiso, Mayumi Bono
  In IVA 2016 Workshop on Chatbots and Conversational Agent Technologies (WOCHAT), Sep 2016.
  [paper] [web]

Collaborators

• Visiting Researcher
• Viktor Schlegel (University of Manchester, Jul 2022-)
• Research Assistant
• Rei Emura (Tohoku University, May 2022-)
• Junpei Suzuki (University of Tokyo, Apr 2021-)
• Akira Kawabata (Nara Institute of Science and Technology, Oct 2021 -- Jun 2022)
• Daiki Asami (University of Delaware, Oct 2021 -- Mar 2022; now visiting student at NII)
• Mana Ashida (Tokyo Metropolitan University, Jun 2021 -- Mar 2022)
• Chihiro Taguchi (Nara Institute of Science and Technology, Apr 2021 -- Sep 2021)

Education

• Apr 2017 -- Mar 2020, Ph.D. in Computer Science
  • Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
  • Ph.D. thesis: Evaluating Natural Language Understanding in Machine Reading Comprehension
    [paper] [slide]
• Apr 2015 -- Mar 2017, M.Sc. in Computer Science
  • Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
  • Master thesis: Prerequisite Skills of Natural Language Understanding for the Analysis and Development of Reading Comprehension
• Apr 2013 -- Mar 2015, B.A. in Letters (Philosophy)
  • Philosophy Course, Faculty of Letters, University of Tokyo
  • Graduation thesis: Language Understanding Based on the Concept of Proper Function (On the biological/teleo- semantics by Ruth G. Millikan)
• Apr 2011 -- Mar 2013
  • Natural Sciences I, College of Arts and Sciences, University of Tokyo

Experience

• Apr 2020 -- Present, Assistant Professor (five years), National Institute of Informatics, Japan
• Apr 2020 -- Present, Visiting Researcher, RIKEN AIP, Japan
• Jun 2020 -- Dec 2021, Visiting Researcher, New York University, US
• May 2017 -- Mar 2020, Research Assistant, RIKEN AIP, Japan
• Nov 2018 -- Mar 2019, Visiting Student, University College London, UK
• Jun 2018 -- Sep 2018, Research Intern, Microsoft Research Montreal (Maluuba), Canada
• Apr 2016 -- Mar 2018, Part-time Engineer, Preferred Networks, Japan
• May 2016 -- Apr 2017, Research Assistant, AIST AIRC, Japan
• Jun 2015 -- Mar 2016, Part-time Researcher, Honda Research Institute, Japan

Funding

• Apr 2022 -- Mar 2025, Grant-in-Aid for Early-Career Scientists, JSPS (3.5M JPY)
• Oct 2021, Research Grant, Google (30K USD)
• Dec 2020 -- Mar 2024, PRESTO, JST (39M+2.9M JPY)
• Sep 2020 -- Mar 2022, Grant-in-Aid for Research Activity Start-up, JSPS (2.2M JPY)
• Oct 2019 -- Mar 2021, ACT-X, JST (4.5M JPY)
• Apr 2018 -- Mar 2020, DC2, JSPS (1.9M JPY)

Activities

• Area chair
• 2022: ACL (Action Editor for ARR Nov 2021), NAACL (Action Editor for ARR Jan 2022), EMNLP (Question Answering)
• 2021: NAACL (Language Resources and Evaluation), ACL-IJCNLP (Resource and Evaluation)
• Reviewer
• 2022: AAAI, ICLR, ICML, COLING, NeurIPS, ...
• 2021: ICLR, IJCAI (senior), EMNLP, NeurIPS, ARR, ACM CSUR
• 2020: EMNLP, COLING

Awards

• Mar 2020, Dean's Award, Graduate School of Information Science and Technology, University of Tokyo
• Mar 2019, Excellent Student Award, National Institute of Informatics

Talks

• Constructing an Explainable Benchmark of Natural Language Understanding.
  Invited talk (online), the 45th IBISML meeting, Japan, Mar 2022.
• Benchmarking Natural Language Understanding: A Psychological and Philosophical Perspective?
  NLP Colloquium (online), Japan, Feb 2022.
• Towards An Explainable Benchmark of Natural Language Understanding.
  STAIR Lab AI Seminar (online), Japan, Nov 2021.
• Evaluation of Natural Language Understanding and Construction of Benchmark.
  AFSA Seminar (online), Japan, Aug 2021.
• Evaluating Natural Language Understanding in Machine Reading Comprehension.
  Invited talk, AIST AIRC, Japan, Aug 2019.
• Evaluation Metrics for the Machine Reading Comprehension Task.
  MiCS, Tohoku University, Japan, Nov 2017.
• Evaluation Metrics for the Machine Reading Comprehension Task.
  Techtalk, Google Tokyo, Japan, Oct 2017.
• An Overview and Analysis of Reading Comprehension Tasks.
  Nagoya NLP seminar, Nagoya University, Japan, Jun 2017.