Penzant.net

Saku Sugawara

菅原朔 (ja)
Assistant Professor at National Institute of Informatics, Tokyo, Japan
Ph.D. in Computer Science (University of Tokyo, 2020)

Email: saku(at)nii.ac.jp

CV / GitHub / Google Scholar

RA を募集しています

Research Area

Computational linguistics, natural language processing
Natural language understanding, machine reading comprehension
Task design, dataset construction

Publications

2022

What Makes Reading Comprehension Questions Difficult?
Saku Sugawara, Nikita Nangia, Alex Warstadt, Samuel R. Bowman
In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), May 2022, pp.6951-6971.ACL 2022
[paper] [arxiv] [bib] [data] [slide] [poster]

2021

Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
In Proceedings of the 3rd Workshop on Machine Reading for Question Answering (MRQA) at EMNLP 2021, Nov 2021, pp.63-72.EMNLP 2021 MRQA Workshop
[paper] [arxiv] [bib]
Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
Kazutoshi Shinoda, Saku Sugawara, Akiko Aizawa
In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (ACL-IJCNLP 2021 SRW), Aug 2021, pp.197-214.ACL-IJCNLP 2021 Student Research Workshop
[paper] [arxiv] [bib]
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks?
Nikita Nangia*, Saku Sugawara*, Harsh Trivedi, Alex Warstadt, Clara Vania, Samuel R. Bowman
In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), Aug 2021, pp.1221-1235.ACL-IJCNLP 2021
[paper] [arxiv] [bib] [data/code] *: equal contribution
Embracing Ambiguity: Shifting the Training Target of NLI Models
Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, Akiko Aizawa
In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021 short paper), Aug 2021, pp.862-869.ACL-IJCNLP 2021 short paper
[paper] [arxiv] [bib]
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Saku Sugawara, Pontus Stenetorp, Akiko Aizawa
In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), Apr 2021, pp.1592-1612.EACL 2021
[paper] [arxiv] [bib] [slide] [poster]

2020

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa
In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020), Dec 2020, pp.6609-6625.COLING 2020
[paper] [arxiv] [bib]
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Saku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko Aizawa
In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI-20), Feb 2020, pp.8918-8927.AAAI 2020
[paper] [arxiv] [bib] [poster] [spotlight] [data]

2018

What Makes Reading Comprehension Questions Easier?
Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), Nov 2018, pp.4208-4219.EMNLP 2018
[paper] [arxiv] [bib] [slide] [data]

2017

Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability
Saku Sugawara, Yusuke Kido, Hikaru Yokono, Akiko Aizawa
In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Aug 2017, pp.806-817.ACL 2017
[paper] [bib] [slide] [data]
Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems
Saku Sugawara, Hikaru Yokono, Akiko Aizawa
In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI-17), Feb 2017, pp.3089-3096.AAAI 2017
[paper] [bib] [slide] [data]

2016

Annotation and Analysis of Discourse Relations, Temporal Relations and Multi-Layered Situational Relations in Japanese Texts
Kimi Kaneko, Saku Sugawara, Koji Mineshima, Daisuke Bekki
In Proceedings of COLING 2016 Workshop on Asian Language Resources 12 (ALR12), Dec 2016, pp.10-19.
[paper] [bib] [web]
An Analysis of Prerequisite Skills for Reading Comprehension
Saku Sugawara, Akiko Aizawa
In Proceedings of EMNLP 2016 Workshop on Uphill Battles in Language Processing (UBLP), Nov 2016, pp.1-5.
[paper] [bib] [poster] [web] [data]
On Dialogue Breakdown: Annotation and Detection - dialogue breakdown detection challenge
Kotaro Funakoshi, Ryuichiro Higashinaka, Michimasa Inaba, Yuka Kobayashi, Saku Sugawara, Katsuya Takanashi, Hiroko Otsuka, Hanae Koiso, Mayumi Bono
In IVA 2016 Workshop on Chatbots and Conversational Agent Technologies (WOCHAT), Sep 2016.
[paper] [web]

Research Assistants

Rei Emura (Tohoku University, May 2022-)
Akira Kawabata (Nara Institute of Science and Technology, Oct 2021-)
Junpei Suzuki (University of Tokyo, Apr 2021-)
Daiki Asami (University of Delaware, Oct 2021 -- Mar 2022; now visiting student at NII)
Mana Ashida (Tokyo Metropolitan University, Jun 2021 -- Mar 2022)
Chihiro Taguchi (Nara Institute of Science and Technology, Apr 2021 -- Sep 2021)

Education

Apr 2017 -- Mar 2020, Ph.D. in Computer Science
- Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
- Ph.D. thesis: Evaluating Natural Language Understanding in Machine Reading Comprehension
  [paper] [slide]
Apr 2015 -- Mar 2017, M.Sc. in Computer Science
- Department of Computer Science, Graduate School of Information Science and Technology, University of Tokyo
- Master thesis: Prerequisite Skills of Natural Language Understanding for the Analysis and Development of Reading Comprehension
Apr 2013 -- Mar 2015, B.A. in Letters (Philosophy)
- Philosophy Course, Faculty of Letters, University of Tokyo
- Graduation thesis: Language Understanding Based on the Concept of Proper Function (On the biological/teleo- semantics by Ruth G. Millikan)
Apr 2011 -- Mar 2013
- Natural Sciences I, College of Arts and Sciences, University of Tokyo

Experience

Apr 2020 -- Present, Assistant Professor (five years), National Institute of Informatics, Japan
Jun 2020 -- Present, Visiting Researcher, New York University, US
Apr 2020 -- Present, Visiting Researcher, RIKEN AIP, Japan
May 2017 -- Mar 2020, Research Assistant, RIKEN AIP, Japan
Nov 2018 -- Mar 2019, Visiting Student, University College London, UK
Jun 2018 -- Sep 2018, Research Intern, Microsoft Research Montreal (Maluuba), Canada
Apr 2016 -- Mar 2018, Part-time Engineer, Preferred Networks, Japan
May 2016 -- Apr 2017, Research Assistant, AIST AIRC, Japan
Jun 2015 -- Mar 2016, Part-time Researcher, Honda Research Institute, Japan

Funding

Apr 2022 -- Mar 2025, Grant-in-Aid for Early-Career Scientists, JSPS (3.5M JPY)
Oct 2021, Research Grant, Google (30K USD)
Dec 2020 -- Mar 2024, PRESTO, JST (39M+2.9M JPY)
Sep 2020 -- Mar 2022, Grant-in-Aid for Research Activity Start-up, JSPS (2.2M JPY)
Oct 2019 -- Mar 2021, ACT-X, JST (4.5M JPY)
Apr 2018 -- Mar 2020, DC2, JSPS (1.9M JPY)

Activities

Area chair

2022: ACL (Action Editor for ARR Nov 2021), NAACL (Action Editor for ARR Jan 2022)
2021: NAACL (Language Resources and Evaluation), ACL-IJCNLP (Resource and Evaluation)

Reviewer

2022: AAAI, ICLR, ICML...
2021: ICLR, IJCAI (senior), EMNLP, NeurIPS, ARR, ACM CSUR
2020: EMNLP, COLING

Awards

Mar 2020, Dean's Award, Graduate School of Information Science and Technology, University of Tokyo
Mar 2019, Excellent Student Award, National Institute of Informatics

Talks

Constructing an Explainable Benchmark of Natural Language Understanding.
Invited talk (online), the 45th IBISML meeting, Japan, Mar 2022.
Benchmarking Natural Language Understanding: A Psychological and Philosophical Perspective?
NLP Colloquium (online), Japan, Feb 2022.
Towards An Explainable Benchmark of Natural Language Understanding.
STAIR Lab AI Seminar (online), Japan, Nov 2021.
Evaluation of Natural Language Understanding and Construction of Benchmark.
AFSA Seminar (online), Japan, Aug 2021.
Evaluating Natural Language Understanding in Machine Reading Comprehension.
Invited talk, AIST AIRC, Japan, Aug 2019.
Evaluation Metrics for the Machine Reading Comprehension Task.
MiCS, Tohoku University, Japan, Nov 2017.
Evaluation Metrics for the Machine Reading Comprehension Task.
Techtalk, Google Tokyo, Japan, Oct 2017.
An Overview and Analysis of Reading Comprehension Tasks.
Nagoya NLP seminar, Nagoya University, Japan, Jun 2017.