Reading list
Foundations for working at the intersection of law, language and AI evaluation. Within each area the list runs from an introduction to the deep end. The research papers can be read alongside the basics.
Legal method and argumentation
Rechtstheorie und Juristische Methodenlehre
Bernd Rüthers, Christian Fischer, Axel Birk · C.H.Beck, 12. Aufl. 2022
The systematic foundation: legal norms, language, sources of law and the application of law in one place.
★ Start here
Die Auslegung von Gesetzen
Rolf Wank, Martin Maties · Vahlen, 7. Aufl. 2023
Short and example-driven: spotting, weighing and transferring arguments of statutory interpretation.
Methodenlehre der Rechtswissenschaft
Karl Larenz, Claus-Wilhelm Canaris · Springer, 3. Aufl. (Studienausgabe) 1995
The dense classic, read alongside Rüthers/Fischer/Birk: interpretation, judicial development of the law and legal systematisation.
Theorie der juristischen Argumentation
Robert Alexy · Suhrkamp
When is a legal justification rational and checkable? Explains why the quality of an answer goes beyond its result.
Law and logic: a review from an argumentation perspective
Henry Prakken, Giovanni Sartor · Artificial Intelligence 227, 2015
The bridge to legal AI: how rules, interpretation and argument are modelled formally.
Paper · Free to read
Semantics, pragmatics and logic
Analyzing Meaning: An Introduction to Semantics and Pragmatics
Paul Kroeger · Language Science Press, 3rd ed. 2022
Implicature, quantifiers, modality and conditionals with exercises, no formal logic required.
★ Start here · Free to read
Language, Proof, and Logic
Dave Barker-Plummer, Jon Barwise, John Etchemendy · CSLI Publications, 2nd ed. 2011
Propositional and first-order logic to practise; the first two parts are enough to start.
Pragmatics
Stephen C. Levinson · Cambridge University Press, 1983
Presupposition, speech acts and deixis: what is said, presupposed or only inferred.
Semantics in Generative Grammar
Irene Heim, Angelika Kratzer · Blackwell, 1998
The formal deep end: how sentence meaning is composed from its parts.
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer Singh · ACL 2020
Turns linguistic capabilities into concrete tests; worth reading early.
Paper · Free to read
Measurement and test design
Testtheorie und Fragebogenkonstruktion
Helfried Moosbrugger, Augustin Kelava (Hrsg.) · Springer, 3. Aufl. 2020
The standard German textbook on test construction, quality criteria, item analysis and reliability.
★ Start here
ECBD: Evidence-Centered Benchmark Design for NLP
Yu Lu Liu et al. · ACL 2024
Links the ability measured to tasks and evidence; best worked through on a benchmark of your own.
★ Start here · Paper · Free to read
Standards for Educational and Psychological Testing
AERA, APA, NCME · 2014
Chapters 1, 2 and 4: what evidence an interpretation of test scores needs.
Free to read
Validating the Interpretations and Uses of Test Scores
Michael T. Kane · Journal of Educational Measurement 50(1), 2013
Validation as a chain of inferences: from an observed answer to a claim about ability.
The Theory and Practice of Item Response Theory
R. J. de Ayala · Guilford Press, 2nd ed. 2022
Item response theory from simple to complex models; assumes statistics.
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
Ziang Xiao, Susu Zhang, Vivian Lai, Q. Vera Liao · EMNLP 2023
Reliability and validity applied to metrics and judges themselves.
Paper · Free to read
Probability, statistics and study design
Regression and Other Stories
Andrew Gelman, Jennifer Hill, Aki Vehtari · Cambridge University Press, 2020
The key statistics reference for everyday research: measurement, simulation, regression and design.
★ Start here · Free to read
OpenIntro Statistics
David M. Diez, Mine Çetinkaya-Rundel, Christopher D. Barr · 4th ed. 2019
Entry point or refresher: inference, group comparisons and first regression models.
Free to read
Introduction to Probability
Joseph K. Blitzstein, Jessica Hwang · CRC Press, 2nd ed. 2019
Probability to work through by hand; needs some calculus.
Free to read
Research Design in the Social Sciences
Graeme Blair, Alexander Coppock, Macartan Humphreys · Princeton University Press, 2023
Planning studies and checking them by simulation, such as human baselines and conditions.
Free to read
Statistical Rethinking
Richard McElreath · CRC Press, 2nd ed. 2020
Bayesian and hierarchical models for when cases, answers and raters depend on each other.
The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing
Rotem Dror, Gili Baumer, Segev Shlomov, Roi Reichart · ACL 2018
Which test for which model comparison; readable right after the basics.
Paper · Free to read
Mathematics and how models work
Mathematics for Machine Learning
Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong · Cambridge University Press, 2020
Linear algebra, probability and optimisation in one place.
★ Start here · Free to read
Introduction to Applied Linear Algebra
Stephen Boyd, Lieven Vandenberghe · Cambridge University Press, 2018
Linear algebra through applications: inner products, norms, distances and least squares.
Free to read
Understanding Deep Learning
Simon J. D. Prince · MIT Press, 2023
Neural networks up to transformers, explained mathematically.
Free to read
Speech and Language Processing
Daniel Jurafsky, James H. Martin · 3rd ed. draft, online
The standard NLP textbook: tokenisation, embeddings, transformers, post-training and RAG.
Free to read
On Calibration of Modern Neural Networks
Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger · ICML 2017
How reliable is a model's confidence? The second question next to accuracy.
Paper · Free to read
Working with data, reproducibly
Python for Data Analysis
Wes McKinney · O'Reilly, 3rd ed. 2022
pandas and NumPy for cleaning, joining and aggregating evaluation data.
★ Start here · Free to read
Good Enough Practices in Scientific Computing
Greg Wilson et al. · PLOS Computational Biology 13(6), 2017
Short and immediately useful: organising data, code and workflows so others can follow.
Free to read
The Turing Way
The Turing Way Community · Online handbook
A reference for reproducible, collaborative research; read section by section as needed.
Free to read
Data Statements for Natural Language Processing
Emily M. Bender, Batya Friedman · TACL 6, 2018
Documenting datasets so it is clear what their results hold for.
Paper · Free to read
Leakage and the Reproducibility Crisis in Machine-Learning-Based Science
Sayash Kapoor, Arvind Narayanan · Patterns 4(9), 2023
A taxonomy of data leakage that inflates scores.
Paper · Free to read
IT and data law for research
Datenrecht. Ein Studienbuch
Moritz Hennemann · C.H.Beck, 2025
Data law as one field: generating, using and trading data.
★ Start here
Rechtsfragen bei Open Science. Ein Leitfaden
Till Kreutzer, Henning Lahmann · Hamburg University Press, 2. Aufl. 2021
An accessible entry point: usage rights, open publication and sharing research material.
Free to read
Datenschutzrecht
Jürgen Kühling, Manuel Klar, Florian Sackmann · C.F. Müller, 6. Aufl. 2025
A systematic introduction to data protection, for example for studies with human ratings.
Urheber- und Urhebervertragsrecht
Haimo Schack · Mohr Siebeck, 11. Aufl. 2025
The copyright deep dive: what you may do with texts, corpora and teaching material.
DS-GVO/BDSG. Kommentar
Jürgen Kühling, Benedikt Buchner (Hrsg.) · C.H.Beck, 4. Aufl. 2024
For looking up concrete data protection questions from data processing and human studies.