Datasets and Models Used to Analyze Legal Documents

Table of Links

Abstract and 1. Introduction

2. Related Work

3. Task, Datasets, Baseline

4. RQ 1: Leveraging the Neighbourhood at Inference

   4.1. Methods

   4.2. Experiments

5. RQ 2: Leveraging the Neighbourhood at Training

   5.1. Methods

   5.2. Experiments

6. RQ 3: Cross-Domain Generalizability

7. Conclusion

8. Limitations

9. Ethics Statement

10. Bibliographical References

3. Task, Datasets, Baseline

Data We experiment on four datasets - (i) Build (Kalamkar et al., 2022) comprises judgments from Indian supreme court, high court, and district courts. It includes publicly available train and validation splits, with 184 and 30 documents respectively with a total of 31865 sentences (an average of 115 per document). These documents pertain to tax and criminal law cases and are annotated with 13 rhetorical role labels, including ‘None’. Given the absence of a public test dataset, we utilize the training dataset for both training and validation, evaluating performance on the validation partition. (ii) Paheli (Bhattacharya et al., 2021) features 50 judgments from the Supreme Court of India across five domains: Criminal, Land and Property, Constitutional, Labour and Industrial, and Intellectual Property Rights, annotated with 7 rhetorical roles. They have total of 9380 sentences with an average of 188 per document. (iii) M-CL / (iv) M-IT (Malik et al., 2022) encompasses judgments from the Supreme Court of India, High Courts, and Tribunal courts. It includes two subsets: M-CL, comprising 50 documents related to Competition Law, and M-IT, with 50 documents related to Income Tax cases. Both subsets are annotated with 7 rhetorical role labels. M-CL has 13,328 sentences (avg. of 266 per document) and M-IT has a total of 7856 sentences (avg. of 157 per document). We split (at document level) Paheli/M-CL/M-IT into 80% train, 10% validation, and 10% test set.

Baseline All of our experiments in this study are built on top of the Hierarchical Sequential Labeling Network, which served as a baseline in prior works (Kalamkar et al., 2022; Santosh et al.,2023). Initially, each sentence xi is encoded independently using a BERT model (Kenton and Toutanova, 2019) to derive token-level representations zi = {zi1, zi2, . . . , zin}. These representations are passed through a Bi-LSTM layer (Hochreiter and Schmidhuber, 1997), followed by an attention pooling layer (Yang et al., 2016), to yield sentence representations s = {s1, s2, . . . , sm}.