Speaker
Description
Retrieving the statutory provisions needed to assess a legal statement is a core task in legal information retrieval. For Romanian, existing resources provide legal examination questions and legislative corpora, but article-level relevance annotations remain a gap. We describe an LLM-assisted approach to constructing a statute retrieval evaluation set for the Romanian Civil Code from professional admission examinations.
The source material consists of publicly available civil law questions from admission examinations for magistrates (INM) and lawyers (INPPA). Each question–answer-option pair forms a separate query, preserving the context needed to interpret the option. Official answer keys establish the expected assessment, taking account of the wording of each question. The annotation task is to identify the provision or provisions needed to support that assessment, including those needed to establish why a statement is false.
Language models propose candidate article identifiers from the complete index of Civil Code article headings, avoiding dependence on a candidate pool supplied by a preselected retrieval system. Each proposed article must be accompanied by a verbatim passage, checked automatically against the corresponding corpus text. This check establishes textual traceability; legal relevance and completeness require separate assessment. Models are prompted independently, and their agreement is used to prioritize annotations for human review.
Legal experts will independently evaluate a subset of the annotations for relevance, sufficiency, and omitted provisions. The evaluation will include both model-agreement and model-disagreement cases, allowing us to examine whether agreement is a useful indicator of annotation quality. We will measure inter-model agreement, quotation-verification failure rates, agreement with expert judgments, and inter-expert agreement, with results analyzed by examination source and statement truth value. An error taxonomy will characterize recurring annotation failures. The planned release comprises the annotation pipeline and an expert-reviewed evaluation subset, with disagreements adjudicated before release.