Speaker
Description
Manual evaluation of terminal logs in cybersecurity labs is difficult for large cohorts, especially in the absence of a centralized infrastructure. Checking the presence of a command in the terminal history cannot confirm that the task was completed. We propose an auditable evaluation framework, based on 4 types of evidence extracted from logs: command, argument, verification and output. The evidence was combined cumulatively on 3 levels of confirmation: syntactic (L1), semantic (L2) and confirmed technical result (L3), corresponding to the log fragment. The framework was tested on 899 eligible logs obtained from 6 heterogeneous labs: network scanning, access control, cryptography, digital signatures, VPN and firewall. For the direct comparison, 550 logs containing the full session transcript were used, with a total of 7192 decisions made, with average scores of 85.34% (L1), 58.59% (L2), and 39.68% (L3). A difference of 45.66 percentage points and a very large effect size (Kendall's W=0.880) was recorded. Procedural tasks had higher scores than multi-stage ones. The framework is sensitive to partial log loss or truncation and invariant to ANSI formatting tests, after data normalization, processes 8.71 logs/second and supports automatic initial cohort assessment, in the absence of centralized collection infrastructure.