KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards
LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existing evaluations focus on knowledge-based assessments or end-to-end agentic tasks, and do not directly measure LLMs' ability to generate executable commands for real-world cybersecurity tools. This gap is critical because cybersecur
Reading in Plain English mode — toggle below to switch
LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existing evaluations focus on knowledge-based assessments or end-to-end agentic tasks, and do not directly measure LLMs' ability to generate executable commands for real-world cybersecurity tools. This gap is critical because cybersecur
This paper introduces a critical benchmark for evaluating LLMs' ability to generate precise command-line interface (CLI) commands for cybersecurity tools, a key component for agentic workflows. For ML engineers, this highlights the challenges of grounding LLMs in real-world, syntax-sensitive environments and the need for robust evaluation metrics beyond general language understanding.