Can LLMs Understand Computer Networks? Towards a Virtual System Administrator
arxiv(2024)
摘要
Recent advancements in Artificial Intelligence, and particularly Large
Language Models (LLMs), offer promising prospects for aiding system
administrators in managing the complexity of modern networks. However, despite
this potential, a significant gap exists in the literature regarding the extent
to which LLMs can understand computer networks. Without empirical evidence,
system administrators might rely on these models without assurance of their
efficacy in performing network-related tasks accurately.
In this paper, we are the first to conduct an exhaustive study on LLMs'
comprehension of computer networks. We formulate several research questions to
determine whether LLMs can provide correct answers when supplied with a network
topology and questions on it. To assess them, we developed a thorough framework
for evaluating LLMs' capabilities in various network-related tasks. We evaluate
our framework on multiple computer networks employing private (e.g., GPT4) and
open-source (e.g., Llama2) models. Our findings demonstrate promising results,
with the best model achieving an average accuracy of 79.3
achieve noteworthy results in small and medium networks, while challenges
persist in comprehending complex network topologies, particularly for
open-source models. Moreover, we provide insight into how prompt engineering
can enhance the accuracy of some tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要