Performance Evaluation of Knowledge Graph Embedding Approaches under Non-adversarial Attacks
0
Sign in to get full access
Overview
- The paper evaluates the performance of various knowledge graph embedding approaches under non-adversarial attacks.
- Knowledge graph embeddings are used to represent entities and relationships in a knowledge graph as vectors, enabling machine learning applications.
- The authors assess the robustness of these embeddings to non-adversarial perturbations, which can occur in real-world scenarios due to incomplete or noisy data.
Plain English Explanation
Knowledge graphs are digital representations of information, where entities (like people, places, or things) are connected by relationships (like "located in" or "works for"). Embedding approaches convert these entities and relationships into numeric vectors that can be used in machine learning models.
However, real-world knowledge graphs often contain errors or missing information, which can negatively impact the performance of models that rely on these embeddings. In this paper, the researchers investigate how different embedding approaches hold up when faced with non-adversarial attacks - changes to the knowledge graph that are unintentional, rather than malicious.
By evaluating the robustness of various embedding methods, the authors aim to provide guidance on which techniques are best suited for real-world applications where the knowledge graph may not be perfect.
Technical Explanation
The paper compares the performance of several prominent knowledge graph embedding approaches, including TransE, DistMult, ComplEx, and RotatE, under different types of non-adversarial attacks. These attacks simulate common issues in real-world knowledge graphs, such as missing entities, missing relationships, and noisy relationships.
The authors evaluate the embeddings on standard link prediction and entity classification tasks, measuring how well the models perform as the knowledge graph is progressively perturbed. They also analyze the generalization capabilities of the embeddings, testing their ability to maintain performance on unseen data.
Critical Analysis
The paper provides a comprehensive evaluation of knowledge graph embedding robustness, offering valuable insights for real-world applications. However, the authors note that the non-adversarial attacks they consider may not capture the full complexity of noisy or incomplete knowledge graphs in practice.
Additionally, the study is limited to a few well-known embedding approaches and does not explore more recent or specialized techniques that may exhibit different robustness properties. Further research could investigate a wider range of embedding methods and attack scenarios to provide a more complete understanding of this important problem.
Conclusion
This paper makes a significant contribution to the field of knowledge graph embeddings by rigorously evaluating their performance under non-adversarial attacks. The findings can help researchers and practitioners choose the most suitable embedding approach for their applications, where the underlying knowledge graph may not be perfectly clean or complete. The insights gained from this work can also inform the development of more robust embedding techniques in the future.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Performance Evaluation of Knowledge Graph Embedding Approaches under Non-adversarial Attacks
Sourabh Kapoor, Arnab Sharma, Michael Roder, Caglar Demir, Axel-Cyrille Ngonga Ngomo
Knowledge Graph Embedding (KGE) transforms a discrete Knowledge Graph (KG) into a continuous vector space facilitating its use in various AI-driven applications like Semantic Search, Question Answering, or Recommenders. While KGE approaches are effective in these applications, most existing approaches assume that all information in the given KG is correct. This enables attackers to influence the output of these approaches, e.g., by perturbing the input. Consequently, the robustness of such KGE approaches has to be addressed. Recent work focused on adversarial attacks. However, non-adversarial attacks on all attack surfaces of these approaches have not been thoroughly examined. We close this gap by evaluating the impact of non-adversarial attacks on the performance of 5 state-of-the-art KGE algorithms on 5 datasets with respect to attacks on 3 attack surfaces-graph, parameter, and label perturbation. Our evaluation results suggest that label perturbation has a strong effect on the KGE performance, followed by parameter perturbation with a moderate and graph with a low effect.
Read more7/10/2024
0
Untargeted Adversarial Attack on Knowledge Graph Embeddings
Tianzhe Zhao, Jiaoyan Chen, Yanchi Ru, Qika Lin, Yuxia Geng, Jun Liu
Knowledge graph embedding (KGE) methods have achieved great success in handling various knowledge graph (KG) downstream tasks. However, KGE methods may learn biased representations on low-quality KGs that are prevalent in the real world. Some recent studies propose adversarial attacks to investigate the vulnerabilities of KGE methods, but their attackers are target-oriented with the KGE method and the target triples to predict are given in advance, which lacks practicability. In this work, we explore untargeted attacks with the aim of reducing the global performances of KGE methods over a set of unknown test triples and conducting systematic analyses on KGE robustness. Considering logic rules can effectively summarize the global structure of a KG, we develop rule-based attack strategies to enhance the attack efficiency. In particular,we consider adversarial deletion which learns rules, applying the rules to score triple importance and delete important triples, and adversarial addition which corrupts the learned rules and applies them for negative triples as perturbations. Extensive experiments on two datasets over three representative classes of KGE methods demonstrate the effectiveness of our proposed untargeted attacks in diminishing the link prediction results. And we also find that different KGE methods exhibit different robustness to untargeted attacks. For example, the robustness of methods engaged with graph neural networks and logic rules depends on the density of the graph. But rule-based methods like NCRL are easily affected by adversarial addition attacks to capture negative rules
Read more5/21/2024
0
On The Expressive Power of Knowledge Graph Embedding Methods
Jiexing Gao, Dmitry Rodin, Vasily Motolygin, Denis Zaytsev
Knowledge Graph Embedding (KGE) is a popular approach, which aims to represent entities and relations of a knowledge graph in latent spaces. Their representations are known as embeddings. To measure the plausibility of triplets, score functions are defined over embedding spaces. Despite wide dissemination of KGE in various tasks, KGE methods have limitations in reasoning abilities. In this paper we propose a mathematical framework to compare reasoning abilities of KGE methods. We show that STransE has a higher capability than TransComplEx, and then present new STransCoRe method, which improves the STransE by combining it with the TransCoRe insights, which can reduce the STransE space complexity.
Read more7/29/2024
0
KGPA: Robustness Evaluation for Large Language Models via Cross-Domain Knowledge Graphs
Aihua Pei (Waseda University), Zehua Yang (Waseda University), Shunan Zhu (Waseda University), Ruoxi Cheng (Southeast University), Ju Jia (Southeast University), Lina Wang (Wuhan University)
Existing frameworks for assessing robustness of large language models (LLMs) overly depend on specific benchmarks, increasing costs and failing to evaluate performance of LLMs in professional domains due to dataset limitations. This paper proposes a framework that systematically evaluates the robustness of LLMs under adversarial attack scenarios by leveraging knowledge graphs (KGs). Our framework generates original prompts from the triplets of knowledge graphs and creates adversarial prompts by poisoning, assessing the robustness of LLMs through the results of these adversarial attacks. We systematically evaluate the effectiveness of this framework and its modules. Experiments show that adversarial robustness of the ChatGPT family ranks as GPT-4-turbo > GPT-4o > GPT-3.5-turbo, and the robustness of large language models is influenced by the professional domains in which they operate.
Read more6/18/2024