Title: Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings

Abstract:

Knowledge Graph Embedding Models (KGEMs) serve as the primary methodology for link prediction tasks aimed at completing knowledge graphs. While standard evaluation frameworks prioritize rank-based metrics like MRR or Hits@$K$, they frequently neglect how random seeds impact the stability of outcomes. Additionally, these conventional metrics mask potential instabilities inherent in individual predictions and the structural arrangement of embedding spaces.

In this study, we perform a comprehensive stability analysis of various KGEMs across multiple datasets. Our results indicate that models achieving high performance actually yield divergent predictions at the triple level and generate significantly variable embedding spaces. By isolating stochastic elements—specifically initialization, triple ordering, negative sampling, dropout, and hardware—we demonstrate that each factor independently contributes to instability to a comparable degree.

Furthermore, we observe that for any specific model, hyperparameter settings that achieve superior MRR scores do not necessarily ensure greater stability. Although voting is recognized as a remediation strategy, our findings show it offers only marginal improvements in stability. These insights expose significant shortcomings in existing benchmarking protocols and cast doubt on the reliability of KGEMs for knowledge graph completion.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC