Correctness before speed
I first built small, directed test graphs where I knew the expected neighbors and reachability results. If two engines interpret the same edge differently, their timings are not a meaningful comparison. I treated matching answers as a prerequisite for every benchmark.
Measure the same work
Graph construction and neighbor lookup are different operations, so I measured them separately. I also distinguished one query from a batch of queries and used repeated measurements rather than relying on a single run. The benchmark UI reports latency, throughput and resource usage in context.
The result was not the assumption
In my early synthetic workloads, NetworkX was faster than LadybugDB for both graph construction and neighbor queries. An improved LadybugDB query path narrowed the gap substantially, which showed me how much implementation details affect a headline benchmark. The results describe these workloads, not a universal ranking of graph databases.
From numbers to a useful product
I later expanded GraphBench toward software dependency analysis: graph structure, centrality, cycles, and failure-impact exploration. The lesson was that a benchmark becomes more useful when users can connect the measurements to an actual decision.
What I would test next
I would add broader query mixes, larger input varieties, cold-versus-warm runs, controlled memory measurement and documented environment settings. A trustworthy benchmark should be reproducible and explicit about its limits.