Validator Update #1 — The double-signing incident

The blockscape validator exists since the birth of Cosmos Hub 3 and is run by a group of blockchain enthusiasts with both high availability and the highest level of security in mind. You can find out more about us at www.blockscape.network.

What Happened?

Today, on May 20th, our Cosmos Hub 3 validator had a double-signing incident at block height 1,955,889. As a result, all of our delegators’ stakes have been partially slashed in the process. Our team worked hard to resolve the issue and managed to find the cause for the error as well as put a fix in place to prevent it from happening again.

We found out that the incident was caused by a misconfiguration in our unique active/active validator setup which entails running multiple validators with the same key pair in parallel. In order to avoid double-signing, we have a custom software solution in place that tracks which validator instance signs each individual block and ultimately only lets one validator sign any given message.

The way our system works is that it keeps a certain number of entries in the permission log that have already been signed — that number was set too low. What happened then was one of our nodes unexpectedly couldn’t keep up with synchronizing its blockchain over the network and operated on an outdated version. The problem was that the permission log didn’t have those entries anymore which ultimately caused it to sign a message at block height 1,955,889 that had already been signed previously by another of our validators. We have significantly increased the number of entries we’re keeping in the permission log to prevent this from happening again.

We found the misconfiguration, we corrected it and we plan to spin up a new validator. We’re already hard at work getting our next-level solution ready for production. You can check out our latest medium article and check out the code for yourself — it’s open-source.

We sincerely apologize to all of our delegators and we hope we can keep or regain your trust in the future. We will share the details of our new validator as soon as possible.

With that being said, we’re going to continue being transparent about what’s happening and how we’re trying to improve in the future.

Thank you for trusting blockscape and see you next time!