CometBFT™ is a fork and successor of Tendermint Core that serves as the official replication engine for powering the Interchain Stack. CometBFT is being built and maintained by the team that previously maintained Tendermint Core. To read more about CometBFT, refer to this article.
The purpose of this blog is to describe how the QA process was done for CometBFT v0.34.27 including the methodologies, key learnings, and takeaways. Release v0.34.27 might seem like just a patch (.27) bump, but in fact, this is the first-ever release of CometBFT since this repository came to life as a successor to Tendermint Core. In particular, release line v0.34 of CometBFT faces a unique challenge because we want it to be fully compatible and integrate smoothly with networks running on the Tendermint Core v0.34 line, i.e., act as a drop-in replacement for Tendermint Core v0.34.
The blog also highlights the importance of QA for ensuring the quality, reliability, and performance of CometBFT software. Lastly, the blog also sheds light on the features and improvements of CometBFT.
A comprehensive testing strategy is crucial for CometBFT to maintain the integrity of the blockchain network. The strategy for v0.34 includes a combination of different approaches, such as unit testing, integration testing, and end-to-end (E2E) testing.
Unit testing involves writing test cases for every code commit, ensuring that changes do not compromise the system's functionality or break existing code. We track code path testing and run randomly generated tests nightly, with longer runs happening weekly.
Integration testing tests the integration of CometBFT components to ensure expected functionality.
E2E testing ensures blockchain network functions correctly and tests different scenarios with varying network conditions while ensuring its performance does not degrade relative to prior releases. A test generator is used for different topologies and configurations while the tests are run with nodes starting at different heights to simulate different network conditions.
Random executions test the resilience and isolate issues. If a test fails during nightly testing, the test is rerun locally to isolate the problem. The manifest file specifies testing conditions and parameters, including permutations such as restarting validators, introducing new validators to perform blocksync and statesync after a large number of heights has been completed, or doing an uncoordinated upgrade of nodes.
CometBFT 0.34 utilizes standard QA tools such as Golints and triggered checks to ensure code quality and detect vulnerabilities. Dependbot is used to automatically update dependencies, while GitHub actions are used for continuous integration to keep the software up to date with the latest changes. Prometheus is used to monitor system performance and prevent potential problems, while load testing ensures the system can handle heavy traffic and identify performance issues. Finally, querying the blockchain state is essential for QA to ensure CometBFT v0.34 is working correctly and providing insights into system performance.
QA is critical for each CometBFT release to ensure correctness, compatibility, bug fixing, and prevent performance regressions. It increases confidence in the system's correctness and ensures compatibility with other software components. QA helps identify and fix bugs through manual and automated testing and prevents performance regressions by identifying and resolving potential issues early in the development cycle. QA also ensures compatibility and seamless integration of CometBFT with other software components in the Interchain Stack, such as Cosmos SDK. QA tests were also performed on CometBFT v0.34 to ensure compatibility with nodes running Tendermint Core 0.34.x in the same chain without breaking compatibility.
CometBFT v0.34.27 fixed a busyloop by adding a sleep mechanism when sending a block part failed; resolved crashes related to event values with slashes; introduced a short-term fix for needProofBlock not finding the previous block meta; and improved the underlying behavior of the non-blocking TrySendEnvelope method.
CometBFT 0.34 added a "match_event" query parameter to the RPC; functionality for uncoordinated minor upgrades; appending the commit hash to the version, new metrics in the consensus module; reduced log spam; and reduced bandwidth consumption of consensus votes. These changes contributed to improving CometBFT's performance, stability, and fault tolerance.
To learn more about CometBFT 0.34 release overview, refer to our changelog.
The migration to CometBFT was challenging as we aimed to ensure that v0.34 of CometBFT can function as a compatible and efficient alternative to Tendermint Core v0.34 without any major disruptions or breaking changes. The team also had to compare and test Tendermint v0.34 with CometBFT v0.34 and rename and rebrand the code. Another challenge was compatibility testing, which involved testing Comet alone, as well as various combinations of Comet and Tendermint. Despite these challenges, the team was able to overcome them and release a stable version of CometBFT.
Challenges may also arise during the QA process, One of these challenges faced is identifying all corner cases. Testing on small transactions is one strategy used to identify potential issues, but some corner cases may still be missed. To mitigate this risk, multiple tests are run and automated testing tools are used to streamline the process
1. Test close to real-world problems Testing should simulate real-world scenarios to ensure the software is robust and reliable.
2. Comprehensive Thorough testing, including functional, non-functional, and performance aspects, is crucial for identifying and resolving issues during development.
3. Customer feedback Involving customers in the testing process is crucial to ensure the software meets their needs and requirements. Customers like Application Framework Developers, Application Developers, Consensus Engine Developers, and Integrators are crucial for the success of CometBFT.
4. Different block sizes Testing with small, large, and empty blocks helps to identify performance issues with different block sizes.
5. Learn from customers’ tests Learning from customer tests can identify issues that may arise during the development process and ensure the software meets customer needs.
6. Simulate the app in Comet Simulating the application in Comet helps identify issues that may arise when the software is deployed in a real-world environment.
To learn more about the QA process in v0.34, be sure to check out our comprehensive QA report
As CometBFT continues to expand its capabilities and serve more use cases in production, the role of Quality Assurance remains critical. In the future, we can expect QA to place a greater emphasis on comprehensive testing, with increased attention given to uncovering corner cases by expanding the scope of testing.
Automation tools will also become more prevalent in the future of CometBFT's QA process, reducing the need for manual intervention and allowing for faster and more efficient testing. This, in turn, will reduce the risk of human error and improve the accuracy of test results.
Pre-release quality will be a major focus for CometBFT, as the platform strives to deliver high-quality software to its users. Rigorous testing will be conducted before each release to identify and resolve any issues or bugs before making the software available to users.
Graphical representations of test results will also become more common in the future of CometBFT's QA process, allowing developers to quickly interpret data and identify any issues or anomalies that may arise during testing.
While CometBFT's performance has remained consistent between versions 0.34 and 0.37, it is important to run statistical tests multiple times to ensure the accuracy and validity of the results. By continuously evolving its testing processes, CometBFT can ensure that it delivers high-quality software to its users as the platform continues to grow and evolve.