IOSG Ventures: A comprehensive interpretation of the community-driven bounty and audit market

Original author: Ray, IOSG Ventures

Preface

As a large-scale computer system, the current system complexity of the blockchain has far exceeded the level of 5 years ago, the degree of infrastructure modularization is more refined, the smart contract logic of the application layer is becoming more and more abundant, and the interaction between contracts is very frequent , more importantly, the number of assets managed by the blockchain system is already very large, so there have been more discussions about the security cycle in the blockchain security community recently (the situation is the same as in 2017, when people think of security, they only think of developers It is very different to write the contract and throw it to the friends of the Ethereum Foundation to take a look and do some basic tests).

Throughout the security life cycle of blockchain programs (from testing, inviting third-party audits to post-event monitoring, update audits), the bug bounty community is like a safety cushion to attract white hats to the blockchain through game theory and cluster work. The code of the project party will be reviewed for the last time, and some smart contract security workers feel that the bug bounty is more like the last person in the line of defense, but I think the bug bounty and audit competitions have the potential to play a greater role in the future, serving as a system that runs through the entire The security lifecycle role improves the overall security of the system.

Of course, there are also bug bounty programs (Bug Bounty or Vulnerabilty Rewards) in the field of traditional network security. First, major technology companies such as Facebook, Google, Microsoft, etc. will deploy bounty programs for their own inhouse security teams and their own product lines. Secondly, Bug bounty third-party platforms represented by HackerOne and Bugcrowd have emerged since around 2015. At present, these two leading security companies rely on the distribution of bounty commissions as their main income, and their annual income can reach nearly 50 million US dollars and 20 million respectively. dollars. In the blockchain world, bounty is a more interesting topic that is often discussed in the security circle. The main reason is that the open source of blockchain code actually makes the cost of hacking and upgrading attack strategies cheaper. In addition, the crypto world strongly advocates clustering Work, Creator and Ownership Economies open to contribution models that make a more open white hat economy even more valuable.

What are bug bounties and audit contests? Why do we need them?

Security is a dynamic game between the attacker and the defender, just as computer security expert and cryptographer Bruce Schneier said, "Security is a process, not a product. It is a way of thinking that must run through the software development process in every aspect.” In the blockchain world, a dark forest where all codes are open source and transparent, a blockchain project that wants to survive for a long time must have an eternal demand for the security of its products/contracts. Chain products all have more or less financial attributes. The most important asset in finance is trust, and the user's trust is only once.

Where are the shortcomings and problems of traditional auditing? What advantages can community-driven bug bounties and audit competitions have to make up for these problems?

Developers using auditing services often find that:

• Even after purchasing the services of a third-party auditing company, there are still problems with the code after auditing. Although the reasons for these problems are different (technical and non-technical), it does not seem to be completely reliable to rely on an auditing company in the end. However, the quality of code audit still depends on the level of the auditor, and customers often lack the ability to discern "who is better".
• The bounty platform and audit competition are a more open "sandbox", and the project code can be reviewed by white hats at will, regardless of background (there may be personnel from professional audit companies, and there may be freelance security analysts), the arsenal is unlimited, and all customers have to do is set a reasonable bounty and pay their contribution when the white hat finds a problem.
• Usually customers will first submit their code that needs to be reviewed by the white hat, define the security level of the vulnerability (usually related to the possible economic loss, the easier the vulnerability that directly causes economic loss, the higher the severity level), bounty budget, testing code scope and even test steps.

How big is the market?

The business model of bounty platforms and audit competitions is usually to draw a portion of the bounty paid by customers or the total bonus pool set up as the service fee of the platform. Customers (project parties) who need code security audits will announce their plans on the bounty platform according to their own needs (which codes need to be audited, how to define the severity of vulnerabilities, and how much reward they are willing to pay), and white hats Vulnerabilities will be found according to the needs of the project side. Once the loopholes are found by the white hats and meet the needs of the project side, the bounty will be distributed to the white hats, and the bounty platform will draw a commission from it as a service fee.

In the field of Web2 traditional network security, the bug bounty platform is also a relatively young direction (appeared after 2012), and currently the largest bug bounty platforms are HackerOne and Bugcrowd. In 2022, HackerOne's annual revenue will reach 58 million US dollars, the company's valuation will reach about 500 million US dollars, and the cumulative bounty paid in history will be 230 million US dollars (150 million US dollars in 2021 and 2022). software vulnerabilities, has more than 1 million registered hackers, and more than 1,000 customers who use HackerOne services every month. Its competitor, Bugcrowd, is expected to make more than $20 million in 2022.

In the field of Web3 security, in 2022, all web3 bug bounty and audit competition platforms will issue a total of 50 million US dollars in bounties to white hat hackers, and the average charging level of such platforms is around 10% to 30%, so it is conservatively estimated The current market size is around $5 m~$15 m, and it is still a very emerging market.

Another interesting thing is that more and more customers are willing to directly use the code auditing services provided by this decentralized security community. The most famous example is that Opensea did not directly find the second-tier auditing service before launching their new platform Seaport. Instead, the third-party audit company chose Code 4 Rena, the largest decentralized audit competition platform at present, and set up a prize pool of 1 million US dollars. Today, the traditional security audit market is increasingly involved (volume human resources, volume technical tools, volume Market BD), will decentralized security services be an important growth in this market? (Currently there are 56 auditing companies in the market, and the revenue of the leading companies in the past year was between US$10 million and US$40 million. I think there is a lot of room for imagination in the decentralized security market).

Bug Bounty Platforms vs Audit Contest Platforms

Although the bug bounty platform has a ten-year development history in web2, the audit competition platform is a new thing in web3 native. The object of the audit competition service is those project parties who are about to launch products or some new functions, and use the power of the decentralized community to help them complete the audit service within a specific time (more than 2 weeks). From this perspective, the audit competition will It will bring no small business threat to traditional audit companies.

Below I will show the differences between the two platforms in terms of participation methods, reward structure, and test coverage:

way of participation

Bug bounty platforms such as Immunefi are usually open projects where anyone can participate at any time. Participants typically independently explore and report vulnerabilities in exchange for rewards. If two people find the same repeated vulnerability, the first-come-first-served principle will be followed, and whoever submits the report first will get the reward first.

Community-driven audit competition platforms (eg Code 4 rena, Sherlock) are often time-limited and compete with participants to find and report vulnerabilities within a certain time frame. Compared with the bounty platform, there will be some teamwork (for example, each project will have a clear assignment of Lead Senior Auditor and Lead Judge, and finally review and summarize all audit results into an audit report to the customer, and these two leaders also follow The principle of decentralization of community elections and competitions). In addition, if two audit competitors find repeated loopholes within the specified time, both of them can get rewards.

reward structure

The actual rewards issued by both will mainly consider the severity of the discovered vulnerability.

The only difference is that a community-driven audit competition platform like Code 4 Rena will have a fixed portion (5% ~ 10%) of the bonus pool for each project allocated to Lead Senior Auditor and Lead Judge, because they actually undertake traditional audit company projects The role of the person in charge.

Another interesting point is that the project party on the bug bounty platform sometimes places project tokens as rewards, but I have also seen that some white hat hackers in the community prefer to get stable coins like USDC and USDT instead of price fluctuations. Project tokens.

scope and focus

Bug bounty platform projects usually have a broad scope, while projects on audit competitions usually have a more focused scope, targeting a specific function or aspect of the software, while requiring white hats to focus on completing the work in a shorter period of time

Projects focused on auditing competitions

Code 4 Rena - An esports-like community-driven audit competition platform

Code 4 Rena has three character types:

1. Auditors (Wardens) review the code. Anyone from a professional security engineer to a novice developer trying to gain more experience can register as an auditor to participate in the public auditing competition.

2. Judges are usually the best engineers in the C 4 community. They determine the severity, effectiveness, and quality of vulnerabilities and evaluate audit performance.

3. Sponsors are project parties, such as Opensea, Blur, ENS, Chainlink, etc. They create bonus pools to attract auditors to audit the code of their projects. Sponsors also have the option to host private, invitation-only competitions for added privacy.

One of the most interesting points is the culture that Code 4 Rena is building: collaboration and teamwork are encouraged. Unlike traditional bug bounty programs, Code 4 Rena pays all auditors who report a valid vulnerability even if the vulnerability has already been reported. This encourages healthy competition among auditors as they are motivated to find high-severity and common vulnerabilities. On this platform, some auditors will form temporary teams to find loopholes together.

business model:

Any project can go to Code 4 rena to start an audit competition program and provide USDC or ETH to set up a basic prize pool (usually the size of the prize pool is $40,000 ~ $100,000), Code 4 rena will charge 20% from the basic prize pool as a platform Service income for organizing competitions, providing reviews, and sorting out audit output reports. The project party can also provide project tokens on top of the basic prize pool to set up an additional prize pool, and Code 4 rena will charge 40% of this additional prize pool.

Sherlock - Community driven auditing with smart contract insurance

Similar to Code 4 rena, Sherlock also has roles such as auditors, sponsors, and judges. The uniqueness of Sherlock lies in the insurance services provided by the platform. Anyone can invest in the insurance pool on the Sherlock platform. Investors deposit USDC into the insurance pool, and agreement customers can purchase services to hedge the risk of smart contracts being hacked. The sources of income for insurance investors include: premiums paid by agreement customers + interest earned by depositing insurance pool funds in other DeFi pools (Aave, Compound, etc.) + Sherlock token incentives. But the investor bears the risk of repaying the policy while reaping the benefits.

Another point that is different from Code 4 rena is the distribution mechanism of audit service income provided by the platform. Compared with Code 4 rena, Sherlock has rules that allow the chief senior security auditor and the chief judge to get a fixed amount (5% ~ 10%) from the bonus pool to properly compensate and motivate the full-time senior auditor. In addition, there are systems of selection and competition for the selection of leadership roles.

How to build a hacker community? What is the biggest concern of Web3 white hats?

After we observe different decentralized security communities (ImmuneFi, Hats Finance, Code 4 Rena, Sherlock, etc.) and chat with some security entrepreneurs, we think that what all decentralized platforms are committed to doing is: to build a more A healthy and efficient communication and collaboration platform. The bounty platform is like a marketplace between Hackers and projects. They must consider their needs from the perspective of hackers (as shown in the table below), and at the same time consider the best Concerned about (audit quality).

Source: 《Bug Hunters’ Perspectives on the Challenges and Benefits of the Bug Bounty Eco》

In addition to some common needs, I also saw some interesting topics in the Immunefi white hat community (the most lively white hat discord community I saw).

for example:

A white hat named Rappie wants to disclose some project loopholes he/she discovered before, and asks what community rules need to be followed. (1. Only disclose bugs that have been fixed. 2. Make sure that any public information has no negative impact on the protocol or its users. Keep confidential information eg: after they fix your SQL injection vulnerability, don't release information about their full database. 3. Make sure that you need to send a private message to the project team before making it public).

A white hat named Noam Yakov has doubts about the definition of a bounty project (this often happens, because usually only serious security vulnerabilities can be rewarded. How does the project define the security level of the vulnerability? Something that the white hats care deeply about, and the community hears about such disputes a lot). In the Uniwhales bounty project, he had doubts about their definition of MEV impact as a serious security vulnerability. In the end, everyone discussed that this type of description does not apply to all MEV situations. For example, for some toxic order flows, the protocol pool can The situation of asset draining is definitely a serious security incident (so it is often not enough to define a set of security level frameworks, and usually a similar role of arbitrator in the platform is required to intervene in different actual cases).

And for a very interesting topic, "What are your demands and expectations for a bounty platform like Immunefi?" A white hat named ckksec gave his answer: 1) Helping these anonymous encrypted white hats earn their labor income Do some legal clarifications like invoicing. 2) The platform should not only have a scoring system for white hats, but also score the quality of the project because white hats often need to spend time distinguishing the quality of the project. 3) For white hats who are willing to open their proflie, the platform can show their workflow. At the same time, it is better for the platform to more transparently display the security analysis report information received by the project party.

What tools can help white hats?

With the fire of LLMs GPT, I have recently heard people frequently discussing whether security audits can also be replaced by AI. The experienced security practitioners I have talked to generally believe that GPT is difficult to directly replace human intelligence. Some low hanging fruit (problems that are easy to find) may be detected by language models, but those problems with medium and high risks still require expert participation. For example, according to the feedback of a senior security expert, for similar data analysis and dynamic analysis, these more complex tests need to be artificially combined with the actual business logic of the protocol to conduct security analysis tests in advance and define the expected target attributes of the test in advance. The most difficult part is to write a good properties and define the correct test field. According to their experiments on GPT, they believe that GPT cannot completely replace humans at this stage.

Of course, there are currently more optimistic results showing that LLM can greatly improve the analysis efficiency of security analysis tools and reduce the false positive rate.

Let's think about this topic from another interesting non-technical perspective. It is a dynamic game between security attackers and defenders. The magic height is one foot higher than the other. Will AI bring relative security to the security attackers? help?

Safety is people-oriented

People will habitually think that software is a cold, mechanical, and logical thing, and improving system security only needs to improve analysis technology and system defense level. However, people lack of thinking about security issues from the perspective of economic incentives and human nature. In the dark forest of open source code, we need a distribution system that is more in line with the assumptions of rational people. Positive and benign economic incentives attract more people who are willing to invest in the blockchain for a long time. People who contribute wisdom to system security join.

The current traditional security audit market structure is stable, and brand reputation is the most important intangible asset of companies in this field. Over time, the influence of top security brands and the trust of customers have steadily increased, but traditional security audits also have their own problems (the business model relies solely on manpower and it is difficult to grow in scale, and leading companies need to balance growth and audit quality. Some companies have encountered such a bottleneck and even affected the value of the brand).

The community-driven security audit competition is an innovative business model. At present, more than 300 customers of the two platforms have gradually found PMF, and the bounty platform is a good supplement to the security life cycle. Although these decentralized platforms are still We have not found a particularly effective token model, but we are very optimistic about the large-scale growth of this market in the future (because the wisdom of the crowd is very suitable for the offensive and defensive game scenarios in the security market).

Will community-driven audit platforms pose a threat to centralized audit firms? We think they will have a healthy mutual competition and complementary relationship. In the short term, when a platform like Code 4 rena forms a certain network effect and has a good track record (the proportion of audited projects being hacked is low), it may indeed give Some centralized companies in the middle and tail will bring certain competitive pressures, but in the long run, this may also force the centralized audit platform to form some commercial cooperation with the community-driven platform, because this can also broaden the customer base of the centralized security audit platform and Improve the audit quality (a bit like the original security bounty project operated independently by a large web2 company and later formed a cooperation logic with third-party platforms such as HackerOne).

Although the direction of the community-driven security platform is to be more DAO-oriented (Forta can actually be included in this category), in the actual operation of the current project, there are still problems such as: how to make the workflow and economic distribution process more transparent and open , How to weigh the privacy and security considerations of the project party, how to more clearly define the relationship between teamwork and personal contribution, how to solve the problem in a relatively fair and professional perspective when conflicts of interest arise, etc. These are the things that security DAOs need to face Right challenge.

Reference:

1. 《HackerOne Year Book》
2. 《Bounty Everything - Hackers and the Making of the Global Bug Marketplace》
3. 《An empirical study of vulnerability rewards programs》
4. 《The 2022 Hacker Report》
5. 《Productivity and Patterns of Activity in Bug Bounty Programs》