For over 12 years, I've been organizing and running hackathons to find and fix security vulnerabilities before products reach the market. These events play a vital role in the product development lifecycle, increasing team collaboration, uncovering problems, and driving innovation. In this article, I would like to share some of my top insights and tips to help organizations create or refine their security hackathon playbooks.
Why hold a security hackathon?
Hackathon events bring together product and security experts for the sole purpose of discovering security vulnerabilities within products. Security experts provide security thinking and knowledge about how to subvert and hack systems, while product experts provide knowledge about the inner workings of a particular target product.
These events are traditionally short-lived (lasting only a few weeks), but to make them successful, the scope of the evaluation must be clearly defined. This may be an architecture or design review, pre-silicon or post-silicon evaluation, looking at an entire platform or device, or specific aspects or features of a product.
There are traditionally four main goals or objectives when conducting a hackathon.
The first is to serve as a supplementary security assessment. The second is to build communities of practice that transfer knowledge among participants. The third is a systematic assessment of the quality of security features and Security Development Lifecycle (SDL) execution by the product team. And fourth, reduce costs by developing and training an experienced internal security research team that can reduce reliance on costly external consultants.
How do you hold a hackathon?
Here are some tips to consider when creating your playbook.
1. Avoid overlooking important details in the planning process – Planning is important before a hackathon. This includes securing commitment from both the participating product and security teams, assigning roles, and understanding where in the product or project lifecycle a hackathon makes sense.
It's also important to check if you have key talent in various areas such as product architecture, design, and security research. The goal of planning is to have a product that is mature enough for security evaluation and testing, but also allows enough time to make significant changes if necessary.
Equally important is having the space, equipment, tools, and infrastructure in place to support assessment activities. Finally, make sure all participants are familiar with the hackathon methodology and process before the event begins.
2. Define the role of the hackathon – These events define several roles that the planning team must fulfill before kickoff.
- Two leaders (we recommend one from the functional team and one from the security team) responsible for planning the event. These individuals can also fill additional roles as needed.
- Security Lead responsible for analyzing security architecture and identifying priority security areas.
- Enabler responsible for setting up the test environment.
- A facilitator who is responsible for guiding the team's activities during an event.
- Security expert. Depending on the product being evaluated, one or more people will be responsible for hardware, firmware, software, network protocols, etc.
- A functional expert is a member of a product team who is familiar with the functional components of a product.
- Resources available on-call, including architects, developers, validators, and infrastructure experts.
3. Start your security team with a clean slate – Understanding the methodology and process is different from knowing the actual details of the product being evaluated.
Product knowledge is not required for security researchers, but for complex products and technologies, it is highly recommended that security teams understand as much as possible about the product's architecture and design. This may mean spending a little more time on the architecture and design training phase (this may mean spending a little more time on the architecture and design training phase (this may be the case in one of his large-scale events, which also includes the security assessment and testing phase, and the final day wrap-up phase). phase).
It's even more helpful to understand what has been tested by the SDL team. Once your security team or subteam is familiar with your product's architecture, you can begin your security assessment by brainstorming a list of attack scenarios and test cases and applying them.
4. Prioritizing evaluation targets – These events are time-limited, so targets should be prioritized based on highest business risk. This could be an area that has been less evaluated by the SDL team, an area that could have a significant security impact, or an end-to-end platform solution that has not been evaluated by his SDL at the IP level or component level. There is a gender.
5. Require daily sync – To ensure that the entire hackathon team is making positive progress, daily synchronization should be mandated. During this synchronization time, sub-teams report on their status, what tests they have performed on the product, what vulnerabilities have been discovered, and any ongoing testing in their assigned product areas. We will explain what kind of test plan we have for this purpose.
Sub-teams should be encouraged not to create exploits for discovered vulnerabilities and to continue exploring other attack vectors if they find themselves in a rat hole with a test case that is not progressing well. there is.
6. Take the time to document your evaluation efforts – These events go by quickly, so it's important to keep track of your team's progress. We encourage participants to frequently note down small things and details in order to consolidate them at the end of the event.
Half of the final day should be set aside for participants to document and report on the security assessment work done. These reports can be created by individual participants or sub-teams. Its purpose is to ensure that security analysis conducted during the event (tests performed, tools used, etc.) is collected and incorporated into the final hackathon report, which can be used for future security assessments. .
7. Follow up the item and finish the loop – Not all items will be completed during the hackathon. For example, an activity might require more test cases or inquiries as part of brainstorming. Make sure these outstanding items are assigned to owners and their activities are completed. This is essential for the final report.
8. Assess vulnerabilities – Identified security vulnerabilities should be reviewed and graded for severity using CVSS scoring.
Both product team members and security researchers should aim to achieve an agreed severity rating for all issues found. Only issues with known security impact and exploitability should be evaluated as security vulnerabilities (other issues should be evaluated as security vulnerabilities with recommendations for security hardening or defense-in-depth, or as unresolved issues that require further investigation after an event). (can be recorded as sighting information).
9. Create a final report of findings – After the hackathon, team members must submit their findings to the product leader as a final report. This includes details about all attack scenarios performed, vulnerabilities discovered, and recommendations for remediation. His two main expected outcomes include identified issues and security recommendations, and long-term implications for security improvements, such as gaps in threat models and SDL execution quality.
10. Evaluate the hackathon process for improvement – Running an effective event is a complex process and there is always room for improvement. We will review not only the process but also the survey results. This can be used to drive systematic improvements to a product's security architecture, design, and evaluation process. This may include deploying security tools by the product team or adding security measures to address different types of vulnerabilities identified during the hackathon.
When properly organized and executed, hackathons can be fun, engaging, and foster innovation. As you work to launch or improve your hackathon event, consider some of the key insights listed above.
