Published on March 17, 2016
1. Assessing Quality in Cyber Risk Forecasting Jack Freund, PhD CISSP, CISM, CISA, CRISC
2. Building Mature Risk Forecasting Gather SME Estimates Develop Threat Library Acquire Data Analyze Forecast Accuracy Data Coding & Clean Up Change Mgmt 1 2 3 4 567 Standardize Risk Language How do I ensure my risk assessments are accurate?
3. Forecast Maturity Scale IndividualTeamOrg Estimates OSINT Actual Maturity increases as more groups participate in table creation The more data available to influence the tables the better High degree of unmitigated Bias Best Forecast Maturity
4. Forecast •Positive connotation •Language allows for variability Prediction •Connotes precision •“Fortune Telling” •Avoid usage Standardize Risk Language 1Let’s talk about prediction
5. RISK Loss Event Frequency (LEF) Loss Magnitude (LM) Threat Event Frequency (TEF) Vulnerability Primary Loss Secondary Risk Contact Frequency Probability Action (PoA) Threat Capability (TCap) Resistance Strength (RS) Secondary Loss Event Frequency Secondary Loss Magnitude FAIR model for IT Value At Risk (VaR) Productivity Loss – Loss that results from an operational inability to deliver products or services Response Costs – Loss associated with the costs of managing an event Replacement Costs – Loss that results from an organization having to replace capital assets Competitive Advantage Loss – Losses resulting from intellectual property or other key competitive differentiators that are compromised or damaged Fines and judgments – Fines or judgments levied against the organization through civil, criminal, or contractual actions Reputation Damage – Loss resulting from an external stakeholder perspective that an organization’s value has decreased and/or that its liability has increased Risk – The probable frequency and probable magnitude of future loss Loss Event Frequency – The Frequency, within a given timeframe, that loss is expected to occur Threat Event Frequency – The Frequency, within a given timeframe, that threat agents are expected to act in a manner that could result in loss Vulnerability – The probability that a threat event will become a loss event Threat Capacity – The level of force a threat agent is able to apply Resistance Strength – A measure of how difficult it is for a threat actor to inflict harm (a.k.a. - Difficulty) Secondary Lose Event Frequency – The percentage of time that secondary stakeholders are likely to react negatively to an event
6. RISK Loss Event Frequency (LEF) Loss Magnitude (LM) Threat Event Frequency (TEF) Vulnerability Primary Loss Secondary Risk Contact Frequency Probability Action (PoA) Threat Capability (TCap) Resistance Strength (RS) Secondary Loss Event Frequency Secondary Loss Magnitude Random Regular International Value Level of Effort Risk Skills • Knowledge • Experience Resources • Time • Materials FAIR model for IT Value At Risk (VaR) Variable Sub-Variable Definition Threat Event Frequency The frequency, within a given timeframe, that threat agents are expected to act in a manner that could result in loss Contact Frequency How often the threat agent comes in contact with an asset – characterized as random, regular, or intentional (targeted) Probability of Action Likelihood of threat agents to act in a manner that could result in loss (a function of the value of the asset under attack, level of effort necessary for an attack, and the attacker’s perception of their personal risk Threat Capability The level of force a threat agent is able to apply. Includes considerations of skills and resources.
7. Threat Agent Library (TAL) Threat Community (TCom) Definition Nation States State sponsored professional groups that are engaged in espionage and either clandestine or overt action. Cyber Criminals A generic term for any group of criminal enterprises or loosely organized criminals. They are reasonably well-funded but not as well as a nation state. Privileged Insiders (Malicious) People inside your organization with specific access levels, knowledge, or some other privilege for which they do not need to overcome any controls to cause harm. Also people in which the organization has placed trust such that if they wanted to do some harm, they could. • Malicious – Those whom intend their actions to cause harm • Error – Those who make mistakes that affect security Privileged Insiders (Errors) Non-Privileged Insiders (Malicious) Everyone inside the organization who isn’t privileged. These are the people who have to overcome some form of resistive control in order to affect harm. Hacktivists/ Eco-Terrorists Generic term for those that are interested in embarrassing and making moral, disciplined, or some other conscientious argument expressed through some cyber means. Casey, T. (2007). Threat Agent Library Helps Identify Information Security Risks. Retrieved from https://communities.intel.com/servlet/JiveServlet/previewBody/1151-102-1-1111/Threat%20Agent%20Library_07-2202w.pdf Develop Threat Library 2
8. Example Threat Profile for Cyber Criminals Motive • Financial Primary Intent • Monetize proceeds of successful attacks Sponsorship •Occasionally the beneficiary of state-sponsored intelligence (e.g. Russia FSB / SVR / GRU, Chinese government, Korean government) General Targets • Financial instruments and their issuing institutions Specific Targets • Financial Services institutions • Retailers Preferred Targets • Liquid asset accounts • Payment cards Concern for Collateral Damage • Limited Capability • High degree of technological skill • Very high degree of social Personal Risk Tolerance • Moderate to High
9. How to Measure Anything Beta-PERT Distribution High Confidence Normal Confidence Low Confidence Facilitated Workshop Help attendees to quantify measurements with the following questions: • Is it more than a million? • Is that the most it could be? • Is that the least it could be? • How confident are you? • Test ranges using equivalent bet Gather SME Estimates 3 Hubbard, D. (2014). How to Measure Anything. (3rd ed.). Hoboken, New Jersey: Wiley.
10. Not Normal Beta-PERT works for many risk scenarios but is not without flaws So-called “long tail” or “Power-Law” distributions model scale-free networks better This is particularly useful for modeling risk events that happen very infrequently, but have a very high impact (not every breach is a Target, in fact, the vast majority are not) However, a perfect fit is often not necessary for what we are doing (striving for accuracy not precision) Loss Exposure LossFrequency Beta-PERT can approximate this by having close Min and Mode values, but is not best fit
11. OSINT Data Sources SEC 10K and 8K filings NetDiligence Cyber Claims Study Verizon DBIR Verizon DBIR Privacy Rights Clearinghouse DataLossDB.org Ponemon (ignore $ per record) Frequency Magnitude Acquire Data 4
12. SEC Reports – Target 10K https://corporate.target.com/annual-reports/pdf-viewer-2013?cover=6725&parts=6727
13. Planning & Direction Collection Processing Analysis and Production Dissemination Threat Capability and Frequency Threat Objectives and Goals Attack Resource and Capability Requirements Risk Analysis Model Target Vulnerabilities & Attack Consequences Highest Risk Attack Scenarios Detailed Attack Plans Intelligence Assessment & Surveillance Priorities NIST SP 800-61 Rev. 2 Krizan, L., H. (1999). Intelligence Essentials for Everyone. Joint Military Intelligence College Willis, H. (2005). Using Risk Analysis to Inform Intelligence Analysis. RAND Post-Incident Activity Containment, Eradication, & Recovery Detection & Analysis Preparation Threat-Incident-Risk Integration Data Incident Response Lifecycle Threat Intelligence Cycle Intelligence Information
14. Data Coding and Cleanup Phase 1 - Narrative Review narrative and assess “flavor” of incident Phase 2 - Coding Review initial coding and align to standard risk language Phase 3 - Alignment Align matching incidents to tables; discard the rest 2nd Pass Data 1st Pass Data Coding & Clean Up 5
15. VERIS Categorization Agent Attribute Confidentiality Integrity Parkerian Hexad Threat Agent Motivation Capability Availability Impact Productivity Response Replacement Fines/Judgements Increased Operational Expense Competative Advantage Reputation ActionAsset Information/Transaction, Amount Type (Server, User, Device…) Vectors/Paths Type of Action Categories Categories (Hacking, Errors…) Controls Response Detection Prevention Ownership/Location/Mgmt
16. Data Coding and Cleanup An employee of Morgan Stanley stole customer information on 350,000 clients including account numbers... BSF (Businesses Financial and Insurance Services) includes health insurance; may or may not be relevant to FinSrv
17. Analyze Forecast Accuracy 6 Threat Community Threat Event Frequency Min Mode Max Nation States 0.2 0.5 1.0 Cyber Criminals 0.5 1.0 4.0 Privileged Insiders 0.05 0.1 0.2 Non-Privileged Insiders 0.2 0.5 1.0 0 0.5 1 1.5 2 2.5 Nation State Cyber Criminal Privileged Insider Non-Privileged Insider Threat Event Frequency Forecast Accuracy (Mode) Create an incident profile for competitors individually. Compare central values across competitors to your SME estimates Create a similar profile for other companies in full data set Compute variance from your data
18. ▲230% ▲52% ▼44% ▼68% Nation State Cyber Criminal Privileged Insider Non-Privileged Insider Competitor Mode Comparisons Variance Charts can be used to show how much under or over forecasting was done in the prior periods These charts can give credence to requests to adjust risk ratings Make periodic reviews/refreshes of forecast accuracies; out-of-band changes as needed Visualizing Forecast Accuracy for Mgmt 1.17 1.77 ▲800% 0.42 Nation State Cyber Criminal Privileged Insider Non-Privileged Insider Non-Competitor Mode Comparisons
19. Analysis Risk Impact Projections New Ratings Consensus Gathering 4 3 1 2 Change Mgmt Change Planning Implementation Evaluation Analysis Communication Forecast Accuracy Change Mgmt7
20. Building Mature Risk Forecasting Gather SME Estimates Develop Threat Library Acquire Data Analyze Forecast Accuracy Data Coding & Clean Up Change Mgmt 1 2 3 4 567 Standardize Risk Language How do I ensure my risk assessments are accurate?
The tool provides users with a systematic and repeatable approach to assessing the security posture of their cyber systems and ... risk assessments, ...
Assessing Credit Risk. 2 Objectives ... loss forecasting, and ... zResidual risk the degree to which the quality of risk
5 Steps to Cyber-Security Risk Assessment Considering the number of botnets, malware, worms and hackers faced every day, organizations need a ...
Probabilistic risk assessment (PRA) ... Cost risk; Reference class forecasting; Risk assessment; Risk Matrix; Extreme risk; Risk management tools;
A Hurricane forecast ... Use the Risk Assessment Tool complete your risk ... (Chemical, Biological, Radiological, Nuclear, Explosives), Arson, Cyber ...
... (IT) issue. In fact, cyber risks ... only 25 percent say that the quality of the information they ... Connecting the dots: ...
Assessing Research Quality. Related Information. Quantitative Research Assessment Tool (pdf) Qualitative Research Assessment Tool (pdf) The quality of ...
Risk assessment is the determination of ... measure of a location's quality of ... its Environment and Assessing the Risks of ...
... club/?book=3319235699Download Cyber-Risk ... Assessing and Managing Earthquake Risk: ... Product Development and Quality Control ...
Managing Risk To Avoid Supply-Chain Breakdown ... low-likelihood risks. For instance, a supplier with quality problems rep- ... forecast risk, ...