Published on March 12, 2014
07/03/14 1 Risk Based QA Michael Agerkvist Petersen firstname.lastname@example.org dk.linkedin.com/in/michaelagerkvist Michael Agerkvist Petersen • QA @ Radiometer Medical • 18+ years with Medical Devices – HW development – SW development – Project Management – Process Improvement – QA • Owner of MDCA (Medical Device Compliance Assistance) – Spare Qme job J
07/03/14 2 Risk Based QA • Based on my experience from: – Making a class C Medical Device at Novo Nordisk – Working with Encrypted Pin Pads (ATM keyboards) at Cryptera – Other projects • This will not be a complete introducQon to (safety) risk management or Risk Based TesQng. Agenda • IntroducQon • Risk Based Quality Assurance – Regulatory Risks – Process Rigor – Risk Based DocumentaQon Rigor – Risk Based TesQng – ProacQve QA • Examples • PiZalls
07/03/14 3 IntroducQon Some deﬁniQons • Quality: The totality of features and characterisQcs of a product or service that bears its ability to saQsfy stated or implied needs [ISO] – There are many diﬀerent customers: Users, OrganizaQon, Regulatory…. • QA: Quality Assurance – many ways to implement this in pracQce – Ensuring that development is in compliance (with deﬁned process) – Doing the actual tesQng – “anything” which helps ensuring Quality to the diﬀerent customers
07/03/14 4 Soeware today • Today soeware controls more and more in our daily life. • Soeware failures may negaQvely aﬀect business, human health or even human life • ResulQng in increasing public and regulatory demands for befer products and even more Qme pressure. Regulatory Compliance • More and more industries gets regulated • Some are more regulated than others Accident Injury or other loss Public reacQon New laws & regulaQons Regulated Industry Nuclear Flight Medical Device Military Payment systems Public Systems • The bar is raised over Qme [Not to scale]
07/03/14 5 TesQng Paradox • TesQng is a structured approach to reduce the number of defects it is the last acQvity before release. So it is oeen the ﬁrst to be sacriﬁced • TesQng is always a sample. You can never test everything, and you can always ﬁnd more to test. • A good test case is one ﬁnding a defect – running a complete test suite without ﬁnding a single defects will not add much value. • The problem with most systemaQc test methods, like black box methods (equivalence parQQoning, boundary value analysis , cause-‐eﬀect graphing, etc.), is that they generate too many test cases, many of them will never ﬁnd a defect. • However – in a regulated world it adds value to run tests which does not ﬁnd defects – a “clean sheet” is needed to make a submission without too many quesQons Risk Based Approach • A way to lessen the the work load. Could be to test more in: – bad areas of the product. – the most important funcQonal areas and product properQes. • And tesQng less in the other areas… • But, how to: – ﬁnd the right areas? – PrioriQzing other QA acQviQes?
07/03/14 6 Risk Based Quality Assurance TradiQonal QA • Consist of: – Review – Dynamic Test – StaQc analysis – Templates – Source Code standards – Traceability analysis – Checklists – Etc • They are typically: – Time consuming – Passive – Always compromised due to Schedule pressure
07/03/14 7 Balancing QA Too li-le • Safety risk • Poor reliability • AddiQonal cost • Project delay • Regulatory failure • Customer complaints Too much • AddiQonal cost • Project delay • Under veriﬁcaQon (in the more important areas) • Increased maintenance Result of poor QA Liability and liQgaQon Recalls Loss of company image In-‐market updates ReducQon in performance No regulatory approval Project delay Death Major Injury Minor Injury Security violaQon ReducQon in performance Loss of data Business cost Customer cost Severity High Low
07/03/14 8 Risk Based QA Approach • Some QA acQviQes will sQll be Passive – They are well known and needed • Some QA acQviQes will be more ProacQve – We know we are going to release with defects -‐ so why not try to miQgate the impact of speciﬁc types of defects in the design? • All QA acQviQes will be prioriQzed – Some defects or lack of process/documentaQon will do more harm than others. So why use the same eﬀort and acQviQes on every feature or enQty? • Use the risk based QA approach to – Find the most criQcal defects as early as possible at the lowest eﬀort/ cost – Find the most criQcal classes of defects and miQgate the impact of them in the design – Balance the rigor of process, documentaQon and design Some risk deﬁniQons • Harm: Physical injury or damage to the health of people, or damage to property or the environment. – From project delay (ﬁnancial loss) to death (e.g. of user) • Hazard: PotenQal source of harm • Probability: Of a harm to occur. – Could correspond to the frequency of funcQonality usage by the user. • Severity: Measure of the possible consequence of a hazard – Customer cost: Loss of data -‐> Security violaQon -‐> Injury-‐ >Death – Business: Project delay-‐>Recalls-‐>Liability&liQgaQon • Risk = Severity x Probability [IEC 14971]
07/03/14 9 Probability • Probability of failure – Usage frequency (funcQons used several Qmes a day vs once in lifeQme) – For Medical Device Soeware, probability for SW defects = 100% -‐ but the probability of Harm needs to take probabiliQes from the enQre chain of events Probability Levels Probability of Harm Probability of Harm Descrip@on Ra@ng DeﬁniQon 5 Frequent Constantly present 4 Probable Have been/ will be reported but no more than once per month 3 Occasional Have been/ will be reported but no more than once per year 2 Remote Have been/ will be reported but no more than once in products lifeQme (~10 years) 1 Improbable Considered unlikely to occur This is not science… And very diﬃcult in real life… So do spend too much Qme ﬁnding the right value
07/03/14 10 Severity Levels Severity of poten@al Harm Harm descrip@on # Term Descrip@on 5 Catastrophic Results in immediate death of paQent or user • Immediate death of person caused by Explosion, Fire or Electrical shock 4 Serious Results in permanent impairment or criQcal injury that would require medical or surgical intervenQon to preclude irreversible impairment or damage • Incorrect medical treatment of paQent due to paQent data mix-‐up • Body part damage (e.g. eye or ﬁngers) 3 Moderate Results in temporary/ reversible injury or temporary/reversible impairment requiring professional medical or surgical intervenQon • Incorrect or inadequate medical treatment 2 Minor Results in temporary injury or impairment not requiring professional medical intervenQon • Minor incorrect or inadequate medical treatment • Delayed medical treatment, loss of sample/new sample required or no result • Equipment damage • Privacy violaQon (e.g. data leak) 1 Negligible Inconvenience or temporary discomfort • UnsaQsﬁed user • Loss of old data QuanQfying Severity & Probability • Amount of severity should happen by considering the diﬀerent viewpoints of the system’s stakeholders. • The probability of impact can only happen indirectly, e.g. by evaluaQon of – frequent of use, – quality indicators like the complexity of the soeware itself, – the quality of the documentaQon – etc.
07/03/14 11 QuanQfying Severity & Probability • Do not spend too much Qme determining the exact values • The relaQve scoring is the most important in order to idenQfy the most criQcal parts. • The scoring could be rather informal and based on a brainstorm Note: For Medical Device Class B and Class C it is expected that the Risk Analysis is more elaborated than stated here Risk levels • Four levels (3 should be OK) • Based on IEC 62304 safety class (A,B,C) • ClassiﬁcaQon used for: – Requirements (i.e. some funcQons are more criQcal, e.g. Risk Control Measures than others) – Structural elements both design and actual implementaQon (i.e. elements implemenQng criQcal requirements) Least criQcal Most criQcal C2 B A C1
07/03/14 12 Ploxng the risks Severity High Low Probability High Low High Risks Medium Risks Low Risks C2 C1 B A Diﬀerent risk perspecQves For a Medical Device: • Safety: Freedom of unacceptable risks. IdenQfying and miQgaQng safety risks to paQents and users. • Eﬀec@veness: Fulﬁl the medical claims, delivering value to the paQent and users. MeeQng the user’s needs. Correctness of the product, meeQng its speciﬁcaQons. • Customer sa@sfac@on: Good user experience, good service, ease of use, free of defects, reliable. • Regulatory: Being in compliance by meeQng the Regulatory ExpectaQons including: Safety, Eﬃcacy and Customer saQsfacQon. • Project: MeeQng the organisaQons expectaQons, including Quality, Safety, EﬀecQveness and Regulatory, but also Qmelines and other tradiQonal project risk related stuﬀ.
07/03/14 13 Diﬀerent risk perspecQves Safety EﬀecQveness Regulatory Customer SaQsfacQon Project risk Regulatory Risk – Process Rigor
07/03/14 14 Process Rigor Common FDA Warning lefer issues – Lack of • test speciﬁcaQons and test results • comprehensive, up-‐to-‐date speciﬁcaQons (design input) – Inadequate • fault handling and stress tesQng • change and release control Regulatory Risks versus ProducQvity & Predictability • RegulaQons and standards does not seek beneﬁts in producQvity or project predictability • But they don’t preclude producQvity and predictability from being important. • So it is your responsibility and interest to have development processes focusing on: – ProducQvity – Predictability Without sacriﬁcing Safety, Customer saQsfacQon, EﬀecQveness and Regulatory risks
07/03/14 15 Regulatory Risks versus ProducQvity & Predictability • Risk Driven Approach: – Regulatory interpretaQon – focus on the intenQon – ConQnuously process improvement within the intenQon and frame of the Regulatory expectaQons – Align process with the diﬀerent risks (e.g. Safety, EﬀecQveness, and Customer saQsfacQon) associated -‐ focus on what really mafers • “Too much will always be too much, but maybe not enough” Regulatory Risk: When not in compliance • Compliance is not created by: – using a checklist – copying from the: • standards • regulaQons • guidance's • CreaQng compliance is about meeQng the “intent” of the standards – not just following “the lefer of the law”
07/03/14 16 Regulatory Risk – Don’t climb too high E.g. Tools validaQon -‐ Balancing between: • what is required and • when value adding stops minimum opQmum Risk Based DocumentaQon Rigor
07/03/14 17 DocumentaQon Rigor • To lifle – Diﬃcult to anchor decisions – High regulatory risk – Project delay – Maintenance is diﬃcult • To much – AddiQonal cost – Less Qme for development (project delay) – Maintenance is diﬃcult – Risk of in-‐consistence – Project delay Requirements Rigor Rigor of requirementsHigh Low Safety UI User Satisfaction Service Risk Low High Efficacy
07/03/14 18 Design Rigor • 62304 allows soeware to be decomposed into soeware items with diﬀerent safety classes – if they are segregated and segregaQon raQonale provided – No raQonale => Safety Class is the same for all Item/Units. – type of segregaQon could vary based on risk and other factors Design Rigor – Item/Unit Level Higher Risk Level requires, more detailed and rigorous: • Architecture descripQon • Detailed Design If the enQre SW is Safety Class C Detailed design is required, but it is sQll possible to use the Risk Based Approach and adjust on details and rigor Least criQcal Component Most criQcal Component UI FuncQon Model Driver HAL SI C2 B C2 B A A C1 C1 Allocate Risk Level to the diﬀerent Items/Units based on their responsibility
07/03/14 19 Risk Based TesQng Risk Based TesQng • IdenQfy the top most criQcal funcQons • Consider: – Evaluate whether the users will idenQfy defects in funcQon or afribute. – Use historical data to idenQfy funcQon areas with many defects • Do extra tesQng in criQcal areas and areas with many defects – Use domain specialists – Extend (automated) regression test when new defects are found
07/03/14 20 For System TesQng Risk level SW System test ac@vi@es C2, C1 • FuncQonal • Exploratory • Consider other test strategies (Stress, Boundary, Stability, State transsion, Recovery) dependent of the FuncQon under test and Risk level B • FuncQonal • Security • Exploratory A • FuncQonal (all requirements have at least one TC) • Allocate Risk Level to the diﬀerent FuncQons/ requirements based on the possible severity and probability. Item/Unit Level tesQng Higher Risk Level requires, more detailed and rigorous QA AcQviQes: • Reviews, • TesQng, • Etc. If appropriate use historical data to idenQfy Item/Units with many defects in order to adjust the Risk Level Least criQcal Component Most criQcal Component UI FuncQon Model Driver HAL SI C2 B C2 B A A C1 C1
07/03/14 21 Item/Unit Level tesQng Risk level Unit Tes@ng ac@vi@es C2 • Formal Code Review by at least one SW Developer plus SW Risk Manager • StaQc Analysis • Soeware Unit Test, 100% decision/condiQon coverage • IntegraQon test using decisions tables and classiﬁcaQon trees C1 • Formal Code Review by at least one SW Developer • StaQc Analysis • Soeware Unit Test, 100% Statement coverage • IntegraQon test using decision tables B • Formal Code Review by at least one SW Developer • StaQc Analysis • Soeware Unit Test, 100% funcQon coverage • IntegraQon test A • Informal Peer Code Review • IntegraQon test part of System Level Test Note: Risk Level A not to be used for Class C Soeware Item/Unit Level tesQng & Complexity Risk level Complexity Reduce C2 McCabe <= 2 AND LoC < 10 Reduce Unit Test to 100% FuncQon coverage C1 McCabe <= 3 AND LoC < 20 Reduce Unit Test to 100% FuncQon coverage B McCabe <= 3 AND LoC < 30 No Unit Test necessary (Not for Class C Soeware A NA NA • Complexity – root cause for many defects – but low complexity code may also require less tesQng. • In source code use complexity metrics to adjust the QA acQviQes • Note: Complexity metrics also part of the code standard
07/03/14 22 ProacQve QA ProacQve QA • Uses same approach as for Medical Device Safety Risk Management (IEC 14971, IEC 80002). • IdenQfy most criQcal SW hazards and their causes, e.g.: – Loss of conﬁguraQon could make the SW/Device useless – Faulty data could result in fault funcQonality and/or results – Never-‐ending waiQng loops could could make the SW/ Device slow or useless • Implement proper miQgaQons in the design
07/03/14 23 Examples Example – Keyboard in ATM Keyboard Dispenser Display Card Reader Ext Keyboard XFS drv XFS drv XFS drv XFS Win XP Bank App PC ATM Master-‐key derived-‐key derived-‐key derived-‐key derived-‐key • <1% source code doing keyboard funcQonality • Remaining related to: • Security: crypt, key-‐ handling, surveillance • Service: ConﬁguraQon, status log etc. Reliability wise – key handling is very important • Without the Master key the keyboard needs to back to the manufacturer • Without derived keys the keyboard needs a service tech. visit
07/03/14 24 Risk Based QA for ATM keyboard • “Spontaneous” loss of keys in the ﬁeld • Risk based approach: New ﬁle system with – CRC Error correcQon – “Black box recorder” (log of ﬁeld events for debugging) – More rigor of requirements and design – Unit tesQng of the new ﬁle system Risk Based QA for ATM keyboard • In the pilot phase -‐ several incidences where keyboard is “completely dead” • In certain situaQons, defects in both the CRC Error correcQon and “Black box recorder” ends up in: “logging an error result in a new error…” • Learnings: – Adding “miQgaQons” in soeware increases complexity – which may end up in more erroneous SW – Remember integraQon and scenario tesQng – Consider some kind of recovery mechanism
07/03/14 25 Example ProacQve QA for VHF radio • VHF Radio stores “vital” data in EEPROM. • During SW test a HW design ﬂaw is found. HW do not give “Power Down” in Qme => “vital” data is corrupted => VHF Radio is useless. • ProacQve QA: – CRC protecQon of data – Shadowing of data – Controlled Scheme for data update – The approach also miQgates SW failures Bafery monitor in Medical Device • A Bafery powered Medical Device have a bafery monitor to inform when charging is needed • ProacQve QA – Bafery power is (also) displayed in number of measurements lee – If number of measurements lee <= 2 then measurements is not possible – When number of measurements lee <=2 then it is always decreased with 1 independently of bafery status – Monitoring algorithm adapts when bafery degrades
07/03/14 26 ICU Monitor in Demo mode • ICU Monitor have a demo mode to show realisQc waveforms in sales situaQon. • Erroneously a ICU Monitor jumped into Demo mode during monitoring of real paQent • ProacQve QA – Changing waveform to non realisQc waveforms – WriQng “Demo” where waveforms are displayed – Timeout on Demo mode – jumping back to real mode aeer a period of in-‐acQvity PiZalls
07/03/14 27 PiZalls • Too much are considered criQcal or too much are considered not-‐criQcal – Risk of not ﬁnding the criQcal defects • Customer and Manufacturer have diﬀerent view of what is criQcal – Risk of un-‐saQsﬁed customer • Management only buys the cost reducQon part of risk based QA – Risk of poor quality PiZalls • No use of historical data – also within the project – If defect trends shows diﬀerent than your risk evaluaQon (e.g. un-‐idenQﬁed criQcal defects) then you should adapt your Risk Based Approach. • Design miQgaQons adds too much complexity – resulQng in other defects, diﬃcult to maintain SW • The “system” to handle Risk Based QA are too complex.
Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...
In this presentation we will describe our experience developing with a highly dyna...
Presentation to the LITA Forum 7th November 2014 Albuquerque, NM
Un recorrido por los cambios que nos generará el wearabletech en el futuro
Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...
... helping professionals like Michael Agerkvist Petersen discover inside ... Join LinkedIn and access Michael Agerkvist’s ... Radiometer Medical. QA ...
Som medlem af LinkedIn kommer du i selskab med 400 ... risk management ... Michael Agerkvist Petersen. Expert within Medical Device Software ...
View Jens Peter Andersen ... Introduction of risk management in Pallas Informatik based on ISO31000:2009 Risk ... Application of risk management to medical ...
This is called pairwise testing or ... One workaround is to complement pairwise testing with other testing and QA ... risk-based design in which ...
Royal DSM is a global science based company active in health nutrition and materials. ... Risk management. System. ... Medical; Packaging & Graphic ...
Baylor College of Medicine in Houston is recognized as a premier academic health science center and is known for excellence ... Find the best medical care.
Advertising Programmes Business Solutions +Google About Google Google.com © 2016 - Privacy - Terms ...