RR 06 Covert Comparison of SC Model Correlation

50 %
50 %
Information about RR 06 Covert Comparison of SC Model Correlation

Published on January 16, 2008

Author: Riccard

Source: authorstream.com

Comparison of Spacecraft Cost Model Correlation Coefficients:  Comparison of Spacecraft Cost Model Correlation Coefficients Raymond P. Covert The Aerospace Corporation 15049 Conference Center Drive, CH1-410 Suite 600 Chantilly, VA 20151 E-mail: raymond.p.covert@aero.org © 2002 The Aerospace Corporation Outline:  Outline Background Calculating Estimate Uncertainty Importance of Correlation Types of Correlation Introduction Frequency Distribution of Correlation Coefficients Comparing Coefficients Between Models Conclusions Calculating Estimate Uncertainty:  Calculating Estimate Uncertainty Total cost variance, s2 s2=sT[r]s [r] = Correlation matrix (full matrix) s = Vector of standard deviations (cost space) Excel Commands SIGMA_TOT=SQRT(MMULT(MMULT(TRANSPOSE(SIGMA),RHO),SIGMA)) Correlation is Essential in calculating variance! Correlation Coefficients:  Correlation Coefficients Different types of correlation Pearson's product-moment (linear) correlation - measure of the linearity between two random variables Spearman's rank correlation - measure of the monotonicity between two random variables. Where uncertainties between two variables are linear - little difference between linear and rank correlation When uncertainties have non-linear relationship - Answers can be remarkably different Spreadsheet-based Monte Carlo simulations use rank correlation May give a different answer than analytically derived Uncertainty and Residuals:  Uncertainty and Residuals Percentage error or standard error are a measure of residual errors Uncertainty and risk calculations Use residual errors to represent uncertainty Correlation between residuals Cost vs. Weight 0 500 1000 1500 2000 2500 3000 0 20 40 60 80 100 Weight (lbs) Cost ($K) Deriving Correlation Coefficients:  Deriving Correlation Coefficients Sample calculation using randomly generated numbers Introduction:  Introduction Correlation must be included when calculating total variance of an estimate Sometimes correlation coefficients are “guessed” We know how to calculate correlation coefficients for pairs of Cost Estimating Relationships (CERs) Do/will these correlation coefficients apply to other models? How inaccurate would this assumption be? Compared analytically derived correlation coefficients from two spacecraft cost models Unmanned Spacecraft Cost Model Version 7 (USCM7) Small Spacecraft Cost Model Version 2000 (SSCM2000) Are correlation coefficients the same? USCM7 Sample Correlation Coefficients:  USCM7 Sample Correlation Coefficients Sample correlation coefficients for USCM7 Weight based, Mean Unbiased Percentage Error (MUPE) CERs Average correlation coefficient = 0.160 Should these coefficients be used for all spacecraft cost models? Correlation and Causality:  Correlation and Causality Natural tendency to relate causes of correlation to value of the coefficients. Correlation does not necessarily imply a causal relationship between two random variables. It is a measure of tendency of one WBS element to cost more than estimated while another WBS element is over or under estimated. Statistically determined If correlation coefficients were due to causal relationships, coefficients between two spacecraft cost models would be similar for similar WBS pairs To be demonstrated Correlation Coefficients for SSCM 2000:  Correlation Coefficients for SSCM 2000 Applied the same method of calculating correlation coefficients in SSCM as we did for USCM7 Derived correlation coefficients for SSCM Average correlation coefficient = 0.051 Need to compare to USCM-7 Similarity of Correlation Distribution:  Similarity of Correlation Distribution Both models show similar distribution of correlation coefficients Comparing Correlation Coefficients:  Comparing Correlation Coefficients Can’t directly compare correlation between USCM7 and SSCM SSCM does not have a NR/T1 breakout, USCM7 does All correlation coefficients appear to be different and random Combine Nonrecurring (NR) +First Flight Unit (T1) data for USCM to have an “apples to apples” comparison Sum NR+T1 for actual costs (Used USCM derived T1 data) Sum NR+T1 for estimates Calculate mean unbiased percentage error (MUPE) Correlate pairs of columns of MUPE data Compare models with similar estimate content (NR+T1) USCM7* and SSCM Correlation:  USCM7* and SSCM Correlation Are correlation coefficients similar? * Modified NR+T1 Comparison of Correlation Coefficients:  Comparison of Correlation Coefficients We subtracted similar correlation coefficients to find the difference between USCM7 and SSCM There are major differences We should not use one set of correlation coefficients for all models The correlation is statistically based, not causal Correlation coefficients are different Correlation Coefficient Differences:  Correlation Coefficient Differences The mean of the differences of correlation coefficients between USCM7 and SSCM are large Due to effects of combining NR & T1 from USCM7 Different correlation between totals and NR and T1 Mean = 0.378 Correlation Distribution with Modified USCM7:  Correlation Distribution with Modified USCM7 NR+T1 correlation distribution is biased and shaped differently Conclusions:  Conclusions Correlation coefficients are very different between two similar spacecraft models If similar, it would provide evidence for causality They are different, which implies the lack of causality Don’t use correlation coefficients derived for a different model Correlation coefficients are distributed similarly between two similar spacecraft models Perhaps there is a natural distribution (lognormal) We can use this to check if “guesses” at correlation coefficients mirror statistics Correlation is biased and redistributed when new estimators are used (NR+T1) Future Research:  Future Research Look for correlation in software CERs Regress equations for NR+T1 in USCM-7 Determine cause of biases and redistribution of correlation coefficients Apply correlation analysis to cost risk Develop better risk databases Segregate risk data Apply principles of determining correlation References:  References Covert, Raymond, “Correlation Coefficients in the USCM 7 Database”, 3rd Annual Joint ISPA/SCEA International Conference, Tyson's Corner, VA, June 14, 2000. Taylor, John, An Introduction to Error Analysis, University Science Books, Mill Valley, CA 1982. Nguyen, P., et al., Space and Missile Systems Center Unmanned Spacecraft Cost Model Seventh Edition, Space and Missile Systems Center, Cost Division, Los Angeles AFB, CA, August 1994. Summers, P., et al., Small Spacecraft Cost Model Version 2000, The Aerospace Corporation, El Segundo, CA, 2000. Book, Stephen A., "Cost Risk analysis: A Tutorial", in conjunction with the Risk Management Symposium Co sponsored by USAF Space and Missile Systems Center and The Aerospace Institute, Manhattan Beach, CA, 2 June 1997 Book, Stephen A, "Why Correlation Matters in Cost Estimating", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999 Garvey, Paul R, " Do Not Use Rank Correlation in Cost Risk Analysis", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999 Lurie,, Philip M. and Goldberg, Matthew S., " Simulating Correlated Random Variables", 32nd Annual DoD Cost Analysis Symposium, Williamsburg, VA, 2-5 February, 1999

Add a comment

Related presentations

Related pages

Linear Regression Models with Logarithmic Transformations

Linear Regression Models with Logarithmic Transformations ... (xr) = r log(x) ... For instance for ^ = .06, e.06 ˇ 1.06, ...
Read more

Quick-R: Multiple Regression

Multiple (Linear) Regression . R provides comprehensive support for multiple linear regression. The topics below are provided in order of increasing ...
Read more

Correlations - Quick-R: Home Page

# Correlations/covariances among numeric variables in # data frame mtcars. Use listwise deletion of missing data. cor(mtcars, use="complete.obs", ...
Read more

R Tutorials--Multiple Regression - Coastal Carolina University

... (see Simple Nonlinear Correlation and Regression) ... in a regression model raises ... resultant comparison of the models with and ...
Read more

Relative risk - Number Needed to Treat - MedCalc

Description. Allows to calculate a relative risk (or risk ratio). The relative risk is the ratio of the proportions of cases having a positive outcome in ...
Read more

Correlation Coefficients - Jerry Dallal's Home Page

CORRELATION COEFFICIENTS. ... Each data set has a correlation ... Could a valid comparison of the relationship between SAT scores and grade ...
Read more

Correlation in R - YouTube

... Spearman's and Kendall's correlation coefficients. ... and generalized least squares models (LM ... Correlation and Covariance in R (R ...
Read more

Regression Analysis: Estimating Relationships

Regression Analysis: Estimating Relationships ... Purpose of Regression Analysis Procedure for Building Regression Models Selecting ... Multiple R. R ...
Read more