DMI Home    About DMI    Services    Publications    Software    Careers    Contact - J.D. Opdyke, CV & Bio

 (©Copyright 2003 J.D. Opdyke.  All Rights Reserved.  See our  website usage policy)

 

 

 

 

Publications / Invited Presentations / Conference Chairs / University Lectures

 

DataMineit professionals have published multiple award-winning papers in risk analytics, statistical computation and optimization techniques that have wide ranging applicability; they provide actionable, testable solutions to concrete and often urgent business problems in Corporate Banking/Operational Risk, Venture Capital, Mortgage Banking/Credit Risk, Financial Services, Retail Pricing and Marketing, Telecommunications, and just about any data rich field.


 

Opdyke, JD (2024), QuantMinds International: Event Guide Featured Article, "Beating the Correlation Breakdown, for Pearson's and Beyond: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," London, November 18-21, 2024.  .pdf Version, Summary of Book Chapter, Book Chapter,  Interactive Excel Workbook.

 

 

Opdyke, JD (2024), Invited Series, LinkedIn Group: Applied Quantitative Investment Management Group, "Beating the Correlation Breakdown, for Pearson's and Beyond: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," July 2024: Combined into 1 Article, All 4 Posts Together in 1 pdf, Post 1, Post 2, Post 3, Post 4.  Interactive Excel Workbook.

 

 

Opdyke, JD (2024), KEYNOTE: Invited Speaker, Applied Quantitative Steam Chair, QuantStrats US, 2024, "Beating the Correlation Breakdown, for Pearson's and Beyond: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," March 12, 2024.
 Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.24-27, Relation to Causal Models--QuantStrats Roundtable

Summary Article

 

·      We live in a multivariate world, and effective modeling of financial portfolios, including their construction, allocation, forecasting, and risk analysis, simply is not possible without explicitly modeling the dependence structure of their assets. Correlation/concordance matrices rightly play a central, ubiquitous, and foundational role here. They often stand as the most impactful parameter in portfolio models, and yet both the literature and practitioners typically fail to treat them with the same level of quantitative and probabilistic rigor as the other estimated parameters in these models. This is especially troubling given the widely documented ‘correlation breakdowns’ that occur during times of extreme market stress, which is when risk analytics, and the consequences of the methods chosen for portfolio construction and allocation, matter the most. 

·       This work builds on prior research covering geometric frameworks to derive the finite-sample distributions of a very broad class of the most widely used correlation matrices and dependence measures – including Pearson’s product moment, Spearman’s Rho, Kendall’s Tau, Szekely’s distance correlation, the Tail Dependence Matrix (see Embrechts et al, 2016, and Shyamalkumar & Tao, 2020), and more. These distributions maintain validity under the most general conditions possible, requiring only the positive definiteness of the matrix. The proposed Nonparametric Angles-based Correlation method (NAbC) unifies estimation of the confidence intervals and p-values associated with each and every pairwise correlation cell, with those associated with the entire matrix, determining both simultaneously, consistently, and quantitatively. It also provides the quantile function for both the individual cells and the entire matrix: when given a matrix of cumulative distribution function values, it provides the unique, corresponding correlation matrix. 

·       All results obtained under challenging, real-world data conditions (e.g. varying degrees of tail heaviness, asymmetry, non-stationarity, and serial correlation in the margins, as well as under complex copula functions and near-singular matrices) are consistent with those well established in the Random Matrix Theory literature. But they are more robust and accurate under many conditions than more complex and limited spectral methods. Finally, NAbC provides something no other approach does: flexible scenario definition and stress testing, allowing for selective perturbation of chosen cells in the matrix, while holding the remaining non-chosen cells constant. NAbC’s range of application is as broad as the widespread and necessary use of these correlation/dependence matrices themselves, making it a potent tool for proactively flagging, probabilistically monitoring, and potentially mitigating and avoiding the worst consequences of correlation breakdowns.

 

Opdyke, JD (2023), Invited Speaker, QuantMindsEdge - AlphaGen and Quant Investing, "Beating the Correlation Breakdown, for Pearson's and Beyond: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," November, 2023.
 Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.24-27,

Summary Article

 

·      Correlation matrices are foundational and ubiquitous in finance, investing, and risk analysis and management, and often the most impactful parameter in the models used in these fields. But they rarely are treated with the same quantitative and probabilistic rigor as the other estimated parameters in these models. This is especially troubling given the widely documented ‘correlation breakdowns’ that occur during times of extreme market stress, which is when risk analytics, and the consequences of the methods chosen for portfolio construction and allocation, matter most. 

·       Using a geometric framework, this work derives the finite-sample distributions of the most widely used correlation matrices (Pearson’s product moment, Spearman’s Rho, and Kendall’s Tau) under the most general conditions possible, requiring only the positive definiteness of the matrix and the existence of the means and variances of the marginal distributions. The Nonparametric Angles-based Correlation (NAbC) method unifies estimation of the confidence intervals and p-values associated with each and every pairwise correlation cell, with those associated with the entire matrix, determining both simultaneously, consistently, and quantitatively. It also provides the quantile function for both the individual cells and the entire matrix: when given a matrix of cumulative distribution function values, it provides the unique, corresponding correlation matrix. 

·       All results obtained under challenging, real-world data conditions (e.g. varying degrees of tail heaviness, asymmetry, non-stationarity, and serial correlation in the margins, as well as under complex copula functions and near-singular matrices) are consistent with those well established in the Random Matrix Theory literature. But they are more robust and accurate under many conditions than more complex and limited spectral methods. Finally, NAbC provides something no other approach does: flexible scenario definition and stress testing, allowing for selective perturbation of chosen cells in the matrix, while holding the remaining non-chosen cells constant. NAbC’s range of application is as broad as the widespread use of these correlation matrices themselves, making it a potent tool for proactively flagging, probabilistically monitoring, and potentially mitigating and avoiding correlation breakdowns.

 

Opdyke, JD (2023), Invited Guest Lecture, Columbia University - School of Professional Studies: Machine Learning for Risk Management, "Beating the Correlation Breakdown: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," March, 2023.
Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.21-24

Summary Article.

 

 

Opdyke, JD (2023), Invited Speaker, QuantStrats-10th Ed., Quant Strategy & Innovation Stream (Full Program, Highlights): "Beating the Correlation Breakdown: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," March 14, 2023.
Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.21-24

Summary Article.

 

 

Opdyke, JD (2022), Invited Speaker, RiskMinds International / RiskFuse: "Beating the Correlation Breakdown: Robust Inference and Flexible Scenarios and Stress Testing for Financial Portfolios," December 6, 2022.
Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.21-24

Summary Article.

 

 

Opdyke, JD (2022), Invited Speaker, QuantMindsEdge-Alpha and Quant Investing: New Research: Applying Machine Learning Techniques to Alpha Generation Models -- "The Correlation Matrix under General Conditions: Robust Inference and Flexible Stress Testing and Scenarios for Financial Portfolios," June 6, 2022.
Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.21-24

Summary Article.

 

 

Opdyke, JD (2021), "The Correlation Matrix under General Conditions: Robust Inference and Flexible Stress Testing and Scenarios for Financial Portfolios," forthcoming, 2022.
Excel Workbook implementation of Fully Analytic Gaussian Identity Matrix results derived on pp.21-24.

 

 

Opdyke, JD (2020), Invited Speaker, QuantMinds International 2020, "Full Probabilistic Control for Direct and Robust, Generalized and Targeted Stressing of the Correlation Matrix
(Even When Eigenvalues are Empirically Challenging)," Hamburg, Germany, November 2-6, 2020.

 

 

Opdyke, JD (2020), Invited Speaker, QuantMinds/RiskMinds Americas 2020, Advances in option pricing, trading and modelling, "Full Probabilistic Control for Direct and Robust, Generalized and Targeted Stressing of the Correlation Matrix (Even When Eigenvalues are Empirically Challenging)," Boston, MA, September 22-23, 2020.

 

 

Opdyke, JD (2019), Invited Speaker, RiskMinds International 2019, Invest Summit, "Getting Extreme VaR Right: Eliminating Convexity and Approximation Biases from Heavy-tailed, Moderately-sized Samples," Amsterdam, Netherlands, December 2-6, 2019.

 

 

Opdyke, JD (2019), Invited Speaker, QuantMinds / RiskMinds Americas 2019, Quant Innovation: Machine Learning, HFT, AI & Data, "Getting Extreme VaR Right: Eliminating Convexity and Approximation Biases from Heavy-tailed, Moderately-sized Samples," Boston, MA, September 9-11, 2019.

 

 

Opdyke, JD (2020), "Getting Extreme VaR Right: Eliminating Convexity and Approximation Biases from Heavy-tailed, Moderately-sized Samples," forthcoming, 2020.

 

 

Opdyke, JD (2018), Predictive Risk Analytics-Data-Driven Risk Measurement and Mitigation for Competitive Market Advantage

 

 

Opdyke, JD, (2017), "Fast, Accurate, Straightforward Extreme Quantiles of Compound Loss Distributions," The Journal of Operational Risk, Volume 12, Issue 4, 1-30, December, 2017.   PREPRINT.  arXiv PREPRINT.  SSRN PREPRINT.

 

 

Opdyke, JD (2016), Invited Speaker, RiskMinds Americas 2016, "If not AMA, or SMA, then What? A Robust, Risk Sensitive, and Internally Consistent OpRisk Capital Estimation and Stress Testing Framework," Chicago, IL, September 20-23, 2016.

 

 

Opdyke, JD (2016), Invited Speaker, Quant Summit USA 2016, Risk.Net, "The Challenges of, and Practical Solutions to, Capital Aggregation and Allocation under Heavy-Tailed, Empirical Loss Distributions," New York, New York, July 12-13, 2016.

 

 

Opdyke, JD (2016), Invited Speaker, Chairman -- Quant Studies for OpRisk Stream, OpRisk North America-2016, "Operational Risk Regulatory Capital Estimation," New York, New York, March 15-16, 2016.

 

 

Opdyke, JD (2016), Invited Speaker, Global Association of Risk Professionals (GARP), 17th Annual Risk Management Convention, "Operational Risk Modeling," New York, New York, March 1-2, 2016.

 

 

Opdyke, JD (2015), Conference Chairperson and Invited Speaker, Operational Risk Management Forum: Marcus Evans North America, "Estimating Operational Risk Capital with Greater Accuracy, Precision, and Robustness," New York, New York, September 16-17, 2015.

 

 

Opdyke, JD (2015), Operational Risk eXchange (ORX) Analytics Forum, "Estimating Operational Risk Capital with Greater Accuracy, Precision, and Robustness," Milan, Italy, May 21-22, 2015.

 

 

Opdyke, JD (2015), Invited Speaker and Moderator, OpRisk North America-2015, “Extreme Losses and Operational Risk Capital: Myths and Realities,” New York, New York, March, 2015.

 

 

Opdyke, JD (2014), Risk Week - Yale, "Estimating Operational Risk Capital with Greater Accuracy, Precision, and Robustness," Risk Seminars - Incisive Media, Invited Speaker, New Haven, CT, December 9-14, 2014.

 

 

Opdyke, JD (2014), Joint Statistical Meetings-2014, "Estimating Operational Risk Capital with Greater Accuracy, Precision, and Robustness," American Statistical Association Proceedings - JSM2014, Section on Risk Analysis, Boston, MA, August 2-7, 2014.

 

 

Opdyke, JD (2014), OpRisk North America-2014, “From Loss Data to Capital: Implementing a Comprehensive Operational Risk Capital Estimation Framework Under the AMA-LDA,” Invited Workshop Leader, 4-session, 6 hour Workshop, March, 2014.

 

 

WINNER - 2015 ORR Innovation Awards, Voted "Paper of the Year" by OPERATIONAL RISK & REGULATION staff in consultation with industry experts.
 Opdyke, JD (2014), "Estimating Operational Risk Capital with Greater Accuracy, Precision, and Robustness", The Journal of Operational Risk, Volume 9, Issue 4, 3-79, December, 2014. current downloads:  Errata to published manuscript, PREPRINT, SSRN PREPRINT, arXiv PREPRINT, Scrib-PREPRINT, PRESENTATION-DECK, scrib-Deck.

 

 

Opdyke, JD (2013), "Bootstraps, Permutation Tests, and Sampling Orders of Magnitude Faster Using SAS®,"  Computational Statistics - WIRE Interdisciplinary Reviews, Vol. 5, Issue 5, 391-405 (.pdf full preprint, .pdf Appendices only, SSRN download, scrib download, KDNuggets).

 

 

Opdyke, JD  (2012), "Better Capital Estimation via Exact Sensitivity Analysis Using the Influence Function," American Bankers Association: ABA Operational Risk Modeling Forum, Invited Speaker, Washington, DC, July 18-20, current downloads: ABSTRACT, .pdf, scrib.

 

 

WINNER - 2012 ORR Innovation Awards, Voted "Paper of the Year" by OPERATIONAL RISK & REGULATION staff in consultation with industry experts.

Opdyke, JD, (2012), "Estimating Operational Risk Capital: the Challenges of Truncation, the Hazards of MLE, and the Promise of Robust Statistics," with Alex Cavallo, The Journal of Operational Risk, 7(3), 3-90.  (preprint download, SSRN download, Scrib download).

 

 

Opdyke, JD, (2012), "Operational Risk Capital Estimation and Planning: Exact Sensitivity Analysis and Business Decision Making Using the Influence Function," with Alex Cavallo, in Operational Risk: New Frontiers Explored, Davis E., ed., Risk Books, London. (preprint download, SSRN download, Scrib download)

 

 

Opdyke, JD, (2011), Robust Statistics vs. MLE for OpRisk Severity Distribution Parameter Estimation (With and Without Truncation), ORX Analytics Forum, San Francisco, California, September 27-29, 2011, Invited Speaker, current downloads: ABSTRACT, .pps, .pdf, scrib.


 

Opdyke, JD, (2011), Robust Statistics vs. MLE for OpRisk Severity Distribution Parameter Estimation, American Bankers Association: ABA Operational Risk Modeling Forum, Charlotte, North Carolina, August 10-11, 2011, Invited Speaker, current downloads .pps, .pdf, scrib.


 

Opdyke, JD, (2011), Bootstraps, Permutation Tests, and Sampling With and Without Replacement Orders of Magnitude Faster Using SAS®, American Statistical Association Proceedings - JSM2011, Section on Statistical Computing, download .pps, .pdf, scrib.


 

Opdyke, JD, January (2011), Permutation Tests (and Sampling Without Replacement) Orders of Magnitude Faster Using SAS®, InterStat. (preprint download, SSRN download, Scribe download, Text file containing  code MFPTUS.sas, KDNuggets)

 

        [download even faster, proprietary versions of these Bootstrap, Permutation Test, and Sampling
         With and Without Replacement algorithms, saved as compiled SAS® macros, at
          http://www.DataMineIt.com/DMI_software.htm]


 

Opdyke, JD, September (2010), Much Faster Bootstraps Using SAS®,  InterStat.
(preprint download, Scribe download, SSRN download, Text file containing SAS® code MFBUS.sas, KDNuggets)


 

Opdyke, JD, March (2010), A Unified Approach to Algorithms Generating Unrestricted and Restricted Integer Compositions and Integer Partitions, Journal of Mathematical Modelling and Algorithms, Vol. 9, No. 1, 53-97. (preprint-direct download, SSRN.com download, including SAS® code, Scribe download with SAS® code, Mathematica® code implementing counting formulae: zipped Mathematica® notebook .nb, .html)

 

    ALREADY USED/CITED IN:

Opdyke, JD, September (2009), A Powerful and Robust Nonparametric Statistic for Joint Mean-Variance Quality Control, InterStat.  (preprint download, Scribe download, SSRN download)


 

Opdyke, JD, December (2007), Comparing Sharpe Ratios:  So where are the p-values?, Journal of Asset Management, Vol. 8, No. 5. 
(preprint, SSRN.com download, Hedge Funds Consistency Index download).
    - SAS Program (email for 1-time password) - p-values from Sharpe Ratio comparisons and Mutual Fund Rankings (.pdf results)

    - Excel Workbook (.xls- 0.64MB) p-values from Sharpe Ratio comparisons and Mutual Fund Rankings

    - JSM2006 PowerPoint Presentation

 

    SELECTED REVIEW QUOTES:

  • “This is a comprehensive, authoritative, scholarly study of the topic." –  tenured economics Professor with over 100 peer-reviewed articles in top journals.

  • "Amazing scholarship ... magnum opus. ...excellent paper." –  tenured statistics Professor with numerous seminal papers and several statistical textbooks that are bibles in the field.
     

  • "I congratulate you ... on a paper that needed to be written." –  Senior Analyst at one of the top investment rating firms with over 20 years of experience.
     

Opdyke, JD, August (2006), Easily Implemented Confidence Intervals and Hypothesis Tests for Sharpe Ratios Under General Conditions, American Statistical Association Proceedings - JSM2006, Business and Economics Statistics Section.


 

Opdyke, JD, (2005), A Single, Powerful, Nonparametric Statistic for Continuous-data Telecommunications ‘Parity Testing,’ Journal of Modern Applied Statistical Methods, Vol. 4, No. 2.


 

Opdyke, JD, (2005), A Nonparametric Statistic for Joint Mean-Variance Quality Control, American Statistical Association Proceedings - JSM2005, Section on Quality and Productivity, copyrighted presentation.


 

Opdyke, JD, October (2004), Misuse of the ‘modified’ t statistic in Regulatory Telecommunications, Telecommunications Policy, Vol. 28, No. 11, 821-866. (REVIEW: http://askdavid.com/reviews/books/telecommunications/150)


 

Opdyke, JD, August (2003), Misuse of the ‘modified’ t-statistic in Regulatory Telecommunications, American Statistical Association Proceedings - JSM2003, Business and Economics Statistics Section.


 

Opdyke, JD, May (2003), Fast Permutation Tests that Maximize Power Under Conventional Monte Carlo Sampling for Pairwise and Multiple Comparisons, Journal of Modern Applied Statistical Methods, Vol. 2, No. 1.


 

Opdyke, JD, August (2002), Fast Two-Sample Permutation Tests, Especially for Multiple Comparisons and Even When One Sample is Large, That Efficiently Maximize Power Under Conventional Monte Carlo Sampling and Allow for Simultaneous Permutation-Style P-value Adjustments, MCP 2002 – The 3rd International Conference on Multiple Comparisons, Bethesda, Maryland. (download .pps)


 

WINNER – Voted Best Paper, Statistics and Pharmacokinetics Section
Opdyke, JD, May (2002), Fast Two-Sample Permutation Tests, Even When One Sample is Large, That Efficiently Maximize Power Under Crude Monte Carlo Sampling, PharmaSUG 2002 -- National Conference of the Pharmaceutical SAS Users Group.

 

Opdyke, JD, April (2000), authored a 100-page proposal to perform a comprehensive, multi-state telecommunications Operations Support Services (OSS) performance measurement audit of a Regional Bell Operating Company, and presented the executive summary before 35 selected members of i) the Public Service Commissions of twelve states, ii) a regulatory research institute with oversight authority, and iii) a telecommunications consulting firm with administrative oversight authority


 

Opdyke, JD, authored and presented before 20 representatives of four Regional Bell Operating Companies the statistical foundations, application, and regulatory remedy (fine) implications of utilizing permutation tests in telecommunications OSS performance measurement “parity testing.”


 

Opdyke, JD, with Raymond S. Hartman and Deloris W. Wright, July (1996), The Use of Regression Techniques in Transfer Pricing Analysis, International Bureau of Fiscal Documentation – Tax Treatment of Transfer Pricing, Supplement No. 18.