Research for Management Policy

Shyam Sunder, Yale UniversityYale-Great Lakes Research Conference

Great Lakes Institute of ManagementChennai, India, December 26, 2010

An Overview

• Social science and social engineering• Robustness of findings to their discovery• Field data: Causal direction, correlation of hypotheticals• Experiments and bench testing• Social science as instrument of policy• Accounting policy beyond social sciences• Challenges for policy research, historical experience• Closed or open systems?

04/21/23 Sunder, Research for Accounting Policy

2

04/21/23 Sunder, Research for Accounting Policy

3

Social sciences as instruments of social engineering

• Social science as the dominant model of business research since mid-20th century

• Although engineering originated well before the origins of natural sciences, advances in natural sciences have enabled us to engineer great many artifacts we consider indispensable today

• Management, too, originated well before policy research; but it does not seem unreasonable to think that we can also take advantage of learning from social sciences to make better social policy

04/21/23 Sunder, Research for Accounting Policy

4

Limits of Parallels

• This parallel can take us only so far• Natural sciences search for, and identify, laws of

nature valid across time and space• Validity, replicability, and predictability of these

laws confers prestige on sciences• But there is another form of scholarship on our

campuses--humanities that regard the behavior of sentient beings as infinitely variable; each of character in Iliad, Macbeth, or Ramayana is unique

• They pursue eternal truths, but admit no laws

04/21/23 Sunder, Research for Accounting Policy

5

What are social sciences?

• “Social” recognizes that the subject of study are sentient beings (with free will that humanists recognize), not marbles or atoms without will (that scientists study)

• “Science” seeks the prestige associated with the search for eternal laws

• Neither sciences nor humanities allow much room for laws of human behavior we seek in social sciences

• Free will and laws of behavior do not sit well together; we want but can’t have it both ways

04/21/23 Sunder, Research for Accounting Policy

6

Laws of Social Sciences

• To serve as a basis for social policy, the “laws” of social sciences must have stability (be robust to their own discovery)

• Since humans learn and adapt, social science findings can alter behavior in ways that tend to invalidate the findings

• Findings which are robust to such adaptation can be called “laws” of social sciences, and may serve as the basis for social policy

04/21/23 Sunder, Research for Accounting Policy

7

How Robust Are Our Findings?

• Independent of the method of research we use, robustness of findings (to their own discovery) is a pre-requisite for their use as basis for policy

• Like unclaimed dollar bills on side-walk, many findings (e.g., small firm and Monday effects) disappear upon being reported

• There are other findings (e.g., determination of price by intersection of demand and supply) are robust in this sense (not merely statistically)

• So the first pre-requisite for usefulness of any research findings for policy is this stability

• Most such laws are properties of institutions, not behavior [1]

04/21/23 Sunder, Research for Accounting Policy

8

Causal Link

• Policy makers want to know if the manipulation of the policy variable under consideration has a directional (causal) link to the desired objective. Correlation does nothing for them.

• Yet, the problem of establishing a causal link between two variables on the basis of field data remains largely unsolved due to endogeneity

• Experimental methods have been presented as an alternative to address this problem, but they, too, have limitations of their own

• Consider both approaches briefly

04/21/23 Sunder, Research for Accounting Policy

9

Problems of Inference from Field Data: Causal Direction

• Labeling of correlation as cause is more of a rule than an exception in accounting research journals

• When correctly labeled as correlation, what can the policy maker do with the finding?

• To claim that the finding is “consistent” with Hypothesis X fails to point out that – It is also consistent with innumerable other

hypotheses not mentioned in the report, and– No hypothesis has been rejected (violating the

essence of Fisher-Neyman-Pearson framework)

Accepting or Failing to Reject the Null Hypothesis

• It is important to note the philosophical difference between accepting the null hypothesis and simply failing to reject it. The "fail to reject" terminology highlights the fact that the null hypothesis is assumed to be true from the start of the test; if there is a lack of evidence against it, it simply continues to be assumed true. The phrase "accept the null hypothesis" may suggest it has been proved simply because it has not been disproved, a logical fallacy known as the argument from ignorance. Unless a test with particularly high power is used, the idea of "accepting" the null hypothesis may be dangerous.

“Absence of evidence is not evidence of absence”

• When a researcher writes the qualified statement "we found no statistically significant difference," which is then misquoted by others as "they found that there was no difference." Actually, statistics cannot be used to prove that there is exactly zero difference between two populations. Failing to find evidence that there is a difference does not constitute evidence that there is no difference. This principle is sometimes described by the maxim "Absence of evidence is not evidence of absence."

Statistical Significance• Attempts to educate researchers on how to avoid pitfalls of using

statistical significance have had little success. In the papers "Significance Tests Harm Progress in Forecasting," and "Statistical Significance Tests are Unnecessary Even When Properly Done,“

• Armstrong makes the case that even when done properly, statistical significance tests are of no value. A number of attempts failed to find empirical evidence supporting the use of significance tests.

• Tests of statistical significance are harmful to the development of scientific knowledge because they distract researchers from the use of proper methods.

• Armstrong suggests authors should avoid tests of statistical significance; instead, they should report on effect sizes, confidence intervals, replications/extensions, and meta-analyses.

• J. Scott Armstrong

Choosing the null: Ex ante, or after looking at the data?

• Some statisticians have commented that pure "significance testing" has what is actually a rather strange goal of detecting the existence of a "real" difference between two populations. In practice a difference can almost always be found given a large enough sample. The typically more relevant goal of science is a determination of causal effect size. The amount and nature of the difference, in other words, is what should be studied.

• Hypothesis testing is controversial when the alternative hypothesis is suspected to be true at the outset of the experiment, making the null hypothesis the reverse of what the experimenter actually believes; it is put forward as a straw man only to allow the data to contradict it. Many statisticians have pointed out that rejecting the null hypothesis says nothing or very little about the likelihood that the null is true.

04/21/23 Sunder, Research for Accounting Policy

14

Problems of Inference from Field Data: Hypotheticals and Efficiency

• Our attempt to produce policy-relevant research often take the following form:

• Info. Sys. 1 Price system 1• Info. Sys. 2 Price system 2• Suppose information system 1 and price function

1 are the status quo and the policy maker wants to know the consequences of changing the information system from 1 to 2

04/21/23 Sunder, Research for Accounting Policy

15

Problems of Inference from Field Data: Hypotheticals and Efficiency

• We can gather data on accounting numbers and prices under status quo and estimate their statistical relationship, R(1)

• In many situations, we can also calculate (or reasonably estimate) what the accounting numbers would have been under the policy alternative (the hypothetical)

• If we could observe prices that would be generated under the policy alternative, we could also estimate the statistical relationship R(2)

• Does a comparison of R(1) and R(2) help the policy makers?

• If stronger R(2) implies preference for the policy alternative, it is trivially simple to push R(2) to the upper limit by simply using prices for accounting

04/21/23 Sunder, Research for Accounting Policy

16

Problems of Inference from Field Data: Hypotheticals and Efficiency

• Of course, we are rarely so lucky as to be able to observe P(2)

• An oft-used practice is to estimate the statistical relationship between the hypothetical I(2) and actually observed P(1), and then compare this R(2)* with R(1) and suggest that a stronger R(2)* implies the alternative to be preferred policy

• Info. Sys. 1 Price system 1• Info. Sys. 2 Price system 1

04/21/23 Sunder, Research for Accounting Policy

17

Logic of Inference

Reporting System 1

Reporting System 2 Price System 2

Price System 1

04/21/23 Sunder, Research for Accounting Policy

18

Logic of Inference

Reporting System 1

Reporting System 2 Price System 2

Price System 1

04/21/23 Sunder, Research for Accounting Policy

19

Logic of Inference and Policy

• This type of inference from field data does not help the policy makers

• We like them to give us a nod to acknowledge our work, and perhaps even support it

• But the logical foundations of such inference, and its implications for policy remain to be worked out

04/21/23 Sunder, Research for Accounting Policy

20

What about Lab Experiments?

• Lab methods allow us to address the causality problem with greater confidence

• But they also raise new challenges for use of findings in making of accounting policy

• Accounting is highly institutionalized (complex interactions, expectations), like engineering

• Experimental methods were developed for social sciences where a single simple example can support a general proposition about existence, or otherwise

• Simpler experiments suffice for basic disciplines such as economics, psychology and physics, but not for accounting policy or bridge design

• Time scale problem: choosing points vs. functions

04/21/23 Sunder, Research for Accounting Policy

21

Bench Testing of Policy

• Bench testing of an accounting policy alternative calls for far greater complexity in design of the lab experiment than in case of testing economic theory

• More complex decisions, larger choice space, need more time (order of magnitude)

• Compare choice of a point on a function with choice of a function

04/21/23 Sunder, Research for Accounting Policy

22

Instruments of social sciences

• Analysis of data gathered from the field• Analysis of controlled experiments in the lab or the field• Abstract mathematical analysis• Historical analysis• Introspection• If accounting research is to contribute to policy, we shall

have to use all the tools at our disposal as and when necessary

• Prior commitment to one or the other tool set is likely to be self-defeating

04/21/23 Sunder, Research for Accounting Policy

23

Way forward: management policy beyond social sciences?

• Management policy includes considerations that go beyond the scope of social sciences

• I do not believe that a good system of management is feasible without broadly accepted social norms of personal responsibility in the business and managerial community

• Our own (academic) community could not function without such norms

• We should not expect development and validation of decision models to be sufficient to create order in the world of management

• Social and mathematical sciences can only do so much. The rest is up to the management community

04/21/23 Sunder, Research for Accounting Policy

24

The Challenge of Policy for Research

• If research does not (at least ultimately) lead to better understanding, practice or policy, it will be ignored

• The methods we have do not always take us in that direction

• Stick to these methods (because that is what get published)

• Or, move beyond the methods if they cannot enlighten us

• Does research affect management policy for better?• History suggests that improving management and

governance is not an easy task

04/21/23 Sunder, Research for Accounting Policy

25

04/21/23 Sunder, Research for Accounting Policy

26

04/21/23 Sunder, Research for Accounting Policy

27

Whither Management: Windows™ or Open Systems

• Comfort vs. choice• Uniformity and stagnation vs. dynamic change• Predictability vs. some disorder• High prices or the advantages of technological progress• Management as an eco-system or a machine (garden or

a building)• Huxley or Hayek• Nanny or personal responsibility• Where does the research community stand? • There are many life times of research agendas here if

we are willing to consider them

04/21/23 Sunder, Research for Accounting Policy

28

References• Gode, D. K. and Shyam Sunder. 1993. “Allocative Efficiency of Markets

with Zero Intelligence Traders: Market as a Partial Substitute for Individual Rationality.” The Journal of Political Economy 101, no. 1 (February 1993): 119-137.

• Sunder, Shyam. “Determinants of Economic Interaction: Behavior or Structure.” Journal of Economic Interaction and Coordination 1, no. 1 (May 2006): 21-32.

• Sunder, Shyam. “What Have We Learned from Experimental Finance?” In Developments on Experimental Economics: New Approaches to Solving Real-world Problems edited by Sobei H. Oda, 91-100. Lecture Notes in Economics and Mathematical Systems 590. Berlin: Springer, 2007.

• William T. Baxter (Professor Emeritus, LSE), made many of these arguments over half-a-century ago (“Recommendations on Accounting Theory” in Baxter and Davidson, Studies in Accounting Theory, 1st edition).

• Sunder, Shyam. 2010. “Was Accounting a Root Cause of the Global Financial Crisis?” Plenary Address to the Annual Meeting of the International Corporate Governance Network, Toronto, June 7.

Thank You!

Shyam.sunder@yale.edu

www.som.yale.edu/faculty/sunder