understanding the financial times - unicon...sep 12, 2015 · activities include conferences,...
TRANSCRIPT
Understanding the Financial Times
Executive Education Rankings:
A 360‐Degree Review
Original Research Sponsored by
September 12, 2015
Tom Cavers, James Pulcrano, Jenny Stine
Preface: UNICON statement accompanying the research report Understanding the Financial Times Executive Education Rankings: A 360-Degree Review
UNICON – The International University Consortium for Executive Education
UNICON is a global consortium of business-school-based executive education organizations. Its primary activities include conferences, research, benchmarking, sharing of best practices, staff development, recruitment/job postings, information-sharing, and extensive networking among members, all centered on the business and practice of business-school-based executive education.
The UNICON Research Committee
The UNICON Research Committee advises the UNICON Board of Directors on research priorities, cultivates a network of research resources and manages the overall research pipeline and projects. The Research Committee is made up of volunteers from UNICON’s member organizations.
UNICON Research Report: Understanding the Financial Times Executive Education Rankings: A 360-Degree Review
The Financial Times (FT) annual rankings of non-degree Executive Education providers are the best known and most discussed in the industry. In late 2014, the UNICON Research Committee commissioned and the UNICON Board approved the UNICON research project “Understanding the Financial Times Executive Education Rankings: A 360-Degree Review” in large measure to better understand and address many recurring questions about the FT’s ranking system.
UNICON is a diverse organization, with representation from over 100 schools. In addition to size and geography, schools are diversified by the expertise, reputation and strength of their faculty, the types and sizes of their customers, and increasingly the breadth and depth of their executive education portfolios. The ability to represent many perspectives in executive education is a great strength of the UNICON and a source of continued learning and vitality in the field. This diversity of views and interests also means that there is no single “UNICON perspective” on its commissioned research topics, including no single perspective on ranking systems. One motivation for commissioning this research project was the view expressed by a number of members that ranking systems can and should better reflect the diversity of executive education offerings, the distinctive strengths of schools, and the evolving nature of the marketplace.
The interpretations and perspectives expressed in this report are those of the researchers - Jenny Stine, Jim Pulcrano, and Tom Cavers, professionals who are deeply familiar with the executive education field and the needs and objectives of its stakeholders. UNICON desired that the research project reflect diverse perspectives of these stakeholders, and that the findings of this report provide deeper understanding of the rankings systems and offer specific suggestions for improvements to the rankings systems to make them more valuable to the entire field, including schools, customers and clients, and expert observers.
25 August 2015
3 | P a g e
Preface
In 1999, a young scholar named Linda Wedlin at Uppsala University in Sweden selected the Financial Times rankings as the topic for her dissertation research. At the time she thought that ranking business schools was something that “might be hot for a year or two.” Now, she acknowledges, “it is more important than any of us could have imagined.”1
It has been seventeen years since the FT launched its Executive Education (EE) rankings.2 Today they are robust, with over 100 schools participating globally. In fact, the FT EE rankings are the only active international rankings of executive education today.3 However, despite their prominence, many questions about them persist within the industry today.
Our project began with a desire to understand more about the executive education rankings: How do they really work? What do they say about quality? Do customers of executive education use the rankings, and if so, how? How do business schools perceive the rankings, and what would they change? And what do the experts have to say? In short, if we were to do a 360-degree review of the FT Executive Education rankings, what would we learn?
To better understand the FT EE rankings – and to get a 360-degree view – we sought out the key stakeholders in the realm of executive education. This included surveying and interviewing four main groups:4
1) Open enrollment participants – 713 surveyed and 5 interviewed 2) Learning & development (L&D) professionals (custom program customers) – 279 surveyed and 9
interviewed 3) Leaders of executive education at business schools – 94 surveyed and 12 interviewed 4) Experts in the field of rankings – 14 interviewed, including both current and former ranking
executives as well as the FT itself
We were particularly interested in how open and custom program customers use the rankings in their decision-making process. We were also interested in how business schools perceive the rankings. We were sure they would have some strong opinions, and they did not disappoint us. To round out our perspective, we wanted to learn what rankings experts – including social scientists, statisticians,
1 L. Wedlin interview, identified with permission. 2 The Financial Times is a leading business news and information organization. It is part of the Financial Times Group, which is a division of Pearson PLC. See Appendix 1: Pearson PLC and the Financial Times for additional context. 3 Early 2015, Business Week decided to drop its biannual Executive Education and EMBA rankings, see http://poetsandquantsforexecs.com/2015/03/17/bw-drops-emba-exec-ed-rankings/. 4 Institution and role of interviewees is detailed in Appendix 16: List of Interviewees by Institution and Role.
4 | P a g e
consultants, accreditation bodies, and both current and former business school rankings professionals – had to say. Finally, we studied the published literature on business school rankings.5
As you will see, at a superficial level, we found a lot of support for executive education rankings in general and even for the FT EE rankings in particular. However, when we looked deeper, we also found some significant disconnects and challenges with the FT’s current approach to the rankings. There are concerns that the FT has not been keeping up with changes in the industry, that their rankings could be much more transparent, and that their methodology could be strengthened.
Our report is organized as follows. First, we look into the FT’s rankings from a technical perspective to understand how they work. Next, we look at the perceptions and use of the rankings by key stakeholders. Third, we take a deep dive into the “disconnects and challenges” these perceptions surface. And finally, we offer two sets of recommendations, both how schools and professionals can get the most out of the current rankings as well as how the FT could address the disconnects and challenges in the longer term.
In addition to the body of the report, we have compiled a collection of items in the appendices and online. We hope that readers will find them useful. A table of contents for the appendices appears on the first appendix page. Additional online materials delivered with our report include:
• An infographic that explains the ranking process • 17 years of ranking data in an Excel table (overall ranks only; not by category) • 17 years of the ranking magazines / reports in PDF
We realize that any research into rankings requires sailing into controversial waters. We hope that readers will accept this work as an objective look into the FT Executive Education rankings and that it will help us to understand both strengths and weaknesses. Ultimately, we hope that we can help stimulate dialogue and make these rankings as useful as possible for all parties involved.
- Tom Cavers, Jim Pulcrano, and Jenny Stine July 20, 2015
5 In addition to references in the footnotes, there is a Bibliographic Essay at the end of this report that gives an overview of recent literature on the rankings.
5 | P a g e
Report Overview
This report is organized in four sections. These include:
• A detailed look into the mechanics of the rankings (“How the Rankings Work”) • An exploration (“360”) of how the rankings are perceived and used by four key stakeholder
groups (open enrollment customers, custom program customers, business schools, and rankings experts). This exploration is built primarily from our survey and interview data. Key insights include:
o Many customers of executive education are aware of rankings, and use them, at least to some degree
o Many business schools are dissatisfied with rankings. At the same time, they are frequently responsible for communicating them.
o Experts stress that rankings are not quality measures and also may fail to capture innovation.
• An exploration of the challenges and disconnects set up in the previous two sections. Here we go deeply into criticisms of the rankings as they relate to what we discovered in our research.
• Detailed recommendations, including specific recommendations for consumers of the rankings, for business schools, and for the Financial Times. Key recommendations include:
o For business schools to consider the value the rankings provide to some customers and clients, and to use the information provided in the report to participate in them more effectively;
o For customers of executive education to be more aware of the limitations of the rankings, and to engage with the FT to make sure the rankings criteria meet their needs; and
o For the FT to work collaboratively with business schools and customers to build a new and better executive education ranking process that addresses issues of transparency, diversity, and customer value.
The body of the report is followed by Appendices with a great deal of additional information for readers who really want to take a deep dive into our research, and into specific questions.
6 | P a g e
Table of Contents
Preface ............................................................................................................................................................ 3
Report Overview ............................................................................................................................................. 5
Table of Contents ............................................................................................................................................ 6
How the Rankings Work .................................................................................................................................. 7
A 360° View: How the Rankings Are Perceived and Used ............................................................................ 13
Open Enrollment Customers ..................................................................................................................... 17
Learning & Development Professionals .................................................................................................... 20
Business Schools ....................................................................................................................................... 23
Rankings Experts ....................................................................................................................................... 34
Challenges and Disconnects .......................................................................................................................... 38
Recommendations ........................................................................................................................................ 61
Conclusion ..................................................................................................................................................... 70
Index of Figures ............................................................................................................................................. 72
Appendices .................................................................................................................................................... 74
Bibliographic Essay ...................................................................................................................................... 137
About the Authors ...................................................................................................................................... 143
About UNICON ............................................................................................................................................ 145
7 | P a g e
How the Rankings Work
As the FT itself acknowledges, there is confusion in the marketplace about how these rankings work.
“We spend one day a week on average responding to requests, meeting with business schools, speaking at events to explain the rankings. Those who come to see us want to know, inevitably, why they dropped in the rankings, etc. We can’t really tell them more than what is in the data, which they have complete access to…. It still seems that many schools do not understand the rankings, even with all our efforts to be transparent.” – FT Interview6
Some of this confusion stems from the inherent complexity of the methodology. Although the FT explains the methodology each year in a short article, there is a lot that goes into it, as you will see. But not everything is shared. While the FT shares a fair amount of data on its website, they do not disclose the actual surveys, survey data, or all of the calculations behind the rankings. As they explained to us, the rankings cannot be re-created using publicly available data.7 So, while we will do our best in this section to elaborate upon the FT’s existing explanations, important aspects of the ranking will still remain a “black box” and will be further explored in later sections.
The FT EE rankings measure providers of “non-degree courses for companies and working managers.” There are three rankings: a custom program ranking, an open enrollment program ranking, and a combined ranking for schools that offer both types of programs. The rankings are published annually in May, both in a magazine that supplements the FT’s newspaper as well as online. Producing the rankings requires nearly six months of work on behalf of both the FT and the schools.
The rankings methodology is based upon survey data from custom program clients and open enrollment program participants as well as organizational metrics provided by the schools themselves. The steps for creating the rankings, which we will outline in detail below, are complex.8
Eligible Schools Chose to Participate
In December, the FT notifies schools of the upcoming ranking process. The FT does not recruit schools to participate, but rather maintains a list of schools that have “registered their intention to take part and are eligible.”9
6 The full interview with the FT appears in Appendix 22: FT Interview Transcript. 7 Ibid. 8 For an overview of start/end dates for all FT business school rankings, see Appendix 13: Financial Times Surveys Schedule. 9 FT email correspondence, June 4, 2015.
8 | P a g e
The FT has two main requirements for eligibility. First, schools must offer non-degree programs that generate at least US$2 million in revenues in the prior year for the program areas in which they wish to participate (open or custom). Revenues are adjusted based upon “average dollar currency exchange rates” for the given year.10 Second, the schools must also be EQUIS or AACSB accredited.11
The total number of schools eventually included in each ranking is based upon the number of eligible schools that elect to participate and have a sufficient response rate from their custom client and open program participant surveys.12
Schools Nominate Programs and Provide Contacts
By February, schools must nominate programs for the FT to survey and provide contact information for those surveys. In the case of open enrollment programs, schools must nominate at least one, and up to two, general management programs and at least one, and up to two, advanced management programs. The general management programs are targeted at mid-level participants and last at least three days. The advanced programs are targeted at senior level participants and last at least five days. Both of the programs must be general in nature (i.e., marketing or technology programs should not apply). All of the emails of the participants who attended the selected programs in the past year must be provided, with a minimum of 100 participants (at least 50 from each program).13 At least 20% of open participants surveyed must respond.
In the case of custom programs, a minimum of 20 clients is requested. However, exceptions are sometimes made and only five clients need to respond for a school to be included in the results. Contacts for custom programs are the individuals in the company with oversight for the program (normally someone from Human Resources), not the program participants. Companies that contract with a school
10 Note that this is in contrast to the Purchase Power Parity or PPP adjustment the FT makes in its MBA rankings. As they explained to us, “We use PPP rates in our [MBA] rankings since we need to compare salaries. However, this is not the case in our criteria for [EE] participation. We do not rank schools on their revenues. So this is why we use the exchange rate in that case.” FT email correspondence, June 26, 2015. 11 Note that accreditation is provided at the school level and is based on degree programs. Executive education programs and offices are not evaluated in the accreditation process. The FT’s accreditation requirements have evolved over the years. In the past, schools with AMBA, AACSB and EQUIS accreditation were eligible. In 2011, the FT dropped the AMBA accreditation, giving AMBA schools until 2015 to meet the new accreditation standard. Additionally, the FT has allowed two organizations without accreditation to stay in the rankings. These are Duke Corporate Education, which has been in the ranking since 2001 (and is technically a separate organization from Duke University’s Fuqua School of Business but is, today, owned by Duke University) and the Center for Creative Leadership, which entered the ranking in 2002. While there are many non-university providers of executive education in the market, to date no others have been included in the FT EE rankings. 12 At the time of this report’s writing, there were over 150 EQUIS and over 700 AACSB accredited schools. Exactly how many offer executive education is unknown. However, for perspective, UNICON has 110 members as of June 2015 and about half of them participate in the FT EE ranking. 13 See Appendix 21: FT Criteria for Inclusion – Executive Education Programs for additional information.
9 | P a g e
for multiple custom programs in a year may only be nominated once and complete one survey for that school, although sometimes exceptions are requested and made.14
The FT Surveys Open Participants and Custom Clients
In late February, the FT sends out the surveys to the contacts provided by the schools.15 The aggregated responses to these surveys represent 80% of the respective ranking. The surveys are approximately 30 questions in length and use a ten-point scale for a given question. There are two versions of the survey, one for custom and one for open enrollment. While the FT does not disclose the actual surveys, we spoke to someone who had taken the survey before. It was described as a “fairly standard and very professional” questionnaire asking individuals to rate the school on various aspects of its programs using a ten point scale, with 10=excellent and 1=poor.16
The FT also asks its respondents to weight the importance of the questions in the survey, a unique approach (as they say, they are “the only ranking of business school programmes that uses this approach”).17 This is done by grouping the approximately 30 questions under 10 criteria, per Figure 1 below. These same criteria are ultimately reported in the rankings. In the surveys, respondents rate the importance of the each criterion, also, as we understand it, using a 1 to 10 scale (presumably with 1= not important and 10=extremely important). Criterion importance ratings are then averaged across all responses for all schools received by the FT that year. This average is shown as the numbers in parenthesis in the figure. These averaged importance scores are then used to weight responses to the questions related to that criterion.
Figure 1: FT Survey Criteria and Respondent Weightings As Reported by FT in Executive Education Rankings 2015
Open Criteria Custom Criteria New skills and learning (8.8) Programme design (8.5) Faculty (8.7) Faculty (8.5) Aims achieved (8.6) Aims achieved (8.4) Course design (8.6) New skills and learning (8.3) Teaching methods and materials (8.3) Preparation (8.3) Quality of participants (7.9) Future use (8.0) Preparation (7.7) Teaching methods and materials (7.9) Facilities (7.5) Value for money (7.7) Follow-up (7.3) Follow-up (7.2)
14 Ibid. 15 Online surveying began in 2012. At one point phone interviews were done, but according to the FT this was at least a decade ago. FT email correspondence, July 1, 2015. 16 Interview with past FT respondent, November 2014. 17 Della Bradshaw, “Business School Rankings: The Love-Hate Relationship,” Journal of Management Development, 26, 1 (2007): 58.
10 | P a g e
Food and accommodation (6.6) Facilities (7.2)
In the case of custom programs, the FT also asks clients to classify the program they are ranking by whether it was functional, general, or strategic, with the latter receiving the highest weighting in the final ranking. They also ask for the seniority of the individual responsible for specifying the course, the size of the client organization, and the number of schools with which the client has commissioned customized courses in the last three years. The “larger” any of these are, the greater weighting the related responses get.18 Note that it is not entirely clear how this weighting is applied or how significant an impact it can have on the custom responses – this calculation is not shared with the public.
The FT currently provides questionnaires in English, French, German, Italian, Portuguese and Spanish.19
Schools Provide the FT with Key Metrics
By March, the surveys are closed and at this point the schools need to provide the FT with organizational data – see Figure 2 below. These metrics represent 20% of the school’s ranking score, with the individual weightings set by the FT, per the figure below. While survey criteria weightings fluctuate each year, these weightings stay the same.
Figure 2: Key Metrics Provided by Schools and FT’s Weightings As Reported by FT in Executive Education Rankings 2015
Open Key Metric Custom Key Metrics Repeat business and growth (5.0) International clients (5.0) Faculty diversity (4.0) Faculty diversity (5.0) International participants (3.0) Overseas programs (4.0) International location (3.0) International participants (3.0) Partner schools (3.0) Partner schools (3.0) Female participants (2.0)
For informational purposes only, the FT also gives schools the option of providing their revenues by open and custom. Only a minority of schools does so. If supplied, they are presented in US dollars based upon average currency exchange rates for the given year.
The Rankings Are Calculated
The FT manages collecting the data and tabulating the surveys. They informed us that they run extensive data checks and frequently contact schools to double-check various items.20 Using the data they have
18 At one point this was called a “Sophisticated Buyer” designation, although the FT seems to have dropped that term. 19 FT email correspondence, June 4, 2015.
11 | P a g e
collected, the FT then works with an outside consultant who runs the ranking calculations for them. There are at least four main steps to the calculation.
First, the consultant converts the responses to questions for each criterion into something called a z-score. Also known as a standard score, the z-score is a statistic that shows where a score lies in relation to the average score.21 Using a z-score enables the FT to identify statistical differences in the responses. This is an important point because, if the z-score is the same for two responses, there is no statistical difference between them (Appendix 20: Z-scores Explained). 22
Second, the consultant applies the weightings previously mentioned to all of the data collected.
Third, the consultant uses the previous three years of data to help smooth out any year-by-year swings in the responses. For schools with three years of consecutive data, responses are weighted 40%:33%:27%, with the current year receiving the greatest weight. For schools with two years of consecutive data, the weighting is 55%:45%.
Finally, the weighted z-scores are ranked. The FT does not disclose the z-scores to the public. FT only publishes the rank number itself. However, they reference the z-scores in their footnotes each year to help explain the “spread” of the results. For example, in 2015’s open ranking “some 300 points separate the top school from the one ranked 75.” (Note that in our correspondence with the FT, they clarified that to arrive at these points they “calculate for each school its z-scores for each criterion individually, then multiply these by their respective weight, and only then add them up to get the school's total score.)
In the case of the combined ranking, the FT simply adds the custom and open z-scores together and then re-ranks the schools. Schools participating in only one of the two program rankings are not eligible for the combined ranking.
20 FT email correspondance, June 4, 2015. 21 A Z-Score is a statistical measurement of a score's relationship to the mean in a group of scores. A Z-score of 0 means the score is the same as the mean. A Z-score can also be positive or negative, indicating whether it is above or below the mean and by how many standard deviations. http://www.investopedia.com/terms/z/zscore.asp#ixzz3dXpAhRaR 22 Digging into this issue further, we looked at the average range as well as the average absolute deviation for schools with 5 years of consecutive data. We compared the FT EE Rankings to its own MBA Ranking. And interestingly, the variability for custom was similar to the MBA’s while the variability for open was less. Additional research by experts with background in related mathematical modeling and statistical analysis would be helpful to further understand this dynamic.
Average Range and Absolute Deviation among Schools with 5 Years Consecutive Data Measure MBA Custom Open Average Range 16.8 16.9 9.8 Average Absolute Deviation 5.4 5.2 3.2
12 | P a g e
On occasion, schools are given the same rank (e.g., in 2015’s open ranking, both INSEAD and ESADE were ranked 7th). This is because, as confirmed with the FT, their aggregate z-scores were equal.
The FT Publishes the Rankings
Each May, the FT publishes the rankings in a supplemental magazine (included with its print edition) and online (a PDF of the supplement can also be found online). In addition to revealing each school’s ranking, the magazine also includes articles about industry trends, interviews with Deans, and other related stories.
The FT shares a great deal of the ranking data it has calculated. The ranks for custom and open are detailed in tables both online and in the magazine. Ranks are provided at an overall level as well by criteria. The online data is available going back until 2007 and is downloadable.23 In its magazine, the FT also reports top ten rankings within select criteria (e.g., top ten for Aims Achieved). The FT also provides a Methodology explanation and a Key explaining each criterion and its weighting. 24
Note that the combined ranking is more visible in the magazine than online. One of our interviewees wondered if the FT only published the combined ranking in the magazine. It is published online in the side bar of the website,25 however the combined ranking is not included in the downloadable data tables.
23 We have created an Excel file of all overall data going back to 2001, and this is available on the UNICON website. 24 With the FT’s permission, we have aggregated these and are providing them as a PDF on the UNICON website. 25 For example, http://im.ft-static.com/content/images/ff4391bc-fa33-11e4-a41c-00144feab7de.pdf
13 | P a g e
A 360° View: How the Rankings Are Perceived and Used
“There are things known and there are things unknown, and in between are the doors of perception.”
- Aldous Huxley
Rankings are a means of simplifying – and arguably oversimplifying – a complex executive education landscape. To better understand this, let us pause briefly and look at short descriptions of three, open enrollment, advanced management programs at top institutions:
AMP, delivered in partnership with the Wharton Leadership Program, is an immersive five-week executive management program based on real-time business dynamics, economic challenges and opportunities, global trends, and market shifts. Wharton’s AMP faculty brings extensive business knowledge, research and expertise to bear on all aspects of this thought leader training program.26
AMP is a life-altering and career-changing program designed to accelerate your personal and professional growth. Through a fully immersive format, AMP will forever change the way you and your company do business. Each year, the HBS faculty reframes the program to reflect the current challenges and emerging opportunities created by disruptive innovation, socioeconomic trends, and market volatility. For eight uninterrupted weeks, you will explore leadership in customer and product markets, changing geopolitical influences, and current capital markets. You will return to your organization with the skills, insights, and confidence to lead change, drive innovation, and sustain a competitive advantage.27
The Advanced Management Program (AMP, formerly the Columbia Senior Executive Program) helps executives respond to evolving leadership challenges, exposing them to groundbreaking ideas and to a diverse community of global business leaders. The Advanced Management Program is offered in two scheduling options: the four-week program and AMP 2x2. During the intensive four-week residential program, the Advanced Management Program weaves together in-depth sessions on leadership, strategy, and execution to provide this knowledge. Executives in the AMP 2x2 scheduling option begin the spring session with participants of the consecutive week program for the first two weeks then return to work. Between the two sessions, executives continue work on designated personal and business projects, start implementing their learning, receive coaching on their progress, and have access to
26 http://executiveeducation.wharton.upenn.edu/for-individuals/all-programs/advanced-management-program 27 http://www.exed.hbs.edu/programs/amp/Pages/default.aspx
14 | P a g e
webinars with additional content. There also is a virtual component to maintain the class connection and community, and share learning and reflections that reinforce bridging theory to practice.28
Reading these descriptions, do you find yourself crystal clear on which program is best for you? We would hazard a guess, probably not. And, for now at least, there is no website offering the option to “select these programs and compare” as is available for other types of large purchases such as cars or major appliances. Yet select you must. Few people have the opportunity (or funding) to attend more than one of these programs in their lifetime.
In a complex and sometimes confusing marketplace, rankings fill an otherwise unmet need by providing a comparative perspective. But just how well do they serve that need? In this section, we will examine how rankings are perceived and used by the main stakeholders we surveyed for this report: open enrollment participants, learning & development professionals, and business schools themselves. We will also report what a few experts have said.
While rankings are clearly valued by open customers, we will see the rankings are used in a fairly superficial and “check the box” way. Learning & Development (L&D) professionals, on the other hand, use them to identify schools to consider but less so in their decision-making process. However, both groups appreciate and value the rankings.
In contrast, many business schools are quite frustrated. They see the FT EE rankings – with some justification, as we will discuss in more detail in our “Disconnects and Challenges” section – as a poor measure of quality, and experts confirm this view. At the same time, business schools are often the ones that communicate their ranking to customers.
About Our “360 Degree” Methodology
Our data come from both online surveys and extensive interviews.
For open enrollment customers, we surveyed 713 customers, over 100 each from North America, Europe, Asia, and Latin America and over 50 from Africa. We had 10 or fewer responses in Oceania and the Middle East. We also interviewed 5 participants of open enrollment programs.
For “L&D” professionals, we surveyed 279 customers. We should note that the vast majority of these survey respondents were from Europe (231). This is a group that knows the executive education market well. 72% were responsible for selecting providers of custom programs and 71% for selecting open programs. Nearly half had attended programs themselves. We also interviewed 9 L&D professionals from a variety of companies across the globe.
28 http://www8.gsb.columbia.edu/execed/program-pages/details/814/AMP?sourceid=meganav
15 | P a g e
For business schools, we conducted a survey of UNICON’s 110 member schools (see Appendix 2: UNICON Member Schools and Their 2015 Open & Custom Program Rankings for a listing of UNICON schools) and had 86 responses (78%). These schools represented a wide geographic distribution, including schools from Europe, the US and Canada, Asia, Africa, Latin America, the Middle East, and Australia. We also surveyed 8 EE additional leaders who are not currently UNICON members but are EQUIS accredited schools.
Finally, to round out our perspective, we interviewed 14 rankings experts. These included professionals who have run current and past rankings of business schools, consultants who offer advice on how to improve a school’s ranking, social scientists who study rankings, and statisticians.
Across All Groups: A Majority Are Aware of the FT EE Rankings
In terms of awareness, the FT EE rankings are widely watched and followed. We found that a majority of the open participants and the vast majority of L&D professionals were aware of or very familiar with the rankings – 58% and 70%, respectively. We found the awareness strongest in Europe, per the figures below. All of the business schools we spoke with were aware of the rankings.
Figure 3: Open Program Participants: Have you reviewed the Financial Times executive education rankings, specifically?
Figure 4: L&D Professionals: Are you aware of the Financial Times rankings of executive education programs?
Only Europe and US/Canada Data Shown Given n<10 for other regions
33%
34%
36%
57%
71%
57%
55%
51%
33%
18%
10%
11%
13%
10%
10%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
US/Canada (n=42)
Latin America (n=74)
Asia (n=39)
Africa (n=21)
Europe (n=136)
Yes No I'm not sure
18%
23%
45%
61%
14%
11%
23%
6%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
US/Canada (n=22)
Europe (n=218)
Yes, very familar Yes, aware I'm not sure No
16 | P a g e
We also asked the three groups surveyed if they valued rankings for executive education. Interestingly, nearly two-thirds in each group (over 80% in each of the customer groups) valued the practice of ranking EE providers at least moderately or better, as seen in the chart below.
Figure 5: How valuable do you believe rankings are as a source of information in selecting executive education programs?29
However, it was also clear that business schools were more critical of rankings than their customers. Whereas nearly 60% of customers generally thought that rankings were “quite” or “extremely valuable,” only 28% of the schools surveyed felt the same. We will take a deeper look at this ahead.
We also asked customers if they valued rankings more when they did not know a school well versus if they did know it. The perceived value of rankings was higher if they did not know the school, and it was surprising just how much higher. The percent answering “quite” or “extremely valuable” nearly doubled.
29 Note: in the open and custom surveys, we asked this question in two parts, for schools you “know well” and for schools you “do not know well”. The data presented in this table is an average of those two responses. Detailed data is in the chart below and in Appendices Appendix 3: Open Enrollment Program Customer Survey Data, Appendix 4: Custom Program Customer Survey Data, and Appendix 5: UNICON Business School EE Survey Data.
2%
15%
21%
26%
44%
37%
33%
27%
26%
31%
12%
12%
8%
2%
4%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Business Schools (n=93)
L&D Custom Customers (averaged*, n=264)
Open Enrollment Customers (averaged*, n=484)
extremely valuable quite valuable moderately valuable slightly valuable not at all valuable
17 | P a g e
Figure 6: How valuable do you believe rankings are as a source of information when selecting an executive education program?
Specifically, we found that the rankings have a much higher value, both for L&D professionals and open enrollment participants if they did not know the school.
Open Enrollment Customers “I trust the FT…I think the [rankings] from the FT is good enough, and if it wasn’t, I am sure some business schools would have already made a proposition.” – Open Enrollment Participant Interview
Most open enrollment customers are individuals who are looking to advance their careers through professional development. Generally, they select and then attend a program of their choice or in collaboration with their manager, versus working through a central HR or learning function. While in the past print brochures were a mainstay, today schools generally market to open enrollment program attendees through the schools’ web sites and other online channels including emails, press releases and social media.30
We found that while the rankings are widely known and used in this marketplace, they only play a supporting role when it comes to final decision-making.
30 See Social Media snapshot in Appendix 6: Social Media and Executive Education Rankings.
2
20
59
42
162
17
90
24
89
137
138
213
75
75
30
94
47
177
72
73
30
28
49
13
91
22
35
1
7
10
2
33
5
9
2
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Business Schools
L&D Custom Customers (know well)
L&D Custom Customers (don't know well)
Open Enrollment Customers (know well)
Open Enrollment Customers (don't know well)
OE - Spanish Survey (conozco bien)
OE - Spanish Survey (NO conozco bien)
Extremely valuable Quite valuable Moderately valuable Slightly valuable Not at all valuable
18 | P a g e
An early step in the process
Many open enrollment program customers check rankings before selecting a program to attend, overall 45% in our surveys.31 However, this amount varied significantly by region as seen in the figure below (Note that Mideast and Oceania data excluded due to small sample size).
Figure 7: Prior to selecting an educational program to attend, did you check rankings? (n=692)
Of the ranking sources mentioned by name, The Financial Times was the most frequent reference (104), followed by The Economist (26) and Business Week (15), though it should be noted that The Economist ranks only MBAs and EMBAS, not EE.
A supporting role, not a determining one
Digging deeper into the interviews and survey data, we learned that open enrollment program participants appreciate the “holistic view of the programs” that the FT provides.32 They also appreciated the “insights into curriculum” and “trends.”33 In survey open program response data, we heard a variety of uses for the rankings, ranging from “supporting my decision” to “endorsing my choice” to “using the list to decide which schools to approach.”
However, when open customers were asked how they use the FT rankings, we learned that the ranking clearly plays a supporting role to the decision, not a determining one. A surprisingly large 55% indicated they had already chosen a school before they checked the FT’s rankings. In addition, only 24% indicated
31 For data supporting this and the following discussion of survey results, see Appendices Appendix 3: Open Enrollment Program Customer Survey Data, Appendix 4: Custom Program Customer Survey Data, and Appendix 5: UNICON Business School EE Survey Data. 32 Open Enrollment Participant interview 33 Open Enrollment Participant Interview
23%
28%
34%
32%
59%
68%
60%
60%
59%
34%
10%
12%
6%
10%
7%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Latin America (n=228)
Asia (n=101)
Africa (n=53)
US/Canada (n=104)
Europe (n=206)
Yes No I'm not sure
19 | P a g e
that they used the list to find a school. And once the decision was made, only a small 18% used the rankings as information to get support to attend a program.34
Figure 8: Which of the following statements describe(s) your process in reviewing the Financial Times rankings? (Select all that apply) (n=147)
This limited use of the FT EE rankings seems to parallel a piece of research we requested from Catalyx, a Swiss market intelligence firm that specializes in crowd-sourcing. (See Appendix 6: Social Media and Executive Education Rankings). Catalyx looked at social media conversations and searches for executive education rankings in early 2015. While searches for the rankings were extensive, online conversations about them were relatively low. From what we can tell, the EE rankings do not seem to provoke many online debates or discussions.
The open customer is definitely not afraid to conduct his or her own survey. 49% of open customers said they also looked at other ranking for schools, such as for MBA programs, as shown in Figure 9 below. Interestingly, a large number looked into the ranking methodology, with 46% reading the detail about the rankings criteria and methodology. However, only 20% looked at detailed data on specific ranking criteria.
34 Note that rankings do contribute to perceptions of reputation, and reputation does influence decision-making. As one of our business school interviewees noted, “Our research with client groups and prospective clients shows that a lot of decision-making has to do with reputation. Ranking is one way to demonstrate this although it may not accurately show the whole picture and is a bit of a game.”
55%
29%
24%
18%
0% 10% 20% 30% 40% 50% 60%
I had already chosen a school/program andscanned the list to see its position
I used the list to find ranked schools in a specificgeographic region
I used the list to find a school
I used ranking information to getsupport/permission to attend the program
20 | P a g e
Figure 9: Which of the following statements describe(s) your process in reviewing the Financial Times rankings? (Select all that apply) n=147
From the data, the FT EE rankings appear to be most important at the front end of the selection process for open customers, both in creating awareness and, to some extent, in supporting decisions. Given that so many open enrollment customers search for and check rankings, it suggests that rankings fulfill an important need.
Although they seem useful, it is worth asking if open program candidates/customers are getting enough information when they look at the rankings. There is good reason to ask this question as “there is growing evidence that consumers… respond differently to information when it is visible or salient than when it is shrouded or opaque.”35 As we will discuss later in this report, many in the EE industry believe that the content behind the FT EE rankings is less transparent than it could be.
Next, let us turn to the L&D professionals.
Learning & Development Professionals While the decision for an open enrollment program generally rests with the individual who will be attending, the customers for custom programs are generally L&D professionals in corporations with responsibility for executive development. These L&D professionals, like open enrollment participants, are surveyed by the FT to develop the rankings. It is important to note that L&D professionals work closely with senior executives in developing programs. However, because the senior executives are not interviewed directly, one could argue that the FT rankings fail to completely capture the perspective of the company. Indeed, in some cases, the non-L&D executives might value certain criteria or have a different vocabulary around executive education in comparison to their L&D counterparts. However, while we think this is an interesting area for future exploration, for the purposes of this report we
35 Michael Luca & Jonathan Smith “Strategic disclosure: The case of business school rankings.” Journal of Economic Behavior and Organization, 112(2015): 18.
49%
46%
20%
0% 10% 20% 30% 40% 50% 60%
I checked several rankings for the school - not justexecutive education but how other programs and
degrees are ranked
I read the detail about rankings criteria and how theschools are ranked
I looked at detailed data on specific ranking criteria
21 | P a g e
assume, like the FT, that L&D leaders and their senior executive teams are aligned when evaluating business schools.
We found that L&D professionals are highly sophisticated customers and that they have a variety of uses for the rankings. However, like open program customers, they saw the rankings as playing a relatively minor part in their decision-making. 36
A sophisticated customer and a variety of uses
According to our survey of L&D professionals, there appears to be a fair amount of importance attached to the FT EE rankings. As shown in Figure 10, ranking in the top ten is “extremely” or “quite” important for over 50%. And just being ranked somewhere is almost equally as important – over 40%. Unlike open enrollment customers who look at the rankings only occasionally and for the purpose of making a single decision, at least some custom program customers are watching the rankings over time.
“If a school drops, I will ask myself what happened. Maybe it’s not fair that I do that, but I do.” – L&D Leader Interview
We also asked about accreditation, another quality indicator. Accreditation was seen as less important than the FT EE rankings. Amusingly, in one of our interviews, a former business school dean recounted a dinner recently with his son and his friends who had all just graduated from one of the top fulltime MBA programs. They all knew perfectly the ranking of their MBA and were quite proud of it. But none of them knew what accreditation was or that it was a pre-requisite to most rankings.
36Here our data aligns with that presented by Kim Maybar-Plaxe, Mark Allen, and Annick Renaud-Coulon about the importance of FT rankings in the corporate university decision-making process. Using a smaller sample, they also found rankings were not a major factor in decisions. “Minding their Business by Flexing our Minds: A Guide to Corporate University Partnerships”, UNICON research report, July 25, 2014, 16-19. (Their survey did not include a question on awareness).
11%
8%
5%
8%
41%
38%
32%
19%
34%
35%
39%
33%
12%
15%
19%
21%
2%
5%
5%
19%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%
If the school is in the top 10 of the FT ExecutiveEducation rankings.
If the school is in the FT Executive Education rankings(whatever position).
If the school has risen or dropped recently in the FTExecutive Education rankings.
Whether the school is accredited by AACSB or EQUIS
Extremely important Quite important Moderately important Slightly important Not at all important
22 | P a g e
Figure 10: When considering providers of executive education, how important are the following? Check all that apply (n=211)
While our data show that custom program rankings are generally seen as important, many of our interviewees were quick to downplay the importance of rankings in their decision-making. They told us that rankings informed their process but did not determine a final decision.
“Normally I don’t pay much attention. I just look to see if a school is on the list and then I move on….We know the big schools.” – L&D Leader Interview
“It’s never been something that I used in a decision. It might have navigated me where to look or it might have helped justify something after the fact, but it has never been core to the decision.” – L&D Leader Interview
“If you’re good at your job, you know who is good, and you don’t need the rankings for this. We certainly don’t choose based on FT rankings...” – L&D Leader Interview
As we conducted additional interviews with L&D professionals, we uncovered more ways in which they used the FT rankings. Similar to open enrollment participants, rankings appear to be most important in early identification and in supporting a selection decision.
“When we’re starting the search for a partner for a new program, we start with the rankings to see who should be invited to the RFP. It’s a starting point, but you won’t win based on the rankings.” – L&D Leader Interview
“When we decided to reinvent our corporate university, we looked at the FT rankings to figure out which 5 schools we wanted to talk to, but after that our decisions were based on how well the schools were able to build a relationship with us.” – L&D Leader Interview
At the same time, the rankings are also cited as being very helpful for people who are earlier in their careers or newer to L&D.
“I’m not really using the rankings these days, though I know the guy I hired last year to run our corporate university does. He came from purchasing and knew nothing about executive education. He found the FT rankings on his own with Google.” – L&D Leader Interview
“Yes, of course I know the FT rankings. I’ve been 10 years in executive education. When I moved from strategic consulting to an L&D role I was completely lost, and the FT was a very valuable resource at that time. Through the FT I learned what the important criteria were. For me, it was a way of seeing what the “global norm” was in executive education. I kept it right on my desk so that I could easily refer to it as needed.” – L&D Leader Interview
In addition, the rankings can be helpful in identifying schools in regions outside of the US and Europe, with several interviewees mentioning Asia in particular.
23 | P a g e
“I like to look at emerging markets to see if any schools are showing up.” – L&D Leader Interview
“After a while I didn’t need the ranking for the big names in the US and Europe. But [the ranking] was still vital for us when we started to work with schools in Asia. For Asian schools, I had no idea who was doing what, so I went to the FT rankings.” – L&D Leader Interview
“Possibly we’d look at the rankings for Asia, as we don’t know the schools there as well, and don’t yet have relationships” – L&D Leader Interview
Rankings are also used to support selection decisions internally, especially given the high cost of top executive development programs.
“I find them useful with procurement. They help me justify why we’re choosing this particular provider and why it costs so much.” – L&D Leader Interview
“We use rankings mainly in our internal communications when we’re describing a program to the audience, to let them know we’re working with a top-ranked school.” – L&D Leader Interview
L&D professionals were not only savvy about how to use the FT EE rankings but also about the marketing behind it.
“The rankings are great branding and help with awareness. The head of the FT rankings, Della, is also very knowledgeable about the industry and a good speaker. Her comments carry a lot of weight.” – L&D Leader Interview
“I find out about the rankings from the schools themselves. The schools tell us, not the FT!” – L&D Leader Interview
L&D professionals represent a much smaller community than the customers of open enrollment programs. If they have been in the field for a number of years, they are probably not only aware of the rankings, but, at some point, they have probably completed surveys for the FT as part of the custom program ranking process. They clearly accept and see some value in the ranking – using it in a variety of ways – but they are also not overly swayed by it in their decision-making process.
Business Schools The next perspective in our 360° exploration is from leaders of executive education organizations within business schools. We were interested not only in how they perceive the FT EE rankings but also in how they manage the FT EE ranking process. In our surveys and interviews, we also gathered their critiques and recommendations which are explored in more detail in the “Disconnects and Challenges” section.
As a point of clarification, only 56% or 51 the 94 schools that responded to our survey currently participate in the rankings. As a result, some of the data you see below will be limited to those that participate in the rankings. 29% have never participated, and 14% participated in the past but do not do
24 | P a g e
so now. Among those not participating, 55% have chosen not to participate while the remaining schools do not meet the minimum enrollment, revenue, or accreditation requirements. (Reasons for choosing not to participate range from a low perceived value of the rankings, to concerns about ranking position, and to the validity of the ranking process.)37
Whereas we see similarities in how both open customers and L&D professionals perceive and use the EE rankings, the perception and use by business schools provides some interesting contrasts.
Among participating schools, satisfaction is mixed at best
More than 40% (17 of 41) of schools participating in open enrollment rankings and close to 60% (26 of 46) participating in custom program rankings are not even moderately satisfied with the rankings. Out of 51 schools, not one school reported being extremely satisfied.
Figure 11: Overall, how satisfied are you with the way your programs are ranked by the Financial Times? (n=51)
Some dissatisfied schools would like to withdraw, but feel they cannot.
“It’s hard to withdraw without becoming the story…If we could withdraw quietly and if other schools did that, we might too.” – Business School EE Leader Interview
And published analyses of the rankings process agree.
“…. opting out is considered tantamount to suicide by many business school deans. A few schools can opt out because they have top-of-mind brand recognition. These schools have nothing to gain and everything to lose from the rankings. Although their nonparticipation may appear as a valiant “protest,” it is very unlikely to thwart the rankings. Ironically, one unintended
37 Business school data from this point forward includes both UNICON schools and 8 schools from EFMD.
9
14
8
12 12 11
12
9
0
5
10
15
Open enrollment Custom programs
Num
ber o
f Sch
ools
not at all satisfied slightly satisfied moderately satisfied quite satisfied extremely satisfied
25 | P a g e
consequence of their protest may be to heighten the controversy surrounding the rankings, thereby increasing reader interest and stimulating greater sales for the media involved.” 38
“The [survey] results show that competition also compels participation in the rankings…. if competing schools, and hence schools to which you compare your own school, are in that group, your school is suddenly outside if it does not participate.” 39
Furthermore, some schools question the value of the FT EE rankings for their customers.
“We’re asking our clients to do something that has no benefit to them, and we’re adding credibility to a ranking that we don’t subscribe to.” – Business School EE Leader Interview
“It certainly provides a level of visibility that can be beneficial. However, in our niche rankings aren't that important.” – Survey open response, Business School
Those who are currently participating have much more favorable view of rankings in general.
Figure 12: How valuable do you believe rankings are as a source of information for participants when selecting an executive education program? (n=91)
We are going to explore the reasons behind school dissatisfaction in the next section. However, when you look at the data regarding whether or not rankings, in general, are valued by executive education leaders, opinions vary based upon whether the school is currently participating in the FT rankings, has never participated, or has participated in the past but not currently.
38W. H. Glick, "Rain Man or Pied Piper? Moving Business Schools Beyond Media Rankings with Mass Customization and Stakeholder Education." Academy of Management Perspectives 22, 1 (2008): 18. 39 Linda Wedlin, Ranking Business Schools: Forming Fields, Identities and Boundaries in International Management Education (Cheltenham, UK, Edward Elgar, 2006): 107.
1
1
0
16
6
2
19
8
3
11
12
5
4
1
2
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Yes, currently
No, never
In the past, but not currently
extremely valuable quite valuable moderately valuable slightly valuable not at all valuable
26 | P a g e
On the plus side: Satisfaction with having a benchmark and with the FT
We learned through the survey that schools value the FT EE rankings because they set “a benchmark…with a strong international blueprint.” Benchmarking, in particular, is valued, and this was reinforced in a number of comments.
“When we talk with companies about their executive development needs, they almost always want to know what other companies are doing. Exec Ed rankings provide confidence to potential clients that we have done valuable work with other companies.” – Survey response, Business School
“The ranking demonstrates that we are the strongest regional player and that we are recognized globally. This helps us attract participants, who might otherwise travel to Europe or the US for training, and helps justify our premium pricing.” – Survey open response, Business School
“I can use this information to differentiate myself from other regional competitors since no one else in my region is even ranked by FT. I don't think it is necessarily the winning criterion for clients but it is something.....” – Survey open response, Business School
We heard from our surveys of Business School leaders that the FT is “doing excellent work and it’s a good reference for international customers.” Some felt that “the process is managed well; the communication of the results is professional and clear.” At least one person felt that the FT “uses a rigorous and excellent rankings methodology.” Another felt “the criteria are really good, and when you take serious action to improve in most of the areas, then your executive education improves.”
It was also pointed out by survey respondents that “the FT is a very credible, very strong international brand” and “product differentiation within the Exec Ed industry is very poor (from the client point of view) and clients are hungry for third-party mechanisms which help rank, rate and review programs and providers...and ultimately, help them choose.”
If a school is participating, you are likely to hear about it
“We promote our ranking by the FT on every occasion possible.” – Survey open response, Business School
In spite of mixed satisfaction with the rankings, the business schools that participate are also responsible for selectively communicating ranking results.40 In fact, 82% of ranked schools (41 of 50) reported that they communicate the rankings results in some way. Per the figure below, 76% share it with faculty and other internal stakeholders at the school. Nearly two-thirds share it in email newsletters and marketing materials. Only a small percentage does NOT communicate the results in some fashion, less than 20%.
40 How rankings are disclosed by schools is discussed in Luca, & Smith. (2015). It is also noted in Bradshaw, (2007).
27 | P a g e
Figure 13: Do you communicate your Financial Times rankings results? (Select all that apply) (n=50)
Our interviews revealed that many schools see a strong marketing and brand value attached to the rankings. They are aware that their customers look for and use this information.
“For open programs, we know that reputation is one of three crucial factors for individuals to choose a program. The others are faculty and cutting-edge content…. We assume that rankings are a big part of reputation.” – Business School EE Leader Interview
“Custom rankings are used to ratify a decision to work with a school – it is one of several factors including clients, accreditation, awards – they are used by senior HR executives and other executives….Rankings also feed into the general awareness of a school.” – Business School EE Leader Interview
Many schools are also using the rankings frequently in their marketing, as both our interviews and a quick scan of press releases and websites can confirm.
“We brag about it all of the time. We talk about having the Triple Crown [three accreditations], plus the FT. We even have the four logos on our letterhead.” – Business School EE Leader Interview
“If we didn’t mention the rankings in all our marketing material, I don’t think customers, alumni, or current participants would have any idea of where we were in the rankings. They get their information from the schools, not from the FT.” – Business School EE Leader Interview
“The fact we have the executive education ranking helps the rest of the school, rather than the other way around. Everyone talks about us being [XX] in the world, but no one remembers it is only for executive education open enrollment programs. It’s not that we don’t say it, but people just remember the number. Our customers probably don’t read the FT and they’re certainly not
76%
64%
64%
58%
54%
44%
18%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Yes, internally to stakeholders such as faculty or…
Yes, in email newsletters or marketing
Yes, internally to staff
Yes, on our web site
Yes. in social media
Yes, in print collateral
No
28 | P a g e
going to go online to look at the rankings. They hear from us and take our word for it.” – Business School EE Leader Interview
“Of the 100 schools listed in the Financial Times annual ranking of top global MBA programmes, published in January 2006, 96 refer to their position in rankings on their websites, through press releases, or in brochures.” 41
Schools are selective, however, in what they reveal, often communicating only the data that best represents the story they are trying to tell.
“When looking at the results we focus even more on the sub-rankings, as these usually make us look better.” – Business School EE Leader Interview
“Depending on the outcome – we will have an appropriate marketing campaign.” – Survey open response, Business School
One use of the FT EE rankings by schools is to focus on placement within a region, even though the FT does not specifically publish these results.
“I thought we needed it to be recognized in the region….I believe it has helped our sales significantly, although it is difficult to be sure. Every time our Dean or I speak in public we mention it. We say that we are in the world rankings and that we’re the only world-class player in [our region], as evidenced by the FT.” – Business School EE Leader Interview
The FT does not do regional rankings and does not publish comparisons of rankings. However, one entrepreneurial school did not let that stop them.42
Some suggestions for publication
Our survey revealed some suggestions related to publication, although overall this was not an area of major concerns. One creative suggestion was to display the results in “dynamic infographics.” Another respondent stressed it “would be much better to educate clients on how best to leverage and utilize
41 Bradshaw (2007): 54. Exceptions were HBS, Wharton, INSEAD, and MIT Sloan.
42 See http://www.hankensse.fi/quality-and-ranking/
29 | P a g e
executive education…then allow clients in a local market to judge quality of their provider – have them draw judgements based on standards.”
Helping clients differentiate beyond open and custom programs, including providing “sub-rankings to compare similar programs (in terms of staff and financial resources) as well as having separate rankings “by region and by size of schools to be more comparative between similar institutions” were also mentioned.
The importance of position
We heard different opinions about the importance of relative positioning in the rankings. For some schools, it was critical to be in the top 20 or top 10.
“Some schools choose not to participate because of the risk they won’t be in the top 20.” – Business School EE Leader Interview
“What is probably most important is that you are in the top 10. Not whether you are 3 or 5. Being in the top 10 is really important.” – Business School EE Leader Interview
Others, though, were more influenced by the idea that the Financial Times rankings are global and were satisfied with being on the final list, even with a lower position.
“Even if we’re XXth, that is still XXth in the world, which is very good in [our region].” – Business School EE Leader Interview
“We can honestly say we’re world class” – Business School EE Leader Interview
“Rankings are a stamp of credibility. It says something that we are in the rankings.” – Business School EE Leader Interview
“It is very important to us, especially custom.” – Business School EE Leader Interview
“The [survey] responses rather indicate that participating in rankings is more about putting oneself in a group, and perhaps even in a new and international group. This suggests that the rankings are a particular kind of marketing effort that promotes a particular aspect of the school, which is not related directly to academic reputation and academic performance. Rather the rankings signal belonging to a top group of schools internationally, and recognition as an ‘elite institution’, and hence signal distinction from other schools.” 43
43 Wedlin (2006): 107.
30 | P a g e
The importance of being ranked, as perceived by business schools, definitely varies by region, with less importance given in the US/Canada and Latin America, and more importance given by schools in Europe and the Middle East.
Figure 14: How important do you feel it is for executive education programs to be ranked by an organization such as the Financial Times? (n=90)
And, while regional reputation may be a positive factor for some, there are also schools in both Oceania and Latin America that cited geography as a reason not to participate.
“No, we have not considered participating… both FT and others are too US/Euro centric for us to be able to be fairly represented. We are a small school (comparatively) in terms of numbers, and remote geographically-speaking.” – Business School EE Leader Interview
“The main reason we don’t participate is the process is more complicated than we think the value add is. Our students don’t care if we participate. In our application process we ask, ‘how did you find the program?’ And one of the responses is by ranking. The response to this is ‘none’.” – Business School EE Leader Interview
“We don’t participate in FT because in Latin America it is less important – the programs are mostly American and the comparison between schools will be difficult.” – Business School EE Leader Interview
By contrast, for schools in emerging markets such as Asia that are also looking to engage clients in more mature markets, including the US and Europe, not being in the rankings is seen as a challenge.
“There are only two rankings that corporate clients for custom programs look at, Business Week and FT, and, since the real growth in executive education is happening globally, because the FT is
21%
40%
20%
5%
20%
4%
0%
46%
40%
40%
33%
40%
13%
50%
18%
0%
20%
33%
10%
47%
0%
11%
0%
0%
19%
30%
28%
50%
4%
20%
20%
10%
0%
9%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Europe (n=28)
Middle East (n=5)
Africa (n=5)
Asia (n=21)
Latin America (n=10)
US/Canada (n=47)
Oceania (n=2)
extremely important quite important moderately important slightly important not at all important
31 | P a g e
international, it is stronger....Not being in the rankings makes my job very difficult. For executive education in the States, people look for this information.” – Business School EE Leader Interview
This was reinforced in our interviews with companies that looked to the rankings for new regions, including Asia.
It is more than just marketing – rankings impact programs and delivery
It is important to note that the impact of the FT EE rankings goes well beyond marketing. In fact, 29 of the 51 (57%) surveyed business schools (UNICON and EQUIS), currently participating in the rankings, reported making programmatic changes based on the rankings process, methodology, or results.44
“We use rankings as a form of customer feedback – it helps us analyze where we can improve the service to get higher up.” – Business School EE Leader Interview
“We continually evaluate the FT data as part of our ongoing effort to improve quality, even within our most powerfully-performing programs.” – Survey open response, Business School
Schools work to identify changes that improve both rank and program quality. At the same time, the schools that are currently making changes caution against doing so for the sake of rankings alone.
“We’d be very cautious of making any change just based on the rankings – the need for this change came through other areas as well. You could be 20th on something, but there might be a very small margin between you and #s 1-19 because it is a very competitive criterion – there has to be other research and data to support making a change, not just the rankings.” – Business School EE Leader Interview
As we will see in the next section, one major frustration for some schools is that they feel their quality metrics and the rankings metrics are not aligned. On the positive side, several interviewed schools said that post-program follow-up resulted in their implementing changes which they felt had a positive impact on both their programs and ranking.
One interesting survey finding is that few schools survey their participants to see if they use the rankings. In fact, 58 of 88 business schools (including both those currently participating and not participating) indicated they did not survey participants to see if they used the rankings to select programs. 17 schools asked about rankings in the application, 16 included rankings in market research, and only 2 schools asked about rankings in exit surveys.
44 See Appendix 5: UNICON Business School EE Survey Data.
32 | P a g e
Figure 15: Do you survey your participants to see if they use rankings to select your programs? (Select all that apply) (n=88)
We believe schools have an opportunity to obtain more and better data, particularly using exit surveys, that could inform their use of the rankings for internal improvement and also, potentially, to improve the school’s position. Specifically, aligning some of the exit survey questions to criteria used by the FT could provide additional insight into areas of improvement.
A fairly time consuming, but not overly difficult process
Collecting data and managing the rankings process internally is a fairly time consuming process for schools. The majority of schools spend 5 or more days each year on the FT EE rankings.
Figure 16: Approximately how much time is spent each year by your organization on managing the Financial Times executive education rankings process? (n=49)
At least 80% use a Customer Relationship Management (CRM) system, and over 85% are at least moderately comfortable sharing data with the FT (See Appendix 5: UNICON Business School EE Survey Data).
Although the rankings can be time consuming, the vast majority of schools view the process as a “not at all” or only “slightly” difficult task. The gathering of internal school data appears to be the most difficult part of the process.
58
17 2 16 0
102030405060
No Yes, informally, in ourapplication
Yes, in our exit survey Yes, as part of our internalmarket research
Num
ber o
f Sch
ools
4 4 7 9
25
0
5
10
15
20
25
1 day 2 days 3 days 4 days 5 or more days
Num
ber o
f Sch
ools
33 | P a g e
Figure 17: How difficult is it for you to collect the following data as part of the Financial Times executive education rankings process? (n=50)
It is clear that the rankings take time, resources (CRM), and focus.
A range of approaches to managing the process
Our survey also asked schools who in their organization was responsible for managing the rankings process internally, and we found a range of approaches. 35% of the schools surveyed and participating in the rankings indicated the marketing director and marketing department had the responsibility. 33% indicated it was either the executive director (12%) or a shared responsibly among senior leaders and managers (21%, frequently executive directors, program managers and marketing working together). Another 10% of schools delegated the process entirely to program managers, which is probably not a good way to manage the process given its complexity and nuances. The remaining 21% relied on a range of offices, including Academic Services, Accreditation officers, and external relations. Three schools had dedicated roles/offices focused solely on rankings.
The shared responsibly approach was evident in our interviews. In our research process it sometimes took some time to clarify who we should speak to. Often, different people were responsible for different parts of the process.
“I have forwarded your email to our Dean of Qualification Programmes, Head of Accreditation, Head of Corporate Communications and our Corporate Business Development Director. I have asked them to recommend who the best person/people would be for you to talk to.” – Business School EE Leader Interview
It is worth noting that this is a process that does get attention internally. Over 66% of schools participating in the rankings reporting that the internal perception of the FT EE ranking is “extremely” or “quite” important (See Appendix 5: UNICON Business School EE Survey Data).
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Custom program contacts
Open enrollment program participant contacts
Institutional data
not at all difficult slightly difficult moderately difficult quite difficult extremely difficult
34 | P a g e
Rankings Experts We spoke with a variety of experts in the field of rankings, including professionals who have run current and past rankings of business schools, consultants who offer advice on how to improve a school’s ranking, social scientists who study rankings and some statisticians.
Do not confuse a media rank with a quality metric
The experts we spoke with were even more direct than the schools about the limitations of rankings in general (not specific to the FT).
“It has to do with reputation and status, but very little to do with quality….The ranking does function for improvement because scales have been set. If we improve those things, it can get better by the scale set by the rankings and that might lead to actual improvement. But, if you define it as improved learning or teaching assessment, then it doesn’t define these.” – Expert Interview
“Many of the things they measure have nothing to do with quality (for example, percent international faculty).” – Expert Interview
Regarding the FT EE rankings, one critique we heard over and over again, from both schools and experts, was that the Financial Times executive education rankings are not keeping up with evolving business needs.
“The customer of today is very different from five years ago. They have a totally different way of working. Do the things that a particular school does help these executives to grow their businesses?” – Business School EE Leader Interview
And, scholars have pointed out that while rankings do not reflect absolute quality, they do reflect a market response and for that reason should be considered thoughtfully.45
“They measure what they measure.”
One of the primary challenges faced by schools is that they want both to improve their ranking and improve their programs. Depending on the school’s strengths, strategy, and resources, these are not necessarily aligned.
45D. C. Jain and M. Golosinski,“Sizing Up the Tyranny of the Ruler.” Academy Of Management Learning & Education, 8, 1 (2009): 104.
35 | P a g e
“They are too much of a black box. To the extent a school wants to use them for improvement, and also given the amount of time it takes a school to respond, it would be great to understand more.” – Expert Interview
“Obviously it doesn’t hurt to have a good ranking, and if you do, you exploit it, but that shouldn’t influence how you run your programs or your school.” – Expert Interview
Some of the lack of transparency is related to complexity. The FT has a thorough and detailed process, which is both a strength and a weakness. This is particularly true for corporate customers who will not invest the time to understand the rankings to the degree that the schools will.
“When I teach the rankings, I put the entire table up and I ask: Do you think this is transparent? Even if you spend an hour you won’t see it as transparent.” – Expert Interview
“Indicators, measures and methods of compiling, computing and interpreting ranking data should be fully transparent to all parties.” 46
There is also no easy way for customers of executive education programs to compare a range of rankings or to understand how differences in ranking positions will translate to a better or worse experience of or investment in their education.
“More sophisticated customers understand there is a difference in being in the top 10 or 15 and number 30. But, where you end up depends on many factors.” – Expert interview
While the FT goes to great lengths to publish a significant amount of detail about the process and their data, there is a suspicion from some critics that, in the end, if everything were disclosed, the rankings might not stand up to professional scrutiny.
“They should all publish all of their data, but they won’t because there aren’t statistically significant differences.” – Expert Interview
In the end, “playing the rankings game” means that a school needs to optimize based on the criteria being measured and the measurement process that the FT follows. This has pluses and minuses for both the schools and the FT. Customers who understand the process more clearly are not necessarily impressed.
“I had a conversation with [a ranked school] a few years ago. The guy there told me that the rankings were so important to them that they put a full team of 6 people on it, to monitor and do what they could to improve on key criteria… In an instant, the rankings lost a lot of credibility for me.” – Business School EE Leader Interview
46S. Marginson, “University Rankings and Social Science.” European Journal of Education, 49, 1 (2014): 50.
36 | P a g e
How to Capture Innovation?
In a previous UNICON research report on corporate universities, innovation was repeatedly cited as a requirement of business schools if they wished to continue to work with companies on custom programs47.
“The school should be a lighthouse of pedagogy. Must have state-of-the-art research, and continuously raise the bar, even by creating new cases. Surprise us with new stuff, please!” – L&D Leader Interview
“We want more innovation than most business schools are willing to offer.” – L&D Leader Interview
However, it appears that the FT EE rankings do not explicitly recognize and reward innovation or non-traditional programs and organizations.
“Today, innovations only come from the small schools, usually unranked, not the top-ranked schools.” – Business School EE Leader Interview
“Non-traditional executive education – firms working the customized space …are pushing the executive education frontiers, and they get no credit for this [because innovation is not included in the rankings criteria]. By not including leadership consultancies in the rankings, you are limiting opportunity for customers, and it also doesn’t push business schools to do better. There is an opportunity to create a ranking – high level – that includes some of these consultancies. It is their biggest hurdle – that they are not in the rankings and there is a lot happening out there. Eventually they will get together in a group and have their own rank list.” – Business School EE Leader Interview
If schools do not innovate because they do not want, partly or in whole, to risk their ranking, they then put themselves at risk of losing clients who want or need innovation. This leads, in turn, to the EE industry as a whole heading towards commodification.
Some experts felt that the remedy for shortcomings of ranking, including the capture of innovations, is to have many different rankings.
“Quoting a friend: ‘there should be enough rankings so that every school can be #1 in one ranking.’ Rankings give visibility recognition and expert validation (that is, to the extent you believe they are unbiased, systematic, and methodical).” – Expert Interview
47 M. Eiter, J. Pulcrano, J. Stine, T. Woll, “Same Solar System, Different Orbits: Opportunities and Challenges in Executive Education and Corporate University Partnerships”, 2014 UNICON report
37 | P a g e
“I think it is a good development that there are lots of different rankings…good that we have lots of them so that you can choose your perspective.” – Expert Interview
Publishing a range of different rankings, based on customer preferences and school strengths (such as innovation), is something Princeton Review did a number of years ago in the undergraduate, college ranking space. Princeton Review publishes ~73 different “best” lists each year, from “best food” to regional rankings to tuition-free schools. With their breadth and depth, these lists provide customers a more detailed and nuanced view of the range of options they can consider when making a decision. In addition, they know where to look for schools with strengths in specific areas. While we realize this could create a greater burden for the FT and for the participating schools, we thought this was an intriguing idea – and one that is also consistent with the way many customers are using rankings today (i.e., that they look at many rankings, not just one).48
“Mass customization breaks the assumption that business school stakeholders have uniform preferences and recognizes that providing stakeholders with customized information is a low-cost affair….. Stakeholders have different preferences. If we recognize those differences, then why not offer individualized rankings to schools?... Given that we are living in the digital age, mass customization of business school rankings is very feasible.” 49
In Sum, Contrasting Views
As we have seen to this point, there is broad awareness of the FT EE rankings and general acceptance of their importance. However, it is also clear that there are real disconnects between how EE customers perceive the rankings – roughly, as a superficial guide – and how many business schools perceive them – in many cases, as a flawed tool that misrepresents what the schools deliver. It is natural that the schools, as the subject of the rankings, would view them with the greatest scrutiny and skepticism. But, as our discussions with experts clearly showed, the schools have legitimate grounds for their concerns. In the next section, we will look more deeply at these criticisms and challenges. In addition, we will offer some initial thoughts on how the rankings might be improved.
48 Examples of different rankings that are available are included in Appendix 12: Which Ranking Are We Talking About? Comparisons of Rankings. 49 Glick (2008): 20
38 | P a g e
Challenges and Disconnects
“In general, rankings are probably seen as a necessary evil. We do not believe they are robust or credible or that they are a fair representation of differentiated quality between providers. However, we understand that the market does not share this view…. and therefore you need to be in the game if possible.” – Business School EE Leader Interview
“Yet this performance indicator exercises an almost hypnotic power. At a time when nations see themselves as ‘global competition states’ and global comparisons gather significance in many domains, university ranking fills a data gap. It renders the arcane and complex world of universities, simple, transparent, and compelling. Ranking seems democratizing: it renders accountable institutions that once held themselves above the common herd. It fits with old ideas about university status and modern contests like football league tables. It is easily understood and remembered.” 50
“…it is the simplistic beauty of rankings and the hidden work of constructing world-class universities at the global level that empowers them so that ‘they travel widely and are easily inserted into new places and for new uses’.” 51
In sports, we accept that there are winners and losers, and absolute position can be determined based on performances at specific events.52 If we see a team that is way up one year and down another, we understand that changes in team composition, coaching, and simply performance of individuals may have had an impact on their ranking.
Executive education organizations, however, do not work to maximize a specific outcome against competitors. Rather, they maximize program quality within their organizations based on participant feedback and direct experience of faculty and staff. Executive education organizations seek to achieve a high level of learning and repeatable quality across all programs, including those aimed at different industries and different career levels. They achieve these developmental objectives through a range of teaching methods and approaches, sometimes complex and often difficult to measure.
50 Marginson (2014): 45 51 J. Enders, J. (2014) “The Academic Arms Race: International Rankings and Global Competition for World-Class Universities” in A. Pettigrew, E. Cornuel, and U. Hommel, eds., The Institutional Development of Business Schools, (Oxford: Oxford University Press, 2014): 159. 52 For example, http://espn.go.com/espn/story/_/id/12317383/sports-nerds-too-easy , for “why sports nerds have it easy”, specifically: tons of data, clear rules, rapid feedback, and clear marks of success. Contrast this to educational programs that are designed to support individual, team and organizational development…they are very different.
39 | P a g e
It is understandable, then, that these organizations might have some issues with being ranked by the FT. In many industries, quality standards are set, often by an industry body. In this case, however, the basis for comparison is not determined by a set of industry rules. It is determined and controlled by the FT.
In this section, we seek to explore the problems underlying the rankings. We have found that there are some fundamental disconnects between what the FT EE rankings measure and what EE providers are offering. In addition, we believe there are some significant challenges – some of the FT’s making and some that are inherent to this industry – in how the FT has chosen to measure schools.53
Does the FT Measure the Right Things?
“The criteria haven’t changed even though business, management, and schools have changed. This doesn’t make any sense.” – Business School EE Leader Interview
In our survey of business school leaders, a number of respondents said that the criteria needed to be updated. There have been only minor changes to the criteria in the past 17 years (See Appendix 8: Criteria Evolution).
Specific recommendations in our survey responses included, “be more explicit/transparent about why they've selected the criteria they use,” update “the survey to reflect a dramatically changed industry,” “include a fuller set of programme criteria which is more reflective of evolved L&D needs,” “update survey to reflect issue important to executives and corporations in 2015,” and “include some ranking questions which reflect virtual learning courses as this is a major trend clients are asking for.”
As one business school survey respondent put it:
"The biggest issue I have with the FT rankings is that the original survey was developed in 1998 and launched in 1999. The survey has more or less stayed intact over the preceding 16 years, yet our industry has dramatically shifted in that period. Are the questions the FT asks really the relevant questions for the future of Exec Ed? Is the survey itself ready for a refresh to reflect the digital age? My recommendation [for] revising the survey is more strategic than process oriented.” – Survey open response, Business School
To understand the importance of the FT’s criteria, we asked both custom and open enrollment clients about their importance in our survey. The responses were similar across the two groups. In terms of importance, course design, new skills and learning, teaching methods, aims achieved, faculty, and quality of participants were among the most important criteria, being ranked as “quite” or “extremely” important by 8% to 10% of respondents.
53 As part of our analysis, we looked at the Berlin Principles for Ranking Higher Education Institutions, which are intended to improve rankings, and compared this to the FT EE ranking process. This analysis is available in Appendix 15: Berlin Principles on Ranking of Higher Education Institutions.
40 | P a g e
Figure 18: Most Important Criteria – Open Enrollment and L&D Customers
On the other end of the scale, however, there were a number of criteria that were considered “not at all” or “slightly” important by between 10% and 18% of respondents. These included: international location, food and accommodation, partner schools, and % female participants.
Figure 19: Least Important Criteria – Open Enrollment and L&D Customers
41 | P a g e
When we compared our data to the published criteria weighing by the FT, the weighting reflected similar preferences. In addition, The FT accounts for this variation by weighting the contributions individual criteria make to the final score based on a survey respondents’ responses to an importance question.
However, a key issue with the way criteria are weighted in the FT process is that it takes place in a closed system – that is, customer importance is accounted for only for the pre-selected criteria. What the weighting process does not reflect is whether or not these are the most important criteria for customers. There might be three or eight or even ten criteria today that are more important to customers than those included in the current survey. In fact, UNICON and others have done research on customer criteria and goals, particularly in the custom program space. This research, along with other customer data, could be used to identify criteria that better meet the needs of today’s customers. New criteria could be tested each year in the survey before being rolled out in the next year. (See Appendix 9: Some Specific Recommendations on Custom Program Criteria)
Companies, looking to invest in executive education programs, come with a very specific list of goals, some of which are well accounted for in the criteria, and others that are not. For example:
“First, is the design able to address our key objectives explicitly? Secondly, does it have a blended mix of solutions? Experiential elements are expected, obviously more in a leadership program than a finance program, but some experiential would always be needed. And is there a digital element? Third, is it cutting-edge? Does it bring new ideas in content or methodology? Fourth, but maybe this is first, what is the design approach? Is it co-designed with us? Do we work together? If we co-create, then I know the thoughts behind the program design, and everything is much more transparent. This is key.” – L&D Leader, Europe
This same company mentioned that they will use the FT custom program rankings to come up with and narrow down a list of prospective schools – but it is not clear that the company’s criteria are well represented in the FT methodology.
Since 1999, executive education has changed. The customers of executive education have changed. The world has changed. It makes sense that, at a minimum, the FT should engage in a process, ideally in collaboration with the schools and customers, to validate that the criteria it uses are as or more important to customers of executive education than other measures that might be applied. It will not be perfect, but it might be better.
How Should Criteria Be Weighted?
Another issue that was raised about this process by our survey responses is that the weighting of criteria is averaged, and then applied to all schools in the same way.
“Customers are asked to weight the relative importance of criteria so this is the ‘customer perspective’ on what is important. But, they do that for the whole population not at the
42 | P a g e
individual school level. For some, criteria that might be reasonable but for others it makes no sense.” – Survey response, Business School
While this is certainly true, it is difficult to imagine another way to do the weighting. If each school had different weighting based on its delivery or goals, or, even if schools were categorized and there were bands, then the data would be even less comparable than it already is.
Even so, this response highlights an important point that came up again and again in the survey responses. The respondents said that the rankings process also needs to attend to the diversity of executive education programs that are offered and the approaches employed. To the extent that averages are applied across schools, the related criteria should reflect characteristics of programs that are shared by all schools.
For example, we heard comments that FT should review its criteria and make optional those that apply to a subset of executive education providers. For example, one respondent wrote, “We have a large Open Program Division, but our programs are not residential. The criteria about housing and food confuses our participants,” or as one respondent put it even more bluntly, “remove bias towards old-fashioned programs run in residential, on-campus facilities.”
Several schools also felt the way that international data is collected and used biases the results toward schools that live in regions where there are more countries in closer proximity.
“Certainly EU schools have some advantage on the data-driven criteria because they are much more international.” – Expert Interview “Eliminate the bias in the questions (example: bias in favor of schools with more countries in nearby geographical area).” – Survey response, Business School
“FT is so skewed to favor exec. ed. groups that have "international" participants that it's of little interest to us.” – Survey response, Business School
And one school pointed out that the institutional data on faculty has not kept up with more recent hiring practices.
“Add/change criteria according to developments in the field of exec. educ. [For instance,] concerning faculty, we increasingly work with a flexible pool of staff to safeguard that external knowledge/expertise is brought into the university; but in the school survey you will be ranked higher with fulltime staff employment. Perhaps EQUIS could be reviewed as standard for criteria.” – Survey response, Business School
43 | P a g e
Is There Adequate Inclusion?
A second area where we heard frustrations related to inclusion criteria. Specifically, schools in emerging markets told us they were unable to participate because they lack the needed accreditation, revenue volume, or number of custom clients.
The number of schools participating in the FT EE rankings has grown quite a bit over the years – from 30 overall in 1999 to over 100 in 2015. In fact, there has been growth in participating schools in the last five years, from 65 to 75 in open and from 65 to 85 in custom. This increase surely reflects the growing acceptance of the rankings, together with increased global competitiveness in the industry, as well as growth of the executive education industry overall.
Figure 20: Total # of Schools Ranked
Year Overall Open Custom 1999 30 30 30 2000 20 30 30 2001 30 40 35 2002 35 45 45 2003 40 45 50 2004 40 45 50 2005 40 45 50 2006 40 45 60 2007 45 50 65 2008 45 50 65 2009 45 55 65 2010 50 60 65 2011 50 65 65 2012 50 65 70 2013 50 70 70 2014 50 70 80 2015 50 75 85
However, European and North American schools make up the vast majority of schools ranked – nearly 70% for both open and custom programs. In the last few years, schools in developing regions have definitely increased their participation, notably with the addition of 3 new Open and 6 Custom schools from Asia in the 2015 ranking. However, the majority of schools entering the ranking in the last five years are still from the EU and North America: 10 of 17 new schools for open and 13 of 27 for custom.54
54 See Appendix 23: Number of Schools by Region and Years in Ranking for a table showing geographic distribution.
44 | P a g e
Accreditation is a relatively new requirement – put in place in 2011 – and does work against newer schools.
“The FT requires accreditation. Is this keeping out good mid-tier schools? Or new schools in Asia?” – Business School EE Leader Interview
On the other hand, there are over 150 EQUIS and over 700 AACSB accredited schools, and many of these schools choose not to be in the ranking. So, it is not clear that accreditation is the largest hurdle. It is surprising, nevertheless, that a global ranking is not more geographically distributed.
Our open survey data from Business School EE Leaders had more than one perspective on the minimum revenue levels and also on applying purchase parity adjustment. (It was noted that these criteria are currently used in the FT MBA rankings, but not for executive education. For a comparison of the MBA and EE rankings, see Appendix 18: Comparison between FT EE and MBA Rankings). Accommodations on the revenue side, particularly for custom programs, matter most in developing economies.
“The USD 2 million cut-off for consideration should vary by country. It is far easier to obtain this level in a developed country with high prices than in a developing country. In our case, our currency has collapsed and this makes the cut-off very hard to reach, though the high quality of our programs and even our premium pricing (for our country) has not changed.” – Survey response, Business School
Another recommendation was to publish regional rankings. This makes sense based on how customers use the data. As we saw in the previous section, rankings are much more valued by customers when they do not know the school and/or region well. It is important to note that there are two dimensions to a regional ranking. One is whether a school is located in a particular region. A second is how capable any school (including those based in other regions) is to deliver in a particular region. Exploring these dimensions further would be of great benefit for customers who are looking to understand how executive education is evolving in developing regions.
There are also large executive education providers today that are not affiliated with business schools, some of which are focused on executive and leadership development. Others players are part of for-profit organizations such as consulting companies.55 One survey respondent suggested: “exclude non-business schools, e.g. CCL and maybe Duke Corporate [education]? Perhaps create a separate ranking for
55 For example, Forum Corporation, Mercer Consulting, Harvard Business Publishing Corporate Learning.
45 | P a g e
CCL that includes consulting companies.”56 This would be interesting to explore since, currently, there is no way to compare products or services of these companies.57
Satisfaction Data Is Problematic In General
A challenge with satisfaction ratings, in general, is that responses to questions are subjective. As one expert has put it, “the average response of a group of respondents to a survey is no more able to accurately identify the absolute quality of a university than the average response from the same people could accurately identify the distance between Earth and Jupiter.”58 Of course, in the case of providing feedback on an executive education program, you are at least assuming they have the shared experience of having made the trip.
On the plus side, executive education, as an industry, relies greatly on participant and client feedback for improvement and validation. Executive education organizations are used to working with satisfaction data, and they are aware of inherent challenges. Related to that, we did not see a lot of general critiques regarding the use of participant and client survey data. However, we did hear many criticisms about the way it is collected and weighted, as will be illustrated in the next few sections.
Another general issue is that not only are responses subjective but also they are not truly comparative. Consider this: if 1000 people visit 50 diverse restaurants (that is, diverse in terms of geography, menus, chefs, customers, etc.) over the course of a year, each orders a pasta dish, then each fills out a survey about their experience, can we really use this data to calculate a numeric rank for which restaurant serves the best Italian food? Of course, the rankings are more nuanced than this – there are 30 questions and a set of sophisticated statistical techniques that are applied. But, in the end, these rankings are a comparison based on questions related to specific (not comparative) experiences.
“The problem with the executive education ranking is that it asks an absolute question and answers it in a relative way. What did you think about the teaching at X school? And then the FT presents that against other schools who’ve been asked the same question.” – Expert Interview
In other words, for the most part, it is not based upon a direct comparison by individual customers. In fact, in most cases, the customers attending these programs may only attend one program in their career. As a result, the framework behind the FT’s rankings is one in which the outcome is comparative, but the inputs are not.
56 Though Duke CE is now 100% owned by Duke University, it is run as a separate entity from Duke’s business school, Fuqua. 57 A recent article on the FT website highlights growing competition in the EE sector, http://www.ft.com/intl/cms/s/2/896cfdc4-f016-11e4-ab73-00144feab7de.html#axzz3e5g28KNN 58 Marginson (2014): 48.
46 | P a g e
Is a 10-point Scale the Way to Go?
There may be specific improvements the FT could make in its use of a 10-point scale. Several survey respondents made the point that the 10-point scale, may, in fact, function like a two or three point scale.
“Everyone rates schools at 8, 9 or 10 – all good scores. I understand that the data gets normalized [into z-scores], but I think it functions in effect like a 3 point scale” – Business School EE Leader Interview “There’s a problem with the way the questions in the surveys are scored – the average score is 9/10. The difference in data between schools is almost meaningless. In addition, the scale has very different interpretations from one region to the next.” – Expert Interview
Also, the anchors for the scale may need to be improved so that it is clear what the response means, thereby creating more comparable data across schools.
“The actual process of filling in the form seems to be OK, but the 0-10 scale can be interpreted in deferent ways, i.e. is 10 'perfection' or 'best I have experienced'?” – Survey response, Business School
Reinforcing this, the FT acknowledged that subjective data is a challenge.
“Culture also has an impact on subjective data that we have not figured out how to remove. The Germans are going give you a 7 when they think you’re very good. The Latins will give you a 10 and the Dutch a 6. Even seasonality is an issue that we don’t know how to reduce… the Swedes don’t like the fact that we do EE in February when everyone in Scandinavia is depressed.” – FT Interview59
While there is no perfect solution to the culture problem, there is recent research in this area that could support the development of a more accurate response scale, including using a 5 or 7 point Likert scale, adding anchors to each number, and making sure these anchors are appropriate for the question being asked (that is, “construct-specific response anchors”).60
59 The full interview with the FT appears in Appendix 22: FT Interview Transcript. 60 Some excellent research on survey and scale design has been done by Professor Hunter Gehlbach at the Harvard Graduate School of Education. See: Artino, Anthony R ; Gehlbach, Hunter, “AM last page. Avoiding four visual-design pitfalls in survey development” Academic Medicine 2012, Vol.87(10), pp.1452; Artino, Anthony R., Gehlbach, Hunter, Durning, Steven J., “AM Last Page: Avoiding Five Common Pitfalls of Survey Design”, Academic Medicine, 2011, Vol.86(10), p.1327-1327; Gehlbach, Hunter, Barge, Scott, “Anchoring and Adjusting in Questionnaire Responses”, Basic and Applied Social Psychology, 2012, Vol.34(5), pp.417-433; and Gehlbach, Hunter, and Maureen E. Brinkworth, “Measure twice, cut down error: A process for enhancing the validity of survey scales”. Review of General Psychology 15, no. 4: 380-387.
47 | P a g e
What Is Really In an “Item”?
Our surveys revealed a related set of concerns around the wording of the questions, or “items” as they are called in the survey world. While we were not able to review the questionnaires, from our survey responses, the questions are clearly known and circulated in the executive education community. Several respondents pointed to the fact that the survey includes questions that “ask more than one thing” and “ask three questions in one.” This is a frequent challenge in survey design, but one that needs to be addressed and resolved.
Other respondents suggested that the questions may be difficult to understand and that more effort needed to be made to “ensure that the respondents actually have a common understanding of what the data/questions actually mean.” One respondent had heard feedback that people who complete the surveys “sometimes feel overwhelmed by the number and specificity of the questions.” Another pointed out, “some questions are ambiguous, and there tends to be a bias towards traditional styles of delivery (content focus, face to face).” And yet another respondent encouraged that questions be reviewed to ensure “non-traditional delivery methods can also be considered positively.”
The assumption that many in the industry have seen the questionnaires begs the question, why not just release them? Schools might try to “game” the rankings by putting the same questions on their exit surveys, but would there really be harm in this if the data improved the experience and, at the same time, made the process more transparent? And, by releasing the questionnaires, the FT could benefit from additional feedback that could improve the questions, and ultimately the rankings. Why not release the surveys with the data each year? It is not clear that the responses (or rankings) are better because the questions are confidential.
Surveying Custom Program Clients: A Data Challenge
As we noted in the satisfaction data, business schools overall are significantly less satisfied with the custom program ranking than with the open enrollment ranking. The custom ranking process involves surveying corporate clients who are responsible for custom programs in their organizations – not the actual participants in the programs. These customers are a very important client base for executive education organizations because custom programs have been a key driver of growth and innovation in the industry. In fact, today over 50% of UNICON members obtained the majority of their revenue from custom programs.61
However, there are a number of data challenges in surveying custom clients. For one, why are no participants surveyed? Several of our respondents felt the FT’s survey should be extended to participants (although this could be challenging as the respondents may not understand the context since frequently
61See Appendix 14: Economic Backdrop for Executive Education Industry.
48 | P a g e
custom programs have strong internal/company branding rather than external/school branding). Interestingly, the FT itself recognizes this challenge.
“We would really like to have more responses from custom program participants. We always get low response rates for custom programs. We’ve tried twice, but the results were not good. We’d like to find a way to do this, to improve the validity of the custom program rankings, but we don’t know what we can do to improve the response rate.” – FT Interview62
Another challenge is having surveys reach enough people who will respond. The FT shared that low response rates can contribute to the variability of custom program rankings.
“Our biggest challenge is getting enough response data for the EE custom rankings. Schools need to have at least 5 clients to be included, and if it is low, then there can be a lot of variability in the rankings. This tends to smooth out if the school has a large number of clients.” – FT Interview63
The FT prefers to have 20 custom clients at a minimum. Allowing for 5 presumably is a way to be more inclusive and thereby allow more schools to participate in the rankings. However, despite the intention to be inclusive, the FT’s small “n” here has raised a perception of bias.
“Big schools with more clients (esp. on custom) are favored… the big schools can pick their best custom clients. For many years, we were lucky if we could meet the minimum” – Business School EE Leader Interview “Schools do ‘cherry pick’ custom clients – which biases in favor of Schools that have large numbers of clients.” – Survey response, Business School
One alternative might be to provide separate rankings depending on the number of clients (for example “boutique custom programs” and “large custom programs”)
A related challenge centers on the fact that you can only nominate a client once, even if you offer multiple programs for her organization.64 While some executive education offices offer many custom programs to many different clients, others have fewer and/or focus their strategy on repeat business with a small number of clients. Unfortunately for them, there are restrictions placed on the contact list for custom. As one respondent pointed out:
62 The full interview with the FT appears in Appendix 22: FT Interview Transcript. 63 Ibid. 64 The FT also indicated it sometimes makes exceptions on this as well. (See Appendix 21: FT Criteria for Inclusion – Executive Education Programs)
49 | P a g e
“They refuse to allow the same contact to respond on two different programs for the same company, even when that person manages both programs. FT says we can't list both programs. What?” – Survey response, Business School
From a school perspective, two programs with the same client is a desirable situation. It builds the relationship and allows for deeper engagement with the company. It would create a burden for the customer to do more than one survey, but, from a data gathering perspective, why not trust them to differentiate the experiences and provide feedback on more than one program?
A final challenge is in the area of client management. From our interviews we learned that, in the past, the FT allowed schools to contact custom program survey respondents, but no longer.
“Earlier, schools might have contacted the clients and encouraged them to participate. Now that is not allowed – the FT does not want you to influence your clients.” – Business School EE Leader Interview
Despite this new rule, we still heard business school leaders worry about other schools “finessing” their clients. One school mentioned holding a targeted, stellar, alumni event just before the questionnaires were sent out, and it was not coincidental. In addition to not solving the issue of perceived influence, we wondered if this rule may, in fact, be limiting the amount of custom data gathered in the name of improving its quality.
The same respondent who noted the alumni event had an interesting suggestion: “Why not allow/enable us [all schools] to encourage them [custom clients] to respond?” Since the schools’ relationships with their custom program clients are important and (hopefully) long-term, having schools support engaging their clients in the survey process could help get more data, and, in this case, more data is better. If the responses are higher due to a strong relationship between the school and the custom program client, it is an indication of quality and customer value, at least from a client engagement perspective.
Measuring custom programs is challenging, and there is work to be done to make the custom program ranking a more level playing field.
Does the FT Use the Right Programs For Its Rankings?
“15 years ago I think they helped the providers of Exec Ed focus on issues beyond the classroom. Most of us saw the class as an end unto itself. The [questions]'s about prep and follow up really did do the industry a favor – however, per my earlier comment, the survey was designed in 1998. We all have made dramatic improvements in many of the areas highlighted by the FT. The key question is what should we be focusing on to provide future value for our clients?” – Survey response, Business School
There is little visibility into what kind of custom programs the FT rankings compare. Programs like London Business School’s $38MM multi-year contract with the Kuwaiti Petroleum Corporation demonstrate that
50 | P a g e
the scope, complexity and longevity of custom programs can be huge.65 Is it correct to compare this program to a custom program that is simply an adapted version of an open program or to a series of 2-day visioning and strategy workshops with the C-suite of a major firm?
This issue came out in our survey as well. Schools specifically requested that shorter programs be included in the custom program rankings. This makes sense for senior executive team programs. These programs are rarely more than one or two days. In fact, shorter programs seem to be a general trend.
“Include courses that are less than 3 days, as shorter courses are a mega-trend in corporate course demands from clients.” – Survey response, Business School
“Custom programs are now much shorter on average, and yet only program longer than three days can be included. FT should decrease the minimum program length to be [one] more representative of market conditions.” – Survey response, Business School
On the open enrollment side, schools continue to adapt to a changing marketplace by offering new, focused approaches to leadership, including programs that address strategically important topics such as innovation, technology management, and intellectual capital. Schools now offer programs such as “The Innovative CIO” or “Essential IT for Non-IT Executives.” However, the FT rankings focus on general management or leadership programs (vs. functional programs such as marketing, finance, or IT). As such, do the rankings fail to capture the value and importance of programs such as these and of the schools that deliver them well? Should the FT be clearer in stating what the EE ranking is truly measuring (and hope that readers understand this) and why it is limited to the general management roles?
Our survey provided insights into these important issues. Respondents asked for “less ambiguous definitions (e.g. what is a senior management program vs [what is] not)” and for the “survey [of] more than two open enrollment programs.” One respondent pointed out that the FT “should survey ALL participants in ALL OE programs, not just the ‘AMP’ and ‘GMP’ programs (unless they rename it to the ‘AMP /GMP ranking’ not ‘executive education ranking’).”
It was also pointed out that a larger selection of open enrollment programs could provide a better sample.
“[The] FT asks for 2 programs in detail. This allows institutions to cherry-pick a couple of standout programs while much of their offering is so-so. A better ranking would base itself on a wider number of programs.” – Survey response, Business School
“For us, 2600 touch points being evaluated by 120 participants, 70-80 of whom will respond… This is a poor design. The other thing that is problematic [is] the requirement for 20 custom
65 Palin, Adam, “London Business School signs $38m Kuwaiti contract”, Financial Times, Nov 18, 2013
51 | P a g e
programs. [We have] 23, so 20 is 90% of our clients. If you have a large client base you can pick the top 10% or 20%.” – Survey response, Business School
Given the breadth of open enrollment programs that are available, there are definitely options in this area that might better meet customer needs.
And Is a “Program” the Right Unit to Measure?
“Corporate clients increasingly expect business school offerings to focus on organizational performance as well as individual improvement, and they increasingly expect business schools to understand and align with their challenges and goals.” 66
In our survey, we received several responses that raised the question, is the program the right unit? There is an argument to be made for taking a broader view of activities included in executive education. It would be interesting to explore if there are major activities and accomplishments (for example, in strategy and organizational development) that are provided by schools and are of high value to clients and indicative of overall quality, but that are not being taken into account in the FT’s survey.
Three suggestions in this area from the survey were:
“To analyze more Executive Education Development and quality as a whole, integrated business and not only partially (open, customized)” – Survey open response, Business School
“For the school survey it is hard to collect data separately for custom and open programs. Increasingly there is a mixture of executive education offered by faculty centers and our EE department, and in our organization structure we don't have these areas separated. In fact our staff works in both kind of programs more and more which enhances the cooperation and exchange of knowledge.”- Survey open response, Business School
“To give importance to the Innovation in Executive Education (Business Development, networks, etc.).” – Survey open response, Business School
Another gap is that the rankings do not explicitly capture many, multifaceted services that executive education providers currently offer that are not integral to the program. Often, companies are looking for more than just a class or program. Instead, they are looking for everything from coaching to blended learning to on-the-job simulations and training. Schools are responding to this demand and expanding the services they offer, as the results of a 2014 UNICON survey show.
66 Frank R. Lloyd and David Newkirk. “Strategies and Choices: University-Based Executive Education Markets and Trends.” UNICON Report dated August 15, 2011, p. 3
52 | P a g e
Figure 21: UNICON Member Services Offered (n=93)67
With its focus on traditional or mainstream programs, the FT rankings may fail to capture important differentiators in the dynamic and complex executive education marketplace. This is in contrast to the MBA rankings, which arguably measure more of a commodity.
Calls For More Transparency: Why Not Level the Playing Field?
“Regarding the rankings, they just feel biased....” – Survey open response, Business School
Can you imagine a sports league where the way final score calculations included weightings that were not made public? The FT currently shares a great deal, but not all, of their methodology. Schools also invest varying amounts of time and energy in understanding the rankings. Of course, schools that have been close to the process for a number of years and have made a strong effort to understand the data the FT provides are more comfortable. As one expert told us, “If you are very close to the process and data there isn’t much mystery about them. If you aren’t close, there is some mystery.” Nevertheless, there are also “black boxes” in the process and calculations that are not transparent. In the case of rankings, it is not the devil that is in the details, but it is the truth.
The need for greater transparency is reinforced by experts and schools alike. It was one of the most repeated comments on our survey.
67 UNICON “State of the Industry Survey” 2014; 93 UNICON schools responded
53 | P a g e
“Indicators, measures and methods of compiling, computing and interpreting ranking data should be fully transparent to all parties.” – Expert Interview
“Regarding lack of transparency, I think the FT could do more. They could put all 30 questions on their site. They could put their rules on the site and advice on how to interpret the questions. They could add some statements about their rules around transparency of data, etc.” – Expert Interview
“I queried the process described on the portal this year and was told there had been an error in the description of the steps to take to complete our data. [The FT should provide] transparency in the methodology.” – Survey response, Business School
“Methodology is opaque – find a way to make it transparent and expose meaningful data (not just statistically derived and rank-order based scores).” – Survey response, Business School
It is important to keep in mind that the business school executive education community is not large. School leaders know each other, and they know relative programmatic strengths. Hiring and promotion happens across schools. As a result, there is a good basis for comparative perspectives. Some of the push back on the rankings comes from direct knowledge of strengths and weaknesses.
“The final operationalization of the ranking scores is not clear, especially comparing results and scores across different business schools. Knowing the market and the different business schools, some differences or some equalities among different schools seem not consistent with the reality and not fully explainable.” – Survey response, Business School
As mentioned earlier, in spite of the challenges inherent in using satisfaction data, schools tend to accept that the rankings are calculated based on it and that the data generally aligns with their internal improvement processes derived from their own exit surveys and related market research. In fact one survey respondent asked for more use of satisfaction data, not less.
“Increase the weighting of criteria relating to quality and client satisfaction and decrease criteria based on quantitative data – number of international program, number of partners, number of faculty from outside home country, etc.” – Survey response, Business School
What schools find very frustrating is that they cannot look at the FT data in detail. Given that programmatic changes may be necessary and take years to improve their score and position, schools should understand the feedback in as much detail as possible. In our survey data, there were many schools that indicated they would like to see the raw data.
“To have all the responses available for future reference – not just limited questions” – Survey response, Business School
“Share raw data” – Survey response, Business School
54 | P a g e
“Share the raw data; make the process transparent” – Survey response, Business School
“The work flow has been improving over the recent years. Yet we would like to have access to the School's raw data, used to compute final rankings, regardless of or having or not been listed in the rankings.” – Survey response, Business School
“Make the evaluation criteria and results available in detail to schools (make it understandable to schools).” – Survey response, Business School
“Provide comparative year-on-year data so a Business School can analyse their performance year-on-year. For example: our raw data (client feedback) can be higher than previous years, however our overall ranking dropped – no transparency as to the impact of the weighting criteria.” – Survey response, Business School
Several also asked for a more robust comparative view.
“Giving us the feedback and maybe allow us to draw comparative data from competitor schools.” – Survey response, Business School
“More transparency in how our ranking is achieved, also compared to competitors. How is it [for instance] possible that universities that do not participate in all rankings or are new on the list are ranked higher than we are?” – Survey response, Business School
And some also wanted better to understand the rankings process itself.
“Better understanding of how the FT surveys our clients (from outreach through post-survey completion).” – Survey response, Business School
“We would like to know how many schools participate in the rankings process.” – Survey response, Business School
Executive education organizations respect and use customer feedback data, and they want to understand the data in depth. From an industry perspective, making the ranking process completely transparent by providing the questionnaires after the survey, providing schools with their raw data, and rationalizing and clarifying all weightings, all these steps would make for a more level playing field and a stronger industry that better serves its clients.
“…as long as rankers are responsible and transparent; as long as they invest properly in serious research and sound data; as long as they are frank about the limitation of the proxies they
55 | P a g e
employ; and as long as they help to educate their users and engage in grown-up debates like this one, rankings can be a positive force in higher education.” 68
Challenges to Weightings, Especially For Custom
In this project, we tried, as much as possible, to understand how various weightings (beyond those that are determined in the survey responses) are applied. While the survey vs. data metric weightings are fairly clear, others weightings are not. For example, in the custom survey, additional weight is given to the responses of individuals who are more senior in their organizations, work in larger organizations, and/or contract with more than one organization for custom programs. Exactly what these weights are and how they apply is not clear. This lack of clarity was of concern in our survey responses. “Provide transparency as to the impact of weighting factors such as employee size and seniority of the sponsor.” One respondent also mentioned, “Value public and charity sector organisations as much as private sector ones.”
The reasoning behind customer weighting was also called into question.
“Companies in the custom survey that have more than one executive education program get greater weighting – why? We think if we get 100% of the company’s business, that’s a good sign.” – Business School EE Leader Interview
“[The FT] also attaches weight to things like the "seniority" of the program sponsor for custom programs; whether they are described as "strategic,” "operational" or "functional"; how many exec. ed. providers a company works with; ask "are you planning to repeat" without asking WHY (or WHY NOT).” – Survey response, Business School
Further, it was pointed out that executive titles and roles vary by geography. As a result, the way the titles are used may not be valid and/or may bias results toward organizations operating in more developed economies.
“Eliminate or reconsider the categories and their impact (rank of the executive, size of the companies, etc.). Some economies are based mainly on small and medium companies, and the schools operating in these economies are negatively impacted… In the same time, in some regions VP is a common job description for senior executives and a lot of people in a company are VPs. In other regions, a company can have only a VP and the senior executives have other job denomination.” – Survey response, Business School
“Ranking of job titles are not necessarily uniform titles used in all parts of the world….Vice President, Partner could be less senior than an Executive.” – Survey response, Business School
68 P. Baty, “Say it loud: I’m a ranker and I’m proud,” Times Higher Education (online), March 11, 2011. http://www.timeshighereducation.co.uk/story.asp?storycode=415470
56 | P a g e
Exploring the “Bullwhip Effect”
“The highest we got was a [single digit]; the lowest was in the high 30s. You know you haven’t radically changed programs to account for that jump.”- Business School EE Leader Interview
“My estimate is that at least half of the ranking innovations are not news, but noise” 69
While big changes in a school’s position get interest and attention, it is also one of the reasons schools do not believe the rankings are valuable. Schools have called this the “bullwhip effect,” and they find it both discouraging and frustrating.
When you look at the variability on a school by school basis, the variability is very apparent. Some schools are relatively stable, but others clearly are not. Looking at the variability in custom program rank over five years from 2010 to 2014 (See Appendix 19: Variability in Position), we see high/low spreads ranging from 2 positions to 39 positions. This variability is experienced by many schools, not just a few. 27 ranked programs in this period had a total change greater than 20 positions, and 16 programs had a standard deviation greater than 10.
The chart below maps year-over-year changes for 9 schools that had the greatest variation in the ranking between 2010 and 2014. For these schools, it is hard to imagine the types of sustained programmatic or satisfaction shifts that could account for this degree of variation.
69 I. Dichev, I. “Comment: The Financial Times business schools ranking: What quality is this signal of quality?” European Management Review, 5(4), (2008): 220
57 | P a g e
Figure 22: Year over Year changes for 9 Schools with Greatest Ranking Variation (2010-2014)
Based on our research, four elements may account for these shifts. First, the satisfaction data used in the surveys is more variable than other types of data. Second, the sample sizes for some schools are relatively low and therefore may have greater variation year by year. Third, schools that invest in understanding the rankings and pay attention to the ranking criteria and process do seem to improve results and enhance stability. Many of the schools we interviewed that did well in the rankings year over year were also the ones that worked hard to optimize their ranking result. Fourth, organizations can go through changes in leadership or direction that affect their rankings during a specific timeframe.
Schools that make specific changes and see improvements in the rankings also note a lag time, most likely due to the three-year smoothing that is applied to the data,
“The frustrating thing is there is quite a lag between making an operational change and that showing up on the rankings.” – Business School EE Leader Interview
At the same time, the FT leverages large ranking shifts for news purposes. This may not be fair to schools that are working to use the rankings as metrics for improvement because they are hampered by the weightings, smoothing, and criteria the FT uses. For example, this year’s FT discussion of the executive education rankings, subtitled “Fine Lines Separate the Elite” and including a thoughtful commentary on the first few positions, but then went on to say:
“Other notable performances in the customised programme rankings include ESCP Europe of France, which enjoyed the biggest jump, up 28 places to 28th. Olin Business
0
10
20
30
40
50
60
70
80
2010 2011 2012 2013 2014
Rank
(Cus
tom
)
Rotterdam School of Management, ErasmusUniversity
Boston University School of Management
Insper
Stanford Graduate School of Business
UCLA: Anderson
MIT: Sloan
ESCP Europe
AGSM at UNSW Business School
ESMT - European School of Management andTechnology
58 | P a g e
School at Washington University had the biggest fall, dropping 31 places to 55th. The Wharton School of the University of Pennsylvania recorded its worst performance, down 21 places to 47th.”70
This type of commentary contributes to the perception that the FT exploits the rankings for news purposes at the expense of the reputations of the schools, as we will see in the next section.
Underlying Purpose and Conflict of Interest
During the six months we have spent working on this project, we have looked deeply at the FT EE rankings methodology, we have read the broader literature on university rankings generally, we have talked directly to the FT, and we have heard from a broad range of experts and stakeholders. Overall, it is our belief that the FT is thoughtful and thorough in their process, that their framework is sophisticated, and that they work hard to produce credible rankings. Developing the rankings is not a simple or inexpensive task given the amount of data they collect, compile, analyze, and publish. They have made quality-related improvements to the process over the years, and they spend significant time with schools to provide additional context and feedback.
At the same time, from our survey, we heard some critiques regarding the rankings’ underlying purpose and possible conflict of interest. Specifically, some survey respondents felt that the entire purpose of the rankings was self-interest on the part of the FT. Some felt rankings exist to “sell copies of the FT” or to “agitate and sell newspapers” and that they “unfairly self-promote FT as an authority on business education as well as generate brand value and equity for FT.” One of our experts addressed this issue directly, providing a more balanced approach:
“Of course rankings are about ads and traffic. But at the same time they also help customers make informed decisions and elevate quality.” – Expert Interview
Several respondents wondered if “the Financial Times [as a newspaper] is the right organization to conduct the rankings.” Another survey respondent said, “I do think an objective external ranking of an industry is a valuable service to consumers/clients of that industry. However, I don't think the FT ranking is particularly objective and I don't believe it ultimately serves the consumers/clients well.”
A related recommendation was that:
“It is important to have ratings--and perhaps rankings--but not for them to be done by the FT or any other publication which has an incentive to sell newspapers/periodicals, etc. Rather a legitimate quality ranking organization should do the ranking of business schools and executive education--JD Power or some organization like that.” – Survey response, Business School
70 http://www.ft.com/intl/cms/s/2/f3d38f18-f019-11e4-ab73-00144feab7de.html#axzz3e5g28KNN, accessed June 25, 2015.
59 | P a g e
This begs the question, however, of what organization has the resources to support an alternative ranking? It is because rankings are of interest to the general public and drive people to read the news that news organizations are willing to make substantial investments in producing them. And, as we have mentioned before, having executive education in the news helps business schools.
One issue that the FT will need to address thoughtfully is the Corporate Learning Alliance in collaboration with IE Business School and several other partners.71 This was called out in a survey response, “Do not allow the ratings organization (in this case the FT) to establish a money-making partnership with one of the ‘raters’.” For the record, as a new entity, the Financial Times-IE Business School Corporate Learning Alliance will not be eligible to be included in the rankings for 3 years (MBA rankings) and until $2M revenues in a calendar year for executive education is achieved. And then, the question of accreditation and eligibility will need to be addressed.
Sharing customer and revenue data with an external organization is sensitive and requires a great deal of trust. In our survey, which was conducted in the month after the CLA was first announced, we asked how comfortable schools were in sharing data with the FT, and 65% reported to be quite or extremely comfortable, which suggests that the FT has forged a fairly high level of trust with organizations currently participating in the rankings.
Figure 23: How comfortable is your organization with sharing information and participant data with the Financial Times for the rankings? (n=51)
71This was announced in December 2014. Other partner schools include Yale School of Management, Fundacao Getulio Vargas in Brazil; Imperial College in the UK; Antai Business School and Renmin Business School in China; Singapore Management University, and EGADE in Mexico. See http://www.ftiecla.com/, http://aboutus.ft.com/2014/12/08/new-financial-times-and-ie-business-school-venture-to-provide-custom-education-for-executives/#axzz3dtXv6vjT, http://poetsandquantsforexecs.com/2014/12/11/convergence-project-joins-ie-with-yale-and-the-ft/#sthash.kP0eBr9n.dpuf.
not at allcomfortable
slightlycomfortable
moderatelycomfortable
quitecomfortable
extremelycomfortable
0.0%10.0%20.0%30.0%40.0%50.0%60.0%
60 | P a g e
It would be beneficial if this trust could be strengthened even more in the future. On the other hand, if the FT is perceived as a direct competitor to the schools it is ranking or as providing more insight into the rankings to partners in this alliance than to others, this will raise significant concerns.
Auditing?
The Financial Times is unique in the MBA rankings industry because it audits the data from schools. For executive education, while it will verify data from time to time, executive education data is not audited. Several survey respondents asked for “more auditing to ensure fairness” and to “verify data.” A specific example one school gave was, “We are under the impression that some schools count their total university staff in for their business school, even though a large part of these staff members do not teach in the Executive Education programs.” It was also mentioned that “Exec Ed practices are so different among B Schools that I'm not sure everyone puts the same kind of data in the surveys.”
Auditing is one option. Simply providing more visibility into the institutional data schools provide is another way of addressing this challenge. Executive education is a diverse industry, as described in previous sections. Engaging the leaders of this industry in the process of selecting measures to verify the data has the potential to improve the rankings and to help with the perception of fairness.
In Sum, a Unique Development Opportunity
In this section, we have discussed a range of disconnects and challenges with the FT EE rankings. In spite of the breadth and depth of the feedback, we did not find what we believe to be “deal breakers” to ranking EE or to having the FT conduct the ranking. Many of the schools’ frustrations with the rankings could be addressed through greater transparency and updating of the criteria and questions. These improvements would have the potential to benefit both the leadership position the FT currently holds and the executive education industry as a whole. Based on the 360 degree view that we gained from our research and in the spirit of providing developmental feedback, we hope that our findings and recommendations in the next section will serve as a bridge between the FT and executive education leaders and promote even better service to their mutual clients and business leaders in the future.
61 | P a g e
Recommendations
Sifting through and synthesizing the survey data, interviews, and the literature review, plus the many conversations between the authors, a raft of possible recommendations surfaced, from the practical to the aspirational to the most-likely-impossible. The future of rankings is for all of us to construct, and there are many plausible scenarios.
To the extent this report serves as an intervention and diagnostic, we believe the best outcome would be for business schools and the Financial Times to collaborate on constructing a better ranking, maybe “FT EE 2.0.” It’s been 17 years and there is room for improvement. Additionally, there is a mutual interest in serving customers of executive education, and a great deal of experience on both sides measuring and evaluating executive education programs. Sure, you would have to restart the clock on comparative historical data, but an EE ranking that better served customers would be a win for everyone involved.
To that end, for the FT, we felt that we could take a longer view with our recommendations to address the “Disconnects and Challenges” that we identified in the previous section. While the FT certainly does a professional job with what is, arguably, a very complicated ranking task, we believe there are opportunities to improve the relevance and quality of their rankings.
Should this not come to pass, however, business schools will need to continue to work within the current framework, and customers will continue to use the rankings in their current form. From that perspective, it was important that our recommendations to business schools and their customers (L&D professionals and open enrollment participants) be focused on the FT EE rankings as of today. Therefore, our recommendations for this audience assume no changes in the near future and assume that all actors will have to work with the rankings “as is.”
This does not mean that we’re advocating that all schools participate in the rankings. Participating in the FT rankings takes time and resources, and depending on the school’s position and strategy, the investment might not feel worthwhile. That said, some of the changes we are recommending are easier to implement than you might think, and may be worth exploring, keeping in mind that one way to create a more even playing field is for everyone to have similar information on how to put their best foot forward in the current context.
In sum, our hope is twofold: to help everyone to better understand and use the rankings today and to inspire dialogue around what the executive education rankings of the future should look like.
And with that preamble, we will first go through recommendations for business schools and customers if nothing changes, and then describe key recommendations we would make to the Financial Times that would result in an improved ranking for executive education programs.
62 | P a g e
If the Rankings Stay the Same: Business Schools
“As much as we grumbled about them, we tried not to just play to rankings, but to learn from them.” – Business School EE Leader Interview
It was interesting – and perhaps not surprising – to see that business schools had a less favorable view of rankings than their customers. Nevertheless it appears that rankings are here to stay. We believe business schools can embrace them to elevate their role within the industry and as a benchmarking tool.
Business schools providing executive education need to respect the investment that the Financial Times puts into the process and publication of the EE rankings and the fact that FT customers value and use them. Business schools also need to find generative and collaborative approaches to providing quality and comparative information to customers. If business schools participate in the rankings, they need to give serious internal attention to the entire process of providing data and learning from the ranking results.
For all schools to get the most out the FT EE rankings, we suggest the following:
1. Take the rankings seriously and be proactive – do not let them just happen to you. We know the rankings are important. Schools should not haphazardly participate and then complain about the results. Instead, they should follow the advice experts will offer:
“How do you rise in the EE rankings? First, you acknowledge how you are going to get measured in your job and let your client know about it. Second, you pay attention to what you’ve learned from customer feedback and address it. And third, you maximize your score on the data-driven metrics.” – Expert Interview
2. Put someone in charge of the rankings as an important part of his or her job. Ideally she reports to the Dean and has a good reputation with the Faculty. In most schools it is typically a senior staff member in charge of external relations. If learning from the rankings is important, then this must be seen as more than a marketing exercise and everyone needs to know who is responsible.
3. Focus on the programs that you are going to have ranked and survey your participants post program. If you want to increase your position in open enrollment, select two to four open programs that fit the FT’s criteria, have multiple locations and partners for those programs, put your best faculty on them, and then give the faculty workload credit for spending extra time with participants and alumni in follow-up. Align evaluations of these programs with FT criteria. Let those programs create a halo effect for the school in the rankings. We are not saying lose sight of the big picture here – for example, you should not let rankings be an excuse for not innovating elsewhere in your program portfolio – but focusing on the programs that will be ranked is clearly required to succeed at the FT EE rankings.
63 | P a g e
4. Create your “FT rankings strategy” and, when appropriate, innovate separately. It’s important to have an FT rankings strategy and execute it well. However, the template created by the FT rankings, especially for open programs, can slow down innovation if it becomes a provider’s sole focus. This can give education boutiques and consultants an opportunity to out-innovate business schools. As a result, business schools that participate in the FT EE rankings need to balance the need to do well in the rankings with the need to innovate – and in this marketplace, no organization can afford to ignore innovation altogether. To strike the right balance, we think schools should distinguish between incremental innovations “within the FT template” (e.g., new AMP content) and breakthrough innovations that are “outside the template” (e.g., fundamentally new delivery methods or entirely new courses.) By doing so, breakthrough innovations can be nurtured “on the side” without the pressure of conforming to the rankings. Once innovation is proven, we think the schools and the FT should bring it into the rankings to help pull the industry forward.
5. Align your rankings strategy with your school’s strategy. If you are a global player, appearing in the top 20 or 30 is probably critical for you. If you are a national or regional player, just being ranked already gives you the right to claim to be world-class. What is on your website? What do all of your faculty and staff put on their LinkedIn Pages? What is in your email signatures? What is on the signage around the school or on the letterhead and in the brochures? Does your local chamber of commerce or economic development organization know what your ranking is and how to use it in promoting the region? Control the narrative. Craft your message to your needs and strengths.
6. Be explicit about what you offer to participants as follow-up – and other criteria, for that matter. If Follow-up remains an important criteria, if Follow-up continues to mean (to the people surveyed) events organized by the school or its alumni chapters, and if focusing on regional open program participants is to the advantage of the school, enhance these. While the FT may not be 100% clear about its criteria, it is important that you are clear so that you can identify what is working and what is not working.
7. Pay attention to your L&D counterpart and her needs. Yes, what your faculty does in the program with the participants is paramount, but it is your client who will be filling in the FT survey. If the school is under-staffed and your administration and marketing people are over-worked, it is not going to help your relationship with the client. Despite a great programmatic outcome, your L&D contact may also remember how she was ignored.
8. Take a long-term view. It may take a while to rise in the rankings. If you sense that your school is going nowhere or if your school is poised to make a major restructuring effort, avoid and/or drop out of the rankings until this passes. The FT will impose a 1-year moratorium on the school that drops out, but, honestly, it will take 2-3 years for your efforts to bear fruit anyway. So, bite the bullet, do what you have to do to turn the school around, and then reap the benefits in the rankings a few years later.
64 | P a g e
9. If you do decide to opt out, find other ways to generate awareness and quality perception. Clearly, customers are paying attention to the rankings. But that does not mean that they are the only way to market your business. Until you are ready to play the rankings game, consider other marketing vehicles to influence customer perception.
10. Proactively share with the FT and the world your quality metrics. Include information in your marketing and on your website about how you evaluate your programs and the criteria you use. Help your customers understand how to measure quality. Survey your customers, update your survey instruments, and share this information with your colleagues and the FT. Remember, rankings fill a need of customers, and there is more than one way to address customer concerns.
If the Rankings Stay the Same: Customers (L&D Professionals and Open Enrollment Participants)
We found that customers of executive education are quite savvy about rankings, especially L&D professionals. At the same time, we believe that the rankings marketplace is a very complex one and that there are many opportunities for confusion and misinformation.
Customers like the idea, debatable though it may be, that there is a level playing field, that quality can be measured, and that key players can be compared against one another. Rankings achieve this, to some degree. Rankings are easily communicated and appear to be easy to understand (although, as we have seen this is somewhat deceptive).72 But, the leveling of the playing field is artificial. It is achieved by applying a relatively narrow set of measures to diverse, experiential, evolving, and globally-practiced educational activities. More needs to be done to illustrate the rich opportunities and range of great practices that exist in executive education.
To help L&D professionals and open enrolment participants get the most out of the rankings, we suggest the following:
1. Have your own strategy for how to use the rankings. If you are experienced in executive education, or have staff members who are, then the rankings may simply be a way to confirm what you already know. It may also be a way to sell your decision to top management and your candidates, “See, we’re sending you to one of the top 5 schools in the world.” If you are new to EE or looking for programs in a part of the world you are unfamiliar with, then the rankings can be a serious tool for discovering potential vendors. However you end up using it, recognize that it is a tool vs. an absolute measure of quality.
2. Develop your own criteria before you rely solely on those of the FT. The FT EE rankings are not completely representative of what any school offers. Only you know what you or your company
72 Expert interview
65 | P a g e
needs, and your needs should drive the decision more than any ranking. Be proactive about sharing your needs with schools when you inquire into their capabilities and during the RFP process as well as when you are working with them to measure whether your goals were achieved.
3. Go beyond just looking at the final column in the rankings. Look at the criteria that are important to you and your company to see if schools that did not do well in the overall ranking perhaps scored higher on the criteria that are critical to you. The ease of reading the rankings belies their complexity and does not necessarily tell you what you want to know.
4. Do not assume a halo effect between rankings. A great MBA does not necessarily mean a great EE program. In many schools the MBA and Masters programs are quite separate from executive education. Ask the school if the same professors teach in both programs being evaluated.
5. Understand what accreditation of a school means for you. The FT only includes schools that are accredited (with the exception of Duke Corporate Education and CCL). Both AACSB and EQUIS evaluate schools based on their degree programs, on the assumption that running good degree programs means the same quality exists in executive education programs. They do not evaluate and accredit individual EE programs. When looking at a business school with degree programs, a lack of accreditation might be a red flag requiring further investigation. But lack of accreditation should not be seen as a negative for organizations such as CCL, THNK, Hyper Island, Moebius, etc. – all high-quality boutiques – that do not possess degree programs.
6. Let the Financial Times know what you need. While business schools reach out frequently to the FT to learn more about how a ranking was calculated or to provide suggestions for improving the rankings process, the actual consumers of the rankings, and most importantly HR leaders, are comparatively silent in this debate. Since the ranking exists for you, understand what the rankings do and do not measure. Then, work with both the FT and schools to make sure the rankings are measuring what matters to you.
Evolving the Rankings: Our Recommendations to the Financial Times
To the FT, we would say that it is critical that you update the rankings to reflect, guide, and serve the market. The executive education industry has evolved and continues to change. Business schools offer a greater variety of services than they ever did. Customers are demanding a greater variety of options and a higher level of strategic value than in the past. And everyone is operating at a faster pace with greater complexity. We feel the current ranking is caught in a rather outdated and limited view of the industry.
To address this, we think the FT should consider the following:
66 | P a g e
1. Reconsider the rather broad, current definition of “program.” Schools are increasingly delivering other services. How can this be captured in the current ranking? Could there be sub-rankings around these services (e.g., best school for coaching)? Should a common set of services be included and some excluded?
2. Update the criteria. It is good that the FT lets customer weight the current criteria. However, we are convinced that the current criteria matter the most to executive education customers today, especially on the custom side and especially given the proliferation of services. We think it is time to consider new criteria that address the complexity of the marketplace.
3. Rank within specialty areas – Strategy, Leadership, Innovation, Finance, and functional expertise in IT, Supply Chain Management, etc. Many customers told us that speciality areas were more important to them than an overall ranking.73 We think this is an area of opportunity as well as a related evolution and/or expansion of the criteria.
4. Provide alternative rankings for large and small custom program providers. The current methodology is biased toward large providers that can select from and submit more custom program clients. The bias towards large providers creates a permanent divide between big and small schools – and one that may overstate the quality at big schools.
5. Consider creating an online ranking tool for EE that allows users to create their own rankings. We think this could be an exciting innovation. Readers could select the criteria that are important to them – maybe even the weightings that they believe make sense for their needs – and then rank schools accordingly.74
6. Consider adopting the Princeton Review approach and publishing more types of rankings, perhaps updating each less frequently. Given that programmatic changes can take some time to show results, it is not clear that a yearly overall ranking is the ideal frequency. Publishing less frequently (in conjunction with providing greater transparency) could lessen the sting of the “bullwhip effect.” At the same time adding more diverse rankings is of value to customers. So why not publish custom program rankings in one year, AMP/GMP rankings in another, topically-focused rankings in a third, emerging market rankings in a fourth, etc.?
7. Consider having the participant and customer surveys in additional languages. This is a real need, especially if the FT hopes to give the world a measure of how Chinese business schools are
73 See, for example: Eiter, Pulcrano, Stine, Woll, “Same Solar System, Different Orbits: Opportunities and Challenges in Executive Education and Corporate University Partnerships“, 2014 UNICON report, pp. 19, 33-4. 74 The FT web site does have a beta “Profiler” tool for Masters, MBAs, EMBAs degrees, http://rankings.ft.com/businessschoolrankings/profiler
67 | P a g e
evolving. We realize the FT is unabashedly international in its perspective. Adding more languages would only serve to further that perspective.
Providing more transparency is a must. We feel that full transparency (including data, weightings, and questions) is the most fundamental thing that the FT can do to address the issues mentioned in this report. We understand and respect that the FT has developed a very thoughtful and detailed methodology and that they likely wish to protect anything of theirs that is proprietary. We also understand there are limits to how much data can be shared (e.g., to respect privacy of both individuals and organizations). However, we feel that data is data. If you really want readers to get the most out of the information – and if you want to quiet the doubters – share your data and your formulas.
Some specific recommendations along the theme of transparency include:
8. Specify, in greater detail, which types of programs are being ranked (both open and custom). This would allow readers to base their choice of school and program on how close they think their requirements match the ranked programs.
9. Be more explicit about what the major criteria mean and share the questions behind them. Respondents had varied interpretations of the top criteria, which calls into question their answers to the survey. While it is understandable that the FT does not release the survey instrument, why not provide more commentary and a deeper dive on what the different criteria measure, and why they are important to customers of executive education? The FT might conduct research in this area (or research in collaboration with organizations like UNICON) to understand whether criteria terms like “program design” and “teaching methods” are understood in the same way by those who respond to the survey.
10. Provide the ranking formula. We believe having the formula out in the open would be a simple way for schools to focus on doing their best and for customers to understand the results. We also question whether revealing the formula represents a real issue of intellectual property, as we believe the FT’s competitive advantage is in the process of collecting, analysing and publishing the data – not necessarily in the formula that are used to calculate the rankings.
11. Face the FT|IE Corporate Learning Alliance issues head on and outsource more of the process. Some schools now perceive the FT as a competitor in certain areas, and talking about “firewalls” is not going to be enough. It is critical that the FT can assure everyone that it remains a neutral arbiter of quality in its executive education rankings. FT must ensure that the confidential data and contact names being provided by the schools will not be misused. FT must be able to guarantee that none of the insights gained through its ranking process inform the work of the CLA. Outsourcing the entire ranking process to a neutral broker would help address these concerns.
68 | P a g e
It is an opportune time to strengthen the methodology. There is no doubt about the FT’s professionalism and thoroughness. Their rankings are well thought out and conscientiously administered. The FT EE rankings are dominant in the market. However, it is also clear that the methodology is not fully representative of the schools it measures, that it is not always measuring the same types of programs, and that it has questionable “black box” elements to it.
To address these issues, we think the FT should consider the following:
12. Update to a Likert scale. Participating in the rankings is a high-stakes activity for executive education providers. We believe the FT should use the most accepted and endorsed social science practices for surveys. A 5 or 7 point Likert scale with intermediate anchors would help filter out bias and inconsistency across respondents and geographies.
13. Use a random and/or more representative sampling methodology. All the schools participating in these rankings can select their best programs that meet the criteria. That is certainly clear and expedient. However, it is also not representative. What would a representative sample look like? Would that help smooth out the shifts? Would it more closely align the ranking with actual school quality?
14. Simplify the ranking down to fewer variables. We understand that the FT has given a lot of thought to its methodology and that, in some respects, they want a lot of variables in order to be as inclusive as possible. However, the mysterious calculations and weightings behind these rankings end up diluting the value. Ultimately, we all want a tool that provides a clear and measurable point of comparison – sometimes keeping it simple is best.
15. Find another way to smooth out the occasional dramatic shifts in results. No school radically changes in 12 months, yet a number of schools have experienced significant annual swings. We suspect this is due to small sample sizes, particularly for custom providers. We think the FT should consider increasing the sample size or lengthening the measurement period (Business Week’s previous 24 month cycle makes it more justifiable to see dramatic shifts).
16. Audit schools on a rotating, regular basis (as with the MBA rankings). There is a perception that some schools “game” the data. Regular audits would dispel this rumour and inhibit those who might be attempting or tempted to do it.
17. Apply purchasing parity to the $2 million revenue minimum. By doing this, top quality schools in developing markets with weak currencies can be included in the rankings.
18. Instead of a numerical ranking, consider qualifying schools by level, i.e. top tier, middle, etc. How do we really differentiate between #17 and #18, or even #16 and #26? Are differences to the customer truly reflected by this precision? The current research methodology is a credible
69 | P a g e
approximation, and, without doubt, it would be too costly and time consuming to make it extremely accurate. If that is the case, we think it is better to acknowledge the methodology’s limitations rather than to aim for false precision.
70 | P a g e
Conclusion
“Most of the rankings today are based on history, what was good in the past. Is that a good way to evaluate a school for yours and your company’s future?” – Expert Interview
As we have now come full circle on our 360° journey, we might reflect on the fact that, in the beginning, it was the business schools themselves, and particularly European schools, that encouraged the FT to publish rankings.75 The fact that the FT has stayed true to its original purpose, a global ranking focused almost exclusively on business school providers of executive education, has been a significant benefit to the industry and to business schools. Not every school may feel that way, and perhaps rankings sting the most for those who have experienced the ‘whiplash effect’, but we also heard other schools state equally strongly that they feel that participating in the rankings has significantly benefited their programs and their customers.
Rankings, as a global practice, appear to be here to stay. They continue to fill an important customer need and promote readership and industry awareness. The FT EE rankings, specifically, have also helped identify areas for improvement for some EE providers.
That said, over the FT rankings’ seventeen years, the world has changed, and business and executive education have changed along with it. The goals, content, delivery methods, and context of executive education have all evolved dramatically – from the global nature of executive development and educational excellence to imperatives for corporate innovation to digital delivery. There is room to update and improve the FT EE rankings: they are not as customer- or industry-centric as they could be and they are not as transparent as they could be. However, we also believe that therein lies a real opportunity to innovate as well.
We believe a new and improved ranking is preferable to the current one, and it is our hope that the FT will take action and think much more broadly about what the ranking could be, and how this could be achieved in collaboration with the business schools and their customers. Experts say rankings aren't about quality, but customers need tools that help them to select providers and programs that meet their needs. Wouldn't it be great if the ranking process could be re-imagined in such a way that it truly provided the valuable comparative perspectives customers need to make great decisions? Wouldn’t it be great if the ranking was a truly powerful tool for matching customer needs with providers’ capabilities? We think so. Intriguingly, we also think this reimagining could lead to not just one ranking, but to many rankings – something that would reflect the richness and diversity of this industry.
As this report was undertaken under the sponsorship of UNICON, permit us to close with a message addressed principally to the world’s business school executive education providers. The FT EE rankings are important. They are not perfect, but the FT does a very good job. The rankings help clients and
75 Bradshaw, 2007, 58
71 | P a g e
participants to make decisions and help business schools to profile themselves. They are a great vehicle to attract readers, sell newspapers, and to feature advertising that promotes executive education programs. If the FT stopped today, someone else would probably step in and do it less well. Thus, it is in the interest of business schools to figure out the best way to deal with the rankings, as a loosely aligned industry and, especially, as individual schools. Work with your clients, participants, and the FT to improve the rankings. This doesn’t mean participating in the rankings is for everyone, but it is worth taking an informed and thoughtful look at possible advantages for your organization. The fact that business schools are selected to stand on this podium, and not education boutiques and consultants, is an amazing opportunity. Embrace it. And – if it means changing the rankings to better reflect the marketplace – own it.
We recognize that change will require engaging in more dialogue and may even require additional resources – both for the schools and the FT. However, the investment will be to better serve the customers and the industry, a goal that all the players can ascribe to. In a world of “open source” and “open education”, allowing for a more “openly designed” rankings process would be an act of leadership.
72 | P a g e
Index of Figures
Figure 1: FT Survey Criteria and Respondent Weightings ............................................................................... 9
Figure 2: Key Metrics Provided by Schools and FT’s Weightings .................................................................. 10
Figure 3: Open Program Participants: Have you reviewed the Financial Times executive education rankings, specifically? ................................................................................................................................... 15
Figure 4: L&D Professionals: Are you aware of the Financial Times rankings of executive education programs? ..................................................................................................................................................... 15
Figure 5: How valuable do you believe rankings are as a source of information in selecting executive education programs? .................................................................................................................................... 16
Figure 6: How valuable do you believe rankings are as a source of information when selecting an executive education program? ..................................................................................................................... 17
Figure 7: Prior to selecting an educational program to attend, did you check rankings? (n=692) .............. 18
Figure 8: Which of the following statements describe(s) your process in reviewing the Financial Times rankings? (Select all that apply) (n=147)....................................................................................................... 19
Figure 9: Which of the following statements describe(s) your process in reviewing the Financial Times rankings? (Select all that apply) n=147 ......................................................................................................... 20
Figure 10: When considering providers of executive education, how important are the following? Check all that apply (n=211) .................................................................................................................................... 22
Figure 11: Overall, how satisfied are you with the way your programs are ranked by the Financial Times? (n=51) ............................................................................................................................................................ 24
Figure 12: How valuable do you believe rankings are as a source of information for participants when selecting an executive education program? (n=91) ...................................................................................... 25
Figure 13: Do you communicate your Financial Times rankings results? (Select all that apply) (n=50)...... 27
Figure 14: How important do you feel it is for executive education programs to be ranked by an organization such as the Financial Times? (n=90) ........................................................................................ 30
Figure 15: Do you survey your participants to see if they use rankings to select your programs? (Select all that apply) (n=88) ......................................................................................................................................... 32
Figure 16: Approximately how much time is spent each year by your organization on managing the Financial Times executive education rankings process? (n=49) ................................................................... 32
Figure 17: How difficult is it for you to collect the following data as part of the Financial Times executive education rankings process? (n=50) ............................................................................................................. 33
Figure 18: Most Important Criteria – Open Enrollment and L&D Customers .............................................. 40
Figure 19: Least Important Criteria – Open Enrollment and L&D Customers .............................................. 40
73 | P a g e
Figure 20: Total # of Schools Ranked ............................................................................................................ 43
Figure 21: UNICON Member Services Offered (n=93) .................................................................................. 52
Figure 22: Year over Year changes for 9 Schools with Greatest Ranking Variation (2010-2014) ................. 57
Figure 23: How comfortable is your organization with sharing information and participant data with the Financial Times for the rankings? (n=51) ...................................................................................................... 59
74 | P a g e
Appendices
Appendix 1: Pearson PLC and the Financial Times ....................................................................................... 75
Appendix 2: UNICON Member Schools and Their 2015 Open & Custom Program Rankings ....................... 76
Appendix 3: Open Enrollment Program Customer Survey Data ................................................................... 79
Appendix 4: Custom Program Customer Survey Data .................................................................................. 86
Appendix 5: UNICON Business School EE Survey Data ................................................................................. 88
Appendix 6: Social Media and Executive Education Rankings ...................................................................... 94
Appendix 7: Evolution of the FT EE Ranking Methodology .......................................................................... 99
Appendix 8: Criteria Evolution .................................................................................................................... 102
Appendix 9: Some Specific Recommendations on Custom Program Criteria ............................................. 103
Appendix 10: 2015 Graphic Showing Topics/Questions behind FT EE Criteria .......................................... 104
Appendix 11: FAQ for Business Schools ...................................................................................................... 105
Appendix 12: Which Ranking Are We Talking About? Comparisons of Rankings ....................................... 110
Appendix 13: Financial Times Surveys Schedule......................................................................................... 113
Appendix 14: Economic Backdrop for Executive Education Industry ......................................................... 114
Appendix 15: Berlin Principles on Ranking of Higher Education Institutions ............................................. 115
Appendix 16: List of Interviewees by Institution and Role ......................................................................... 119
Appendix 17: Correlation between MBA and EE Rankings? ....................................................................... 121
Appendix 18: Comparison between FT EE and MBA Rankings ................................................................... 123
Appendix 19: Variability in Position ............................................................................................................ 125
Appendix 20: Z-scores Explained ................................................................................................................ 128
Appendix 21: FT Criteria for Inclusion – Executive Education Programs .................................................... 130
Appendix 22: FT Interview Transcript ......................................................................................................... 132
Appendix 23: Number of Schools by Region and Years in Ranking ............................................................ 136
75 | P a g e
Appendix 1: Pearson PLC and the Financial Times
Pearson PLC is the owner of the Financial Times Group and Penguin books. Pearson characterizes itself as “the world’s leading learning organization”76, a “global media and education group”. As of the end of 2014, Pearson had 40,000 employees in more than 70 countries and was listed on the London and New York stock exchanges. Educational ventures owned by Pearson include Wall Street English, MyEnglishLab, CTI Education Group (in South Africa), enVisionMath2.0, and MasteringChemistry. Pearson has developed the Efficacy Framework to evaluate the impact of educational products77.
The Financial Times Group comprises: the FT newspaper and FT.com, Financial Publishing, FT Chinese, FT Labs, Medley Global Advisors (MGA), ExecSense, the New York Institute of Finance, a 50% shareholding in The Economist, and a joint venture with Vedomosti in Russia78.
The FT has a combined paid print and digital circulation of 720,000. FT’s digital subscriptions surpassed print circulation in 2012 and now account for 70% total circulation. Mobile is an increasingly important channel for the FT, driving almost half of total traffic. FT education products now serve two thirds of the world’s top 50 business schools79.
76 https://www.pearson.com/ 77 http://efficacy.pearson.com/efficacy-tool/ 78 http://aboutus.ft.com/corporate-information/#axzz3eHD7woeK 79 http://aboutus.ft.com/#axzz3eHD7woeK
76 | P a g e
Appendix 2: UNICON Member Schools and Their 2015 Open & Custom Program Rankings
UNICON Member School 2015 Open 2015 Custom Aalto University 43 58 American University In Cairo 74 Ashridge 36 22 Asian Institute Of Management Babson College 24 BI Norwegian Business School 62 65 Boston University 62 46 Carnegie Mellon University 52 Case Western Reserve University CEIBS – China-Europe International Business School 24 34 Cheung Kong Graduate School Of Business City University London Columbia University 20 42 Cornell University Cranfield School Of Management 33 10 Dartmouth College – Tuck Duke University – Fuqua Emory University 35 ESADE Business School 7 12 ESIC Business & Marketing School Fundação Dom Cabral 12 33 Georgetown University – McDonough 21 Georgia Institute of Technology Harvard Business School 4 18 HEC Paris 2 2 Hong Kong University Of Science And Technology IAE Business School – Austral University 55 30 IE Business School 29 IESA – Instituto De Estudios Superiores De Administración IESE Business School 3 1 IHM Business School IMD 1 6 Imperial College London 60 INCAE Business School 42 40 Indian School of Business INSEAD 7 11 Insper Institute Of Education & Research 46 49
77 | P a g e
UNICON Member School 2015 Open 2015 Custom ITAM- Instituto Tecnológico Autónomo De México Lahore University Of Management Sciences (Lums) London Business School 17 4 Macquarie University Massachusetts Institute of Technology – Sloan 25 18 Melbourne Business School – Mt. Eliza 43 26 Michigan State University MIP – Politecnico Di Milano School Of Management 76 Moscow School of Management Skolkovo Nanyang Technological University National University of Singapore 64 16 New York University Northeastern University – D'Amore-McKim Northwestern University – Kellogg 14 41 Nyenrode Business Universiteit 51 77 Ohio State University – Fisher College Peking University – Guanghua 54 44 Pennsylvania State University – Smeal College Pontificia Universidad Católica De Chile (PUC) Porto Business School 73 70 Queen's University 27 Queensland University of Technology (QUT) 63 Renmin University Of China Rice University Rutgers, The State University Of New Jersey Santa Clara University Sciences Po Simmons College Singapore Management University Smith College SMU – Southern Methodist University – Cox Stanford University 15 Stockholm School of Economics 37 31 Tecnologico De Monterrey (ITESM) The University of Tennessee – Haslam College Thunderbird School of Global Management at ASU 26 20 Tsinghua University Universidad De Los Andes 40 38 Universidad De San Andrés (UDESA) University of Alberta 65 70
78 | P a g e
UNICON Member School 2015 Open 2015 Custom University of British Columbia – Sauder 43 University of Calgary University of California, Berkeley – Haas University of California, Irvine University of California, Los Angeles – Anderson 31 38 University of Cambridge – Judge 50 67 University of Cape Town 56 University of Chicago – Booth 5 13 University of Georgia – Terry College University of Manchester 48 University of Maryland – Smith School University of Michigan – Ross 9 32 University of Minnesota – Carlson University of North Carolina at Chapel Hill – Kenan-Flagler 9 University of Notre Dame – Mendoza University of Oxford – Said 10 23 University of Pennsylvania – Wharton 23 47 University of Pittsburgh – Katz University of Pretoria – Gibs 48 53 University of Reading – Henley 29 University of South Carolina 33 University of Southern California – Marshall School University of St. Gallen 27 37 University of Texas, Austin – McCombs 59 University of The Witwatersrand 73 University of Toronto – Rotman 19 62 University of Virginia – Darden 11 36 University of Washington – Foster University of Western Australia UNSW Australia 47 54 Washington University In St. Louis – Olin 17 55 Western University – Ivey 20 45 Wisconsin School of Business Yonsei University
79 | P a g e
Appendix 3: Open Enrollment Program Customer Survey Data
We surveyed past enrollees attending programs at ten, globally-distributed UNICON schools. Two surveys were administered, one in English and one in Spanish. The Spanish data is included here for comparative purposes, but the figures in the body of the report are based on the English data.
In what geographic region are you based
Answer Options Response Percent Response Count
Africa 10.8% 54
Asia 20.3% 101
Europe 41.4% 206
Middle East 2.0% 10
US/Canada 21.1% 105
Oceania 0.8% 4
Latin America 3.6% 18
answered question 498 skipped question 1
¿En qué región geográfica se encuentra?
Answer Options Response Percent
Response Count
África 0.0% 0
Asia 0.0% 0
Europa 1.4% 3
Medio Oriente 0.0% 0
EUA/Canadá 0.0% 0
Oceanía 0.0% 0
América Latina 98.6% 209
answered question 212 skipped question 1
80 | P a g e
In what geographic region(s) have you attended executive education program(s):
Answer Options Response Percent Response Count
Africa 8.5% 42
Asia 20.3% 101
Europe 49.5% 246
Middle East 0.6% 3
US/Canada 36.0% 179
Oceania 0.6% 3
Latin America 1.6% 8
answered question 497 skipped question 2
¿En qué regiones geográficas ha asistido a programas de educación educativa?
Answer Options Response Percent
Response Count
África 0.0% 0
Asia 0.5% 1
Europa 14.1% 30
Medio Oriente 0.0% 0
EUA/Canadá 16.4% 35
Oceanía 0.9% 2
América Latina 94.4% 201
answered question 213 skipped question 0
Prior to selecting an educational program to attend, did you check rankings?
Answer Options Response Percent
Response Count
Yes 44.9% 223
No 47.1% 234
I'm not sure 8.0% 40
If yes, which rankings did you review? 139
answered question 497 skipped question 2
Detail – Source Total Percent
Multiple (online) 62 48%
Financial Times 35 27%
Online (not specified) 15 12%
Company 6 5%
Recommendations 4 3%
81 | P a g e
Regional 2 2%
Reputation 2 2%
Business Week 1 1%
National 1 1%
US News 1 1%
Total 129
Frequency – in multiple (n=62) Total Percent
Financial Times 52 45%
The Economist 26 22%
Business Week 14 12%
Wall Street Journal 7 6%
Forbes 4 3%
Poets & Quants 3 3%
US News and World Report 3 3%
Newsweek 2 2%
Times Higher 2 2%
Financial Times Deutschland 1 1%
Internet forums 1 1%
Princeton Review 1 1%
Total 116
Antes de elegir el programa de educación al que asistiría, ¿revisó las clasificaciones (los ránkings)?
Answer Options Response Percent
Response Count
Sí 20.0% 42
No 70.0% 147
No estoy seguro/a 10.0% 21
Si respondió que sí, ¿qué clasificaciones/ránkings revisó? 20
answered question 210 skipped question 3
100 mejores universidades forbes el economista
En Google busque las mejores escuelas para posgrados
QS University Rankings
Rankings de periódicos
En México clasificación Reforma
Mejores Universidades de México.
Diversos, vía google.
FT
the times, QS
mejores universidades a nivel posgrado
82 | P a g e
Expansión
Mejores Universidades Costos Programa Lo comparo con universidades reconocidas
US News, Economist, Business Week
Las 10 mejores IES del país.
las mejores universidades de america latina
Internet
revista expansion
Expansion
Prior to selecting an educational program to attend, did you check rankings?
In what geographic region are you based
Answer Options
Answer Options Africa Asia Europe Middle
East US/Canada Oceania Latin America
Response Percent
Response Count
Yes Yes 18 28 122 7 33 4 10 44.8% 222
No No 32 61 70 3 61 0 7 47.2% 234
I'm not sure I'm not sure 3 12 14 0 10 0 1 8.1% 40
If yes, which rankings did you review? 139
answered question 496
skipped question 2
How valuable do you believe rankings are as a source of information when selecting an executive education program?
Answer Options Not at all valuable
Slightly valuable
Moderately valuable
Quite valuable
Extremely valuable
Response Count
For schools I know well (n=481) 33 91 177 138 42 481
For schools I do not know well (474) 5 22 72 213 162 474
answered question 484 skipped question 15
83 | P a g e
¿Qué tan valiosas considera que son las clasificaciones como una fuente de información a la hora de seleccionar un programa de educación ejecutiva?
Answer Options Para nada importante
Ligeramente importante
Moderadamente importante
Muy importante
Extremadamente Importante
Response Count
Para escuelas que conozco bien 9 35 73 75 17 209
Para escuelas que NO conozco bien 2 1 30 75 90 198
answered question 209 skipped question 4
Have you reviewed the Financial Times executive education rankings, specifically?
Answer Options Response Percent
Response Count
yes 58.2% 152
no 31.8% 83
I'm not sure 10.0% 26
answered question 261 skipped question 238
Have you reviewed the Financial Times executive education rankings, specifically?
In what geographic region are you based
Answer Options Africa Asia Europe Middle East US/Canada Oceania Latin
America Response Percent
Response Count
yes 12 14 97 6 14 2 6 58.1% 151
no 7 20 25 1 24 1 5 31.9% 83
I'm not sure 2 5 14 0 4 1 0 10.0% 26
answered question 260
skipped question 238
¿Ha revisado las clasificaciones/ránkings de educación ejecutiva del periódico Financial Times, específicamente?
Answer Options Response Percent
Response Count
Sí 30.2% 19
No 57.1% 36
No estoy seguro/a 12.7% 8
answered question 63 skipped question 150
84 | P a g e
Which of the following statements describe(s) your process in reviewing the Financial Times rankings? (select all that apply)
Answer Options Response Percent
Response Count
I had already chosen a school/program and scanned the list to see its position 55.1% 81
I checked several rankings for the school – not just executive education but how other programs and degrees are ranked
49.0% 72
I read the detail about rankings criteria and how the schools are ranked 45.6% 67
I used the list to find ranked schools in a specific geographic region 28.6% 42
I used the list to find a school 23.8% 35
I looked at detailed data on specific ranking criteria 19.7% 29 I used ranking information to get support/permission to attend the program 18.4% 27
Other (please specify) 9
answered question 147 skipped question 352
¿Cuáles de las siguientes oraciones describe(n) su proceso de revisar las clasificaciones de Financial Times? (Seleccione todas las que apliquen)
Answer Options Response Percent
Response Count
Ya había elegido una escuela/un programa y escaneé la lista para revisar su posición 36.8% 7
Utilicé la lista para encontrar una escuela 42.1% 8 Utilicé información de clasificaciones/ránkings para obtener apoyo/permiso para asistir al programa 21.1% 4
Utilicé la lista para encontrar escuelas clasificadas en una región geográfica especifica 21.1% 4
Leí a detalle acerca del criterio de las clasificaciones/ránkings y de cómo están evaluadas las escuelas
26.3% 5
Revisé información detallada sobre criterios específicos de clasificación 15.8% 3
Revisé varias clasificaciones de la escuela – no sólo de educación ejecutiva sino que de cómo son clasificados otros programas y títulos también
36.8% 7
Otro (Especifique): 0
answered question 19 skipped question 194
85 | P a g e
How important are the following criteria to you in evaluating an executive education program?
Answer Options Not at all important
Slightly important
Moderately important
Quite important
Extremely important
Response Count
Selection/Preparation 9 22 129 229 68 457
Course design 2 1 36 192 235 466
Teaching methods and materials 3 4 45 208 206 466
Faculty 5 5 51 182 220 463
Quality of participants 2 16 62 192 194 466
New skills and learning 2 4 27 199 232 464
Follow-up (post-program) 19 55 181 158 51 464
Aims Achieved/Objectives met 4 10 48 216 183 461
Food and accommodation 38 111 189 104 24 466
Facilities 8 62 164 182 48 464
% Female participants 138 94 143 68 24 467
% International participants 53 53 99 169 93 467
Repeat business and growth 49 74 149 147 40 459
International location 87 66 134 135 42 464
Partner schools 89 111 141 101 24 466
Faculty diversity 43 64 115 176 68 466
answered question 467 skipped question 32
¿Qué tan importante son los siguientes criterios para usted a la hora de evaluar un programa de educación ejecutiva?
Answer Options Para nada valiosas
Ligeramente valiosas
Moderadamente valiosas
Muy valiosas
Extremadamente valiosas
Response Count
Selección/Preparación 2 3 37 106 53 201
Diseño del curso 2 0 10 110 79 201
Facultad 1 13 40 89 57 200
Cualidad de los participantes 5 9 51 79 57 201
Nuevas habilidades y aprendizaje 2 1 10 81 107 201
Seguimientos (Después del programa) 3 16 49 82 50 200
Objetivos alcanzados 1 3 9 81 106 200
Comida y alojamiento 15 48 80 42 15 200
Comodidades 8 32 76 67 16 199
% de Participantes Femeninas 54 37 61 37 12 201
% de Participantes Internacionales 36 30 68 48 16 198
Negocio repetido y crecimiento 7 20 56 82 32 197
Ubicación internacional 12 27 52 85 22 198
Escuelas hermanas 22 24 70 64 20 200
Diversidad de la facultad 12 21 59 77 31 200
answered question 201 skipped question 12
86 | P a g e
Appendix 4: Custom Program Customer Survey Data
Are you aware of the Financial Times rankings of executive education programs?
Answer Options Response Percent Response Count
Yes, very familiar 23.9% 62
Yes, aware 57.5% 149
I'm not sure 11.2% 29
No 7.3% 19
answered question 259
skipped question 20
When considering providers of executive education, how important are the following?
Answer Options Not at all important
Slightly important
Moderately important
Quite important
Extremely important
Response Count
If the school is in the top 10 of the FT Executive Education rankings. 4 25 72 86 23 210
If the school is in the FT Executive Education rankings (whatever position). 10 32 72 78 16 208
If the school has risen or dropped recently in the FT Executive Education rankings.
11 39 81 68 11 210
Whether the school is accredited by AACSB or EQUIS 38 44 68 38 17 205
answered question 211 skipped question 68
How valuable do you believe rankings are as a source of information in selecting executive education programs?
Answer Options Not at all valuable
Slightly valuable
Moderately valuable
Quite valuable
Extremely valuable
Response Count
For schools I know well 10 49 94 89 20 262
For schools I do not know well 2 13 47 137 59 258
answered question 264
skipped question 15
87 | P a g e
How important are the following criteria to you in evaluating an executive education program?
Answer Options Not at all important
Slightly important
Moderately important
Quite important
Extremely important
Response Count
Selection/Preparation 2 9 50 136 45 242
Course design 0 1 11 89 154 255
Teaching methods and materials 0 1 16 98 138 253
Faculty 1 6 25 70 152 254
Quality of participants 2 2 22 136 90 252
New skills and learning 0 1 22 109 120 252
Follow-up (post-program) 3 19 58 126 48 254
Aims Achieved/Objectives met 0 1 16 104 131 252
Food and accommodation 12 60 117 59 6 254
Facilities 5 31 107 95 15 253
% Female participants 15 31 99 82 26 253
% International participants 6 13 46 126 61 252
Repeat business and growth 6 27 88 93 25 239
International location 16 45 86 76 29 252
Partner schools 15 48 104 67 14 248
Faculty diversity 8 30 61 100 54 253
answered question 255 skipped question 24
“For the L&D professionals it’s important, but I’ve never had a senior executive or participant as me about them or mention them.” - L&D Leader, Europe
“One thing that isn’t clear to me from the FT is each school’s ability to deliver in different parts of the world. Do they have a network that is efficient and provides the same quality as their home campus?” - L&D Leader, Europe
“When the rankings first came out, they were really important. I worked at HKUST and we came out at #49. However, I think the rankings are less important these days because there are so many of them. There are all kinds of business school rankings. If you’re not a learning professional, it’s very confusing.” - L&D Leader, Asia
88 | P a g e
Appendix 5: UNICON Business School EE Survey Data
Which geographic region(s) represent the strongest market(s) for your executive education programs?
Answer Options Response Percent
Response Count
Africa 5.4% 5
Asia 22.6% 21
Europe 31.2% 29
Middle East 5.4% 5
US/Canada 52.7% 49
Oceania 2.2% 2
Latin America 10.8% 10
answered question 93 skipped question 1
On average, how many unique custom clients do you average a year?
Answer Options Response Percent
Response Count
0-5 3.3% 3
6-10 16.5% 15
11-30 54.9% 50
>30 25.3% 23
answered question 91 skipped question 3
On average, how many unique open enrollment programs do you run each year? (that is, not including iterations of the same program)
Answer Options Response Percent
Response Count
0-5 18.7% 17
6-10 18.7% 17
11-30 36.3% 33
>30 26.4% 24
answered question 91 skipped question 3
Do you survey your participants to see if they use rankings to select your programs? (select all that apply)
Answer Options Response Percent
Response Count
No 65.9% 58
Yes, informally, in our application 19.3% 17
89 | P a g e
Yes, in our exit survey 2.3% 2
Yes, as part of our internal market research 18.2% 16
Other (please specify) 2
answered question 88 skipped question 6
How valuable do you believe rankings are as a source of information for participants when selecting an executive education program?
Answer Options Response Percent
Response Count
not at all valuable 7.7% 7
slightly valuable 30.8% 28
moderately valuable 33.0% 30
quite valuable 26.4% 24
extremely valuable 2.2% 2
answered question 91 skipped question 3
How important do you feel it is for executive education programs to be ranked by an organization such as the Financial Times?
Answer Options Response Percent
Response Count
not at all important 5.5% 5
slightly important 20.9% 19
moderately important 33.0% 30
quite important 28.6% 26
extremely important 12.1% 11
Can you tell us why you feel this way? 52
answered question 91 skipped question 3
How important do you feel it is for executive education programs to be ranked by an organization such as the Financial Times?
Which geographic region(s) represent the strongest market(s) for your executive education programs?
Answer Options Africa Asia Europe Middle East US/Canada Oceania Latin
America Response Percent
Response Count
not at all important 1 2 1 1 4 0 0 5.6% 5
slightly important 0 4 3 0 13 1 3 20.0% 18
moderately important 1 7 5 0 22 0 1 33.3% 30
quite important 2 7 13 2 6 1 4 28.9% 26
extremely important 1 1 6 2 2 0 2 12.2% 11
answered question 90
skipped question 3
90 | P a g e
Do you participate in the Financial Times executive education rankings? (custom and/or open enrollment)
Answer Options Response Percent
Response Count
Yes, currently 56.0% 51
No, never 30.8% 28
In the past, but not currently 13.2% 12
What other rankings do you participate in: 25
answered question 91 skipped question 3
Reasons given for not participating:
• We do not believe the rankings support our operation. • Do not see the added value. • Has minimal impact on decision-making of sophisticated and experienced buyers • For starters, we are too small to qualify. But even if we did qualify, our offerings are all in a particular niche so
the general rankings aren't that relevant. • We decided to focus on other more critical items for now • We are a boutique ExecEd business. We meet the minimum requirements, but would find it difficult to impose
the process on our select clients. • This year we are not participating as we are undergoing some significant changes. • My university and business school have strong and trusted brands in our primary market(s). Executive
Education rankings are not necessary to support that. • We don't think it is important to the purchasing decision process • The FT ranking process is horribly flawed and poorly managed. • Our Dean feels he wants to focus on academics and research and does not see the value of rankings. • Since some of clients are from mainland China and the FT does not provide Chinese survey, it is difficult to ask
them to fill the survey. • Never asked to be • We do not typically have 20 custom clients in a one-year period. • We'd rather be not ranked, than ranked very very low. • We would like to know the criteria to participate • Our school’s business model is different of the traditional, so FT criteria is not relevant • Our program is small and the time and effort related to participating against larger programs will not
necessarily result in a greater benefit • It is too costly for the benefits we get. Our market is very regional and our clients never mention the ranking. • We consider ourselves to be a niche program and we don't believe we would fare well in the general rankings.
Do you participate in the Financial Times executive education rankings? (custom and/or open enrollment)
Which geographic region(s) represent the strongest market(s) for your executive education programs?
91 | P a g e
Answer Options Africa Asia Europe Middle East US/Canada Oceania Latin
America Response
Percent Response
Count
Yes, currently 4 8 19 4 25 1 6 55.6% 50
No, never 1 8 8 0 14 0 4 31.1% 28
In the past, but not currently 0 5 1 1 8 1 0 13.3% 12
What other rankings do you participate in: 37
answered question 90
skipped question 3
If your executive education programs are not currently ranked by the Financial Times, can you tell us why? (select all that apply)
Answer Options Response Percent
Response Count
We do not meet the FT minimum revenue criteria 24.2% 8
We do not meet the FT accreditation criteria 9.1% 3
We have chosen not to participate 66.7% 22
We do not have a sufficient number of custom clients 24.2% 8 We do not have a sufficient number of open enrollment programs that meet Financial Times criteria 15.2% 5
We do not think FT rankings are important to our clients or participants 21.2% 7
We went through the FT process but we were not ranked 0.0% 0
Other (please specify) 5
answered question 33 skipped question 61
Overall, how satisfied are you with the way your programs are ranked by the Financial Times? [ranked schools only]
Answer Options not at
all satisfied
slightly satisfied
moderately satisfied
quite satisfied
extremely satisfied n/a Response
Count
Open enrollment 9 8 12 12 0 10 51
Custom programs 14 12 11 9 0 4 50
answered question 51 skipped question 43
Do you communicate your Financial Times rankings results? (select all that apply) [ranked schools only]
Answer Options Response Percent
Response Count
No 18.0% 9
Yes, in email newsletters or marketing 64.0% 32
Yes, on our web site 58.0% 29
Yes, in print collateral 44.0% 22
Yes. in social media 54.0% 27
Yes, internally to staff 64.0% 32
92 | P a g e
Yes, internally to stakeholders such as faculty or leadership 76.0% 38
Other (please specify) 3
answered question 50 skipped question 44
How important is your Financial Times ranking viewed internally within your organization? [ranked schools only]
Answer Options Response Percent
Response Count
not at all important 5.9% 3
slightly important 9.8% 5
moderately important 17.6% 9
quite important 41.2% 21
extremely important 25.5% 13
answered question 51 skipped question 43
How important is your Financial Times ranking viewed internally within your organization? [ranked schools only]
Which geographic region(s) represent the strongest market(s) for your executive education programs?
Answer Options Africa Asia Europe Middle East US/Canada Oceania Latin
America Response
Percent Response
Count
not at all important 1 1 0 0 2 0 0 6.0% 3
slightly important 0 1 1 0 5 0 1 10.0% 5
moderately important 0 1 2 0 7 1 1 18.0% 9
quite important 2 3 10 3 7 0 1 40.0% 20
extremely important 1 2 6 1 4 0 3 26.0% 13
answered question 50
skipped question 43
Approximately how much time is spent each year by your organization on managing the Financial Times executive education rankings process? [ranked schools only]
Answer Options Response Percent
Response Count
1 day 8.2% 4
2 days 8.2% 4
3 days 14.3% 7
4 days 18.4% 9
5 or more days 51.0% 25
answered question 49 skipped question 45
93 | P a g e
Have you made any programmatic changes related to the Financial Times executive education rankings process, methodology, or results? (Select all that apply) [ranked schools only]
Answer Options Response Percent
Response Count
We have updated our program exit surveys to include FT criteria 32.3% 10
We have addressed specific programmatic issues highlighted in our rankings data 93.5% 29
Other (please specify) 11
answered question 31 skipped question 63
94 | P a g e
Appendix 6: Social Media and Executive Education Rankings
Catalyx Web Snapshot: Who is talking about Rankings & Executive Education & why?
Using data monitoring tools we extracted all of the relevant online content regarding Executive Education (filtering out MBA references), from Mar 1 –Apr 29, 2015, looking at global, English language mentions. There were 3,346 mentions of ‘executive education’ published during this period.
Twitter numbers need to be understood from 2 perspectives:
a) Very often they are emotional reactions, not necessarily content or fact driven. They are responses to something.
b) Retweeting inflates the original trigger tremendously.
Two pieces to a web snapshot:
– Conversational: postings, tweets, what people say
– Search: The key words being searched for.
Catalyx was surprised by how little conversation was going on in this space. On the other hand, seeing what people are searching for, they are obviously interested. So, executives want the information, on EE and on EE rankings, but they do not tweet and talk about it online. We hypothesize that if we added back MBA into the filter we would see much more social conversation.
The weekly cycle of conversations confirms that this is a professional topic. People search and talk about EE while at work, or at least during the workweek. This is not something one does much of on the weekend (would probably be different for MBAs or even EMBAs).
Schools do put out material, and people do search for that material, but schools do not seem to cause conversations to happen about that material, except for some prominent examples (i.e. conversation with MIT’s Peter Hirst and then mentions on Facebook).
95 | P a g e
780 mentions – long tail of News
96 | P a g e
Rankings were not amongst the top 20 topics discussed online during this period. Overall there were only 52 mentions of “Executive Education” that also included “rankings”.
Site Volume Site Volume1 www.virtual-strategy.com 17 26 www.usgbc.org 32 www.topix.com 13 27 www.bioportfolio.com 33 www.prnewswire.com 12 28 www.releasewire.com 34 www.topix.net 12 29 www.qatarisbooming.com 35 www.prweb.com 9 30 www.top-consultant.com 36 www.jbs.cam.ac.uk 9 31 www.humanresourcesonline.net 37 www.enhancedonlinenews.com 9 32 www.hsph.harvard.edu 38 www.zawya.com 8 33 www.gulfbase.com 39 morningnewsbeat.com 6 34 startups.co.uk 3
10 patch.com 6 35 www.hedgeweek.com 311 hbr.org 6 36 www.lsu.edu 312 news.money.ca 5 37 www.dnaindia.com 313 www.broadwayworld.com 5 38 www.cambridgenetwork.co.uk 314 www.nwaonline.com 4 39 albertaventure.com 315 www.modernghana.com 4 40 newswit.com 316 www.benzinga.com 4 41 news.websitegear.com 317 www.prnewswire.co.uk 4 42 www.ezunsecured.com 318 finance.yahoo.com 4 43 invest.5ver.com 319 www.business.rutgers.edu 4 44 globenewswire.com 320 markets.financialcontent.com 4 45 www.business-standard.com 321 news.yahoo.com 4 46 www.forbes.com 322 www.huffingtonpost.com 4 47 www.menafn.com 323 issuu.com 4 48 www.asianetnews.net 224 www.newswise.com 4 49 www.bi-me.com 225 qatarisbooming.com 4 50 www.apsense.com 2
97 | P a g e
And what do people search for online?
Though people do not talk about rankings online, they certainly are interested in them, as evidenced by what they search for:
And what do people find when they search online?
At the time of the research, using Google Keyword Analytics, if the search term includes “executive education,” the first site that people are offered is Exed.hbs.edu, and 2nd is Stanford GSBA, followed by LBS, Wharton and IMD. If the searcher puts terms such as “rankings executive education” or “Financial Times Executive Education” into Google they will then first get FT.com, followed by RankingsFT.com, Bloomberg.com/bschools and then IMD, followed by HEC in Paris or FDC in Brazil.
98 | P a g e
Project completed by: www.thecatalyx.com With thanks to Guy White.
99 | P a g e
Appendix 7: Evolution of the FT EE Ranking Methodology
1999 was the first year the FT published its EE rankings as well as its first international MBA ranking.80 It was an important moment for European business schools. As FT Business Education Editor Della Bradshaw explained: “One of the main driving forces behind the publication of the first Financial Times ranking in 1999 was a group of the top European business schools, which felt that the US-only focus of the two leading rankings, Business Week and US News and World Report, ignored the growing number of excellent European business schools.” 81
“One of the things I can tell you is that business schools hate rankings, but in fact we started producing them because we were under pressure from some of the big business schools in Europe to produce rankings… What they said was that all rankings are U.S. based…and if Business Week only produces rankings of U.S. schools then that looks as if it is the world ranking and not just a U.S. ranking. So we were under pressure from these schools because they wanted to see a proper international ranking….They wanted an international ranking that had them in it, which is of course what they got. Then they didn’t like it because they didn’t like where they were!” 82
When the FT EE rankings launched, there was only one ranking table (vs. today’s three separate tables for open, custom and overall). This first table included 30 schools, 28 of which offered both open and custom programs at the time (as the FT noted in its footnotes to the table, two of the schools only offered open). Surveys represented 70% of the ranking weight and were conducted over the phone, while the school-provided metrics represented the remaining 30%. There were 9 criteria for the surveys and 4 criteria for the school metrics. Schools were required to have $5M in total executive education revenue.
Interestingly, the 1999 rankings included a published “score” for each school – both an overall score and a score for nearly all of the criteria (several of the criteria were only ranked). This score was based on a scale of 0 to 100, with 100 being the highest possible. The FT continued to use and publish this score for several years.
In 2000, the FT made some significant shifts. They created separate ranking tables for open and custom and then a combined ranking table for schools that offered both. They also changed the minimum requirements: schools needed US$2M in revenue in the program area they wished to participate (open or custom). Finally, they made an innovative change to the surveys: respondents were asked to weight the relative importance of the survey criteria, a practice the FT continues to this today. (Note, however,
80 Wedlin, Linda, (2007),”The role of rankings in codifying a business school template: classifications, diffusion and mediated isomorphism in organizational fields”, European Management Review, Vol 4, pp.24-39 81 Della Bradshaw, (2007),"Business school rankings: the love-hate relationship", Journal of Management Development, Vol. 26 Iss 1 pp. 54 – 60 82 2001 FT article quoted in Wedlin (2006): 130.
100 | P a g e
that the weighting of the overall survey relative to the overall school metrics weighting is still determined by the FT.)
Between 2001 and 2006, the FT continued to “tweak” the rankings, making a number of modifications and enhancements, most notably:
• In 2001, customer and participant survey weightings were increased from 70% to 80% of the ranking. In addition, for custom programs more weight was allocated to individual responses based upon a) the seniority of the individual responsible for specifying the course, b) the size of the client organization, and c) the number of schools with which the client has commissioned customized courses in the last three years.
• In 2002, the FT began using z-scores to calculate the rankings. In addition, they began weighting responses over three years (50/25/25).
• In 2003, the FT stopped reporting scores and switched to the current rank-only format. They also changed the three-year weighting to 40/33/27.
• In 2005, they added a three-year average rank.
Since then, there have only been a few other notable changes. In 2011, the FT removed AMBA from its accreditation standards (AMBA schools were given until 2015 to be accredited by either EQUIS or AACSB). The FT also disclosed its minimum response rates of 20%/20 participants for open programs and 5 minimum responses for custom clients. (Prior to this, the FT had not disclosed a minimum response rate in its methodology sections.) And in 2012, the FT began conducting its surveys online.
Year Key Methodology Points / Changes 1999 Schools with US$5m in revenue, both open and custom
Survey respondents were responsible for making purchase decision 70% of ranking based upon 9 criteria in customer surveys Top potential score = 100 (index format)
2000 Schools with US$2m in revenue (open or custom) Top ten clients for custom, purchasers are surveyed Two flagship programs for open, participants are surveyed For custom, 11 criteria in customer survey make up 70% of ranking, 7 criteria from schools 30% For open, 10 criteria in customer survey make up 70% of ranking, 6 criteria from the schools 30% Weighting of customer criteria determined by respondents, the rest by FT
2001 For custom, 11 criteria in customer survey make up 80% of ranking, 6 criteria from schools 20% For open, 10 criteria in customer survey make up 80% of ranking, 6 criteria from the schools 20% “Sophisticated user” weight added – more schools, more seniority, bigger firms = greater weight
2002 Z-scores added to account both for relative and absolute differences Three-year weighting (50/25/25)
2003 Change from index (top score=100) to rank format (both within categories and overall) Change to each year’s weighting (40/33/27)
2004 No change 2005 Added three year average rank 2006 Top 12 clients for custom 2007 Number of custom clients not disclosed
101 | P a g e
2008 No change 2009 n/a 2010 n/a 2011 n/a 2012 Internationally accredited
For open, 20% survey completion rate and at least 20 respondents For custom, at least 5 clients complete survey
2013 No change 2014 No change
102 | P a g e
Appendix 8: Criteria Evolution
Since 2000, the criteria have been remarkably stable. (We do not know if the underlying questions have been equally stable, but we assume they have been close.) Below are tables outlining how the criteria started in 2000 and the observed changes to them over the years. There appear to have been more tweaks to Custom criteria over the years, which started out with 11 survey and 7 school metric criteria and now has 10 and 5 respectively. Regardless, most of the changes have been simply name changes.
Note that we did not include 1999 in this analysis given that the criteria in the first ranking were not separate for custom and open providers.
FT Open Criteria and Years in Which Changes Were Made (Since 2000)
Year Survey Criteria & Changes School Metrics & Changes 2000 10 Criteria, 70% weighting:
Preparation Course design Teaching materials Faculty Quality of participants
New skills and learning Follow-up Fulfillment Food & accommodation Facilities
6 Criteria, 30% weighting: Women participants International participants Repeat business and growth International location Partners Faculty participation
2001 Survey increased to 80% of weighting School Metrics decreased to 20% of weighting Replaced “Faculty participation” with “Faculty Diversity”
2003 Renamed “Fulfillment” to “Aims Achieved” 2009 Added open revenues to the table (where provided) 2012 Renamed “Teaching materials” to “Teaching methods &
materials”
FT Custom Criteria and Years in Which Changes Were Made (Since 2000)
Year Survey Criteria & Changes School Metrics & Changes 2000 11 Criteria, 70% weighting:
Preparation Course design Teaching materials Faculty New skills and learning
Follow-up Fulfillment Food & accommodation Facilities Value for Money Future use
7 Criteria, 30% weighting: International companies International participants Programmes overseas Consortia Partners/Schools Partners/others Faculty involvement
2001 Survey increased to 80% of weighting School Metrics decreased to 20% of weighting Replaced “Faculty involvement” with “Faculty Diversity”
2002 Eliminated Partners/Others (e.g., partners like management consultancies)
2003 Renamed “Fulfillment” to “Aims Achieved” Renamed “Programmes overseas” to “overseas Programmes” 2007 Eliminated Consortia 2008 Renamed “Course Design” to “Programme Design”
Renamed “Teaching materials” to “Teaching methods & materials”
2009 Added # of Custom Response & Years of Custom Data as well as Custom revenues to the table (where provided)
2010 Eliminated Food & Accommodation 012 Renamed “Teaching materials” to “Teaching methods &
materials”
103 | P a g e
Appendix 9: Some Specific Recommendations on Custom Program Criteria
A recent UNICON report reinforces the fact that custom program clients, in particular, are looking for a short list of synergistic elements that includes:
• Application. UNICON clients want program activities that help participants apply what they learn. This item received the highest rating of the thirty-three items on the survey. Desire for application also appears in the high value placed on having professors who link ideas, principles, concepts, and theories to practical problems, and professors who give real and practical examples during class.
• Integration. Clients want an integrated learning experience. They want the parts of the program to work together in a logical and consistent way, and they want the connections between the parts to be clear. They also want a skilled faculty orchestrator who can integrate program activities.
• Practical and inclusive professors. UNICON clients want professors who link ideas, principals, concepts, and theories to practical problems, professors who give real and practical examples during class, and professors who can facilitate group discussion and encourage participants to share their ideas. (Weldon, UNICON research, 2014)
From our discussions some other things we would add to this list
• One of the key challenges in executive education is translating pure research. There is the question of faculty’s ability to make pure research relevant. Is there a measure for this? Business school research is expensive, and there are questions about its relevance to managers and corporations. A question to consider is: How could the FT rankings influence research to become more pragmatic for managers and provide more value to schools?
• There should be some measure of teaching quality.
104 | P a g e
Appendix 10: 2015 Graphic Showing Topics/Questions behind FT EE Criteria
The 2015 EE magazine had a very interesting graphic on page 22 and 23. In it, the FT reveals the topics behind the questions that form the basis for their criteria. They also share the average rating on the ten-point scale given by respondents to each of these of topics. Interestingly, they were nearly all 8s and 9s.
105 | P a g e
Appendix 11: FAQ for Business Schools
Over the course of conducting this research a variety of questions about the FT EE rankings kept coming up. While we may have addressed many of these questions in the body of this report, we thought a FAQ might be a quick way to answer them directly.
Are the FT EE rankings dominated by European Schools?
As a matter of fact, yes, they are. In 2015, European schools represented 7 of the top 10 combined, 7 of the top 10 in custom, and 6 of the top 10 in open. This is in contrast to the FT MBA rankings, where only 3 of the top 10 were from Europe. (For perspective, Europe represents ~13% of the world’s GDP vs. 19% for the US and 15% for China83.)
To explain the EU dominance in the EE vs. the MBA ranking, some have cited greater salary jumps in the US for MBAs (salaries drive the FT MBA ranking) as well as the European schools’ greater focus upon EE as a means of raising funds (US schools generally receive a greater share from alumni donations)84.
However, as recently as 2013, half of the top 10 EE schools were from the US. We see no reason that US schools cannot compete here.
Still, are the rankings biased in favor of European schools?
The rankings are biased in terms of an international perspective, and unabashedly so. As Della Bradshaw said in a 2014 Times Higher Education interview, “The global perspective definitely characterizes the FT Rankings.”85. This international perspective is reflected in the ranking criteria, many of which are focused on international aspects of the schools, e.g., international clients, international participants, overseas programs, etc.
Given the inherently international nature of European schools, it makes sense that they do well in the rankings. But given the increasingly international nature of all business, it also makes sense that qualified schools anywhere in the world – as long as they have a strong international perspective and solid programs – can compete.
Okay, but how about cultural biases – surely the rankings favor schools in countries that tend to rate things more positively?
This is a real issue, and one that neither researchers in general nor the FT have solved. As the FT said to us in an interview:
83 http://blog-imfdirect.imf.org/2014/03/26/china-size-matters/ 84 http://www.forbes.com/sites/adamgordon/2015/05/21/ft-rankings/ 85 Interview Della Bradshaw -Times Higher Education, 1/7/2014
106 | P a g e
“Culture also has an impact on subjective data that we have not figured out how to remove. The Germans are going give you a 7 when they think you’re very good. The Latins will give you a 10 and the Dutch a 6. Even seasonality is an issue that we don’t know how to reduce…the Swedes don’t like the fact that we do EE in February when everyone in Scandinavia is depressed!”86 Despite that, we are not sure if this is a big issue in the EE rankings. It seems that US schools would do better if that were the case. In addition, in 2015 the FT shared the ratings by criteria and questions – and they were generally 8s and 9s (see Appendix 10: 2015 Graphic Showing Topics/Questions behind FT EE Criteria). Best advice: think carefully about the countries from which your nominated programs come.
Do the rankings favor big schools?
Arguably, yes, for several reasons. First, obviously, the school can meet the minimum qualifications to participate in terms of revenue. Second, the more programs a school offers, the more flexibility they have in which programs they nominate to be surveyed by the FT – essentially, they can pick their best programs and focus on making sure those go well. Third, a larger school is likely to have more resources to devote towards administration and follow-up of the ranking. Fourth, being larger, the school has probably existed longer, has degree programs and is probably accredited by AACSB and/or EQUIS. Finally, a larger school is more likely to have an international footprint and/or partnerships – which, as discussed, the FT unabashedly favors.
All that said, we do not think this should necessarily dissuade schools form participating in the ranking. For one, when we interviewed L&D professionals, it was clear that they saw beyond the ranking position and took a more nuanced view of choosing an EE provider. Often, they may be looking for a school that is strong in a particular region and/or strong in a particular category. In this marketplace, where you are in the ranking may not be as important as just being in the ranking (though significant falls in the ranking would not be a good thing). We do not think small schools should be afraid of participating in the ranking as long as they have reasonable expectations and put the right amount of effort into it.
Are the rankings an “apples to apples” comparison of programs?
No, they are not. Schools are allowed to choose which programs they include. In the case of open, they are required to be general in nature. However, that does not mean they are the same type of programs. Similarly for custom, the programs surveyed are often very different – although for custom, they do ask for schools to characterize the programs as strategic, general or functional in nature, with the strategic programs receiving the greatest weighting.
86 Interview with the FT, October 2014
107 | P a g e
Despite this, we are sympathetic to the FT here. They are trying to measure something that is arguably a lot more complex and harder to measure than an MBA program. They do it in a very professional manner. And they have been clear and consistent over the years. We certainly think they could do more, but we do not necessarily think that someone else would do it better.
Do the rankings measure statistically meaningful differences between the schools?
As we were told by one statistician, “meaningful” is not a statistical term. However, the short answer is yes, the rankings do measure statistical differences. As we explained in the “How the Rankings Work” section, the FT uses a statistical measure called the z-score to identify statistical differences in the data. When there is no difference, the FT does the right thing and shows schools as having the same rank.
We believe there are more serious critiques to be made regarding the methodology, most notably in the areas of sampling methodology and criteria weightings. The sampling methodology is the most troublesome. Since schools nominate the programs that are to be surveyed, the resulting data is hardly randomized or representative in any way. It is “cherry picked.” Yes, the playing field is level in the sense that every school has the option of picking its best programs. But, to describe each sample representative of the school is by no means accurate or fair. Another major problem with the sampling methodology is the underlying lack of comparability between the programs. Frankly, there is just no way of knowing how similar programs truly are. Is it fair, then, to rank them against each other?
Weightings are another area that could be critiqued. The FT assigns a number of weights that are quite subjective and/or even biased, especially to favor of bigger and/or more international of schools. As a result, the data is hardly “clean.” We would love to see the FT share all of its data so that some of these biases could be removed.
Despite all of this, we repeat that we are sympathetic to the FT here. They are trying to measure something that is extremely complex and nuanced. They are highly professional in their approach and, in our interviews with them, very cognizant of the limitations in their rankings. What is most important is that readers of the rankings understand what they are reading.
Can the rankings be gamed?
What can we say to this? What does “gamed” really mean? If you mean, “Can schools cheat?,” we would say no. From everything we have seen, the FT makes sure this process is handled in an extremely “above board” fashion. It would not hurt for the FT to audit the EE ranking, but the fact they are auditing the MBA ranking gives us a high degree of confidence in the integrity of this ranking.
At the same time, in some respects all rankings are inherently a game. And perhaps the fact that EE programs are nominated by the schools makes these rankings even more of a game. What is more, the FT EE rankings are not a completely fair game because it is biased in favor of big, international schools that offer traditional types of programs.
108 | P a g e
Hopefully, the rules of the game are pretty clear to readers by now. We see no reason why a school cannot do well in the rankings as long as it has reasonable expectations, puts in the right amount of focus and effort, and takes the time to educate its clients and peers.
Can you recreate the rankings using the data provided?
Alas, no. We tried this ourselves and even asked the FT if it is possible. It is not.
I’ve noticed the number of schools in the rankings increase – where is that coming from?
Per Table 22 and 23, the majority of new schools in the last 10 years have come from the developed economies of the Europe and North America. The FT does not solicit schools. Instead, schools have come to them.
If you have been waiting for the FT to call, we suggest you stop waiting. If you want to be included in the annual ranking analysis, pick up the phone and give them a call.
Our school offers highly specialized vs. general programs – what should we do?
This is a tough one, but our best advice would be to sit on the sidelines. The FT EE rankings are focused on general management programs. We do not think that schools taking more of a niche approach to the marketplace will get the full credit they deserve, especially if they focus on things such as leadership, entrepreneurship, innovation, technology, and other hot-button issues in executive education today.
All that said, we know that the FT is aware of this issue and considering changes. In our interview, for example, Della addressed the topic of entrepreneurship for both the EE and the MBA rankings:
“For some time we’ve been collecting and measuring entrepreneurship data in our survey, but we haven’t used it yet. Not yet sure how to use it. In the MBA alumni survey we ask if they have started their own business, whether that is fulltime or a hobby, whether they’ve raised money, who from (VCs, friends, family, etc.). This isn’t just about entrepreneurship programs. We wondered if this data might help in the normal degree program rankings.”
Our hope is that the EE rankings evolve to reflect the more diverse nature of this marketplace so that schools with niches can truly shine.
Our school has a strong reputation overall, but our EE ranking is much weaker – why should we risk it?
This is a tough situation to be in. There are a handful of schools that have stellar reputations overall (for their university and/or for their MBA program) but do not do as well in the EE rankings. (See Appendix 17: Correlation between MBA and EE Rankings? for more on EE and MBA Rankings correlation.) As we have said, we certainly think that most L&D professionals see beyond this. Nonetheless, we understand that no school wants to receive a ranking that it feels is sub-par.
109 | P a g e
“Some new entrants do manage to enter the rankings in a high position. This was true both of the Saïd Business School (Oxford) and the Judge School (Cambridge)” pointing to the key role played by the strength of the university brand. 87
Our belief is that commitment and focus are the best answers here. First, the school should commit to do as well in the ranking as it can, investing the time and effort to insure that it maintains or improves its ranking year in and year out. Second, the school should focus on the criteria where it truly excels and insure that it performs well on them so that it can rightfully claim excellence in key areas. Finally, the school should think carefully about what its true competitive set is within the ranking – e.g., others players in a region – and focus on its performance relative to them.
We realize every school’s situation is unique – Deans change, commitment to EE sometimes wavers, etc. – but we do believe that a sustained and focused commitment to the EE ranking will bear fruit over time.
We are still on the fence: should our school participate in the FT rankings?
Rankings are not going to go away. They are a great vehicle to attract readers and advertise. If the FT stopped today, someone would step in, and probably do it less well. Thus, business schools have to figure out the best way to deal with them. As we said in the conclusion, the fact that business schools are placed on this podium, and not boutiques and consultants, is an amazing opportunity for the schools. We say, embrace it!
If we put more effort into the rankings, will it translate into better results?
It is a fair question to ask. Like any organization, business schools need to prioritize their efforts in order to make sure they are getting a return for it. While we cannot identify explicit causality in our data, we did see some relationship between ranking satisfaction and effort, particularly on the custom side.
Each school’s circumstance is unique, and so any decision on how much effort to put into the ranking really has to be on a case-by-case basis.
87 Wilson, 2011, 462
110 | P a g e
Appendix 12: Which Ranking Are We Talking About? Comparisons of Rankings
Time and again during the authors’ interviews it was necessary to re-specify that our focus was on the FT Executive Education rankings, and to re-ask questions to be sure that the interviewee’s response indeed applied to the FT EE rankings and not to one of the myriad of other rankings that exist, most of which are unrelated to executive education.
In Wikipedia88 we find 21 global education rankings listed, plus a long list of regional and national rankings. The criteria used vary widely, from the number of exchange students attending or traveling from each university (QS's Asian University Rankings), to the number of billionaire alumni (CUAA), or to the difficulty of the entrance exam (Hensachi Rankings), or to those with the highest scientific impact (European Union). There are also composite reports, a ranking based upon the rankings, such as Poets & Quants which has an EMBA ranking culled from the FT, US News, the Economist, and Business Week. In one of our corporate L&D interviews, it was mentioned that a consultancy called Entendéo created their own version of the rankings, something that combined the work of the FT with several other rankings.
Different rankings rank different things, and it is not always clear to the reader what is being evaluated.
“Rankings should highlight core capabilities that these programs get at: If you are #1, what are you #1 in? And if so, is that because you are the best at it or you are just doing what no one else is doing? That way you create areas of focus and have a plethora of choices.” – L&D Interview
“When I spend $5400 on a program with an executive I go to schools I know already.” – L&D Interview
“The MBA ranking is the most important – it’s the proxy for the overall institution.” – Expert Interview
The Academic Ranking of World Universities (ARWU) compiled by the Shanghai Jiao Tong University and now maintained by the Shanghai Ranking Consultancy, has provided annual global rankings of universities since 2003, making it the earliest of its kind. In 2014, the 5 top-ranked schools, in order, were Harvard, Stanford, MIT, the University of California-, and Cambridge. The ranking is funded by the Chinese government, and its initial purpose was to measure the gap between Chinese and "world class" universities. One of the primary criticisms of ARWU's methodology is that it is biased towards the natural sciences and English language science journals over other subjects. Moreover, the ARWU is known for "relying solely on research indicators,” and "the ranking is heavily weighted toward institutions whose
88 http://en.wikipedia.org/wiki/College_and_university_rankings
111 | P a g e
faculty or alumni have won Nobel Prizes." It does not measure "the quality of teaching or the quality of humanities.”89
The Times Higher Education World University Rankings90 2014-2015 list the best global universities and provide the only international university performance tables to judge world class universities across all of their core missions: teaching, research, knowledge transfer, and international outlook. The top universities rankings employ 13, carefully calibrated, performance indicators to provide the most comprehensive and balanced comparisons available, which are trusted by students, academics, university leaders, industry, and governments. The top 5 schools, in order, are Caltech, Harvard, Oxford, Stanford, and Cambridge.
The Times Higher Education World Reputation Rankings91 2015 employ the world's largest invitation-only academic opinion survey to provide the definitive list of the top 100 most powerful, global university brands. A spin-off of the annual Times Higher Education World University Rankings, the reputation league table is based on nothing more than subjective judgement. But it is the considered expert judgement of senior, published academics – the people best placed to know the most about excellence in our universities. The top 5 schools ranked, in order, were Harvard, Cambridge, Oxford, MIT, and Stanford.
The Eduniversal Best Masters Ranking92 is a true assessment of global masters and MBA programs, designed to help prospective students determine the best choice for their graduate studies. This is the only ranking that evaluates individual masters programs worldwide in 30 fields of study. Each year this ranking evaluates more than 12,000 postgraduate programs. The final ranking is determined through a global survey of 5,000 international recruiters, 800,000 students, and over 10,000 representatives from 1,000 academic institutions in 154 countries.
When searching online using “rankings executive education” or “Financial Times Executive Education,” the searcher will first go to one of the FT sites, followed immediately by Bloomberg (see Social Media Appendix 6: Social Media and Executive Education Rankings).
GraduatePrograms.com ran a survey to see which schools ranked highest for networking.93 Probably skewed towards MBA alumni, the top 5 schools were, in order, Stanford, Columbia, Harvard, Insead, and Texas A&M.
One might ask if there a need for multiple rankings?
89 http://www.shanghairanking.com/ 90 https://www.timeshighereducation.co.uk/world-university-rankings/2014-15/world-ranking 91 https://www.timeshighereducation.co.uk/world-university-rankings/2015/reputation-ranking 92 http://www.best-masters.com/ 93 http://www.graduateprograms.com/top-business-schools-for-networking/
112 | P a g e
“I think the one from the FT is good enough.” – Open Enrollment Participant Interview
“There should be enough rankings so that every school can be #1 in one ranking” – Expert Interview
But for executive education, do we actually have multiple rankings now, on a global scale? There are national rankings, but is there now any credible ranking that rivals the FT? Our answer would be, not at this time.
“Bloomberg Businessweek said yesterday that it will no longer publish rankings on Executive MBA or executive education programs. In an email sent to business school officials, the magazine gave no reason for giving up on its EMBA and Exec Ed rankings which were launched in 1991.” – Poets & Quants for Executives, March 17, 2015
“In the 17 years the FT has produced executive education provider evaluations, it has become without peer in this part of the rankings industry and is widely accepted among executive education department themselves as a credible industry judge.” – Expert Interview
113 | P a g e
Appendix 13: Financial Times Surveys Schedule
Ranking Surveys Open Surveys Close Publication
Global MBA September October January
Online MBA January February March
Executive Education February March May
Masters in Finance March April June
Masters in Management April May September
EMBA June July October
European Business Schools N/A N/A December
114 | P a g e
Appendix 14: Economic Backdrop for Executive Education Industry
(Excerpted from the UNICON State of the Industry Survey, 2013/14)
93 schools participated
Average revenue growth of 7.4% per annum from 2011/12 to 2013/14.
• Europe = +6.0% • LATAM = +84.3% • USA/Can = – 2.8% • Asia/Oceania = +15.1%
115 | P a g e
Appendix 15: Berlin Principles on Ranking of Higher Education Institutions
The Berlin Principles on Ranking of Higher Education Institutions94, published on May 20, 2006, for the purpose of improving all rankings of higher education institutions, provides 16 pragmatic requirements on A) Purposes and Goals of Rankings, B) Design and Weighting of Indicators, C) Collection and Processing of Data and D) Presentation of Ranking Results. In this appendix, we compare these principles to what we understand the FT EE rankings to include:
A) Purposes and Goals of Rankings
1. Be one of a number of diverse approaches to the assessment of higher education inputs, processes, and outputs. Rankings can provide comparative information and improved understanding of higher education, but should not be the main method for assessing what higher education is and does. Rankings provide a market-based perspective that can complement the work of government, accrediting authorities, and independent review agencies.
YES
2. Be clear about their purpose and their target groups. Rankings have to be designed with due regard to their purpose. Indicators designed to meet a particular objective or to inform one target group may not be adequate for different purposes or target groups.
YES, BUT FT RANKINGS COULD USE MORE CLARIFICATION, WHILE ACCEPTING THAT SOME CONSTITUENCIES WILL NOT DIG DEEP TO UNDERSTAND THE CAVEATS
3. Recognize the diversity of institutions and take the different missions and goals of institutions into account. Quality measures for research-oriented institutions, for example, are quite different from those that are appropriate for institutions that provide broad access to underserved communities. Institutions that are being ranked and the experts that inform the ranking process should be consulted often.
NO, BUT IF THE FT EE RANKINGS ARE QUALIFIED AS BEING FOCUSED ON MAINSTREAM BUSINESS SCHOOL EXECUTIVE EDUCATION WORK, THAN THIS QUESTION DOES NOT MATTER AS MUCH.
4. Provide clarity about the range of information sources for rankings and the messages each source generates. The relevance of ranking results depends on the audiences receiving the information and the sources of that information (such as databases, students, professors, employers). Good practice would be
94 http://www.ihep.org/research/publications/berlin-principles-ranking-higher-education-institutions
116 | P a g e
to combine the different perspectives provided by those sources in order to get a more complete view of each higher education institution included in the ranking.
YES
5. Specify the linguistic, cultural, economic, and historical contexts of the educational systems being ranked. International rankings in particular should be aware of possible biases and be precise about their objective. Not all nations or systems share the same values and beliefs about what constitutes “quality” in tertiary institutions, and ranking systems should not be devised to force such comparisons.
PROBABLY NOT, AS THE FT EE RANKINGS ASPIRE TO AN INTERNATIONAL TEMPLATE OF WHAT A BUSINESS SCHOOL IS.
B) Design and Weighting of Indicators
6. Be transparent regarding the methodology used for creating the rankings. The choice of methods used to prepare rankings should be clear and unambiguous. This transparency should include the calculation of indicators as well as the origin of data.
YES, TO A LARGE DEGREE
7. Choose indicators according to their relevance and validity. The choice of data should be grounded in recognition of the ability of each measure to represent quality and academic and institutional strengths, and not availability of data. Be clear about why measures were included and what they are meant to represent.
PARTIALLY
8. Measure outcomes in preference to inputs whenever possible. Data on inputs are relevant as they reflect the general condition of a given establishment and are more frequently available. Measures of outcomes provide a more accurate assessment of the standing and/or quality of a given institution or program, and compilers of rankings should ensure that an appropriate balance is achieved.
YES
9. Make the weights assigned to different indicators (if used) prominent and limit changes to them. Changes in weights make it difficult for consumers to discern whether an institution’s or program’s status changed in the rankings due to an inherent difference or due to a methodological change.
YES
C) Collection and Processing of Data
117 | P a g e
10. Pay due attention to ethical standards and the good practice recommendations articulated in these Principles. In order to assure the credibility of each ranking, those responsible for collecting and using data and undertaking on-site visits should be as objective and impartial as possible.
YES
11. Use audited and verifiable data whenever possible. Such data have several advantages, including the fact that they have been accepted by institutions and that they are comparable and compatible across institutions.
YES, WE THINK SO.
12. Include data that are collected with proper procedures for scientific data collection. Data collected from an unrepresentative or skewed subset of students, faculty, or other parties may not accurately represent an institution or program and should be excluded.
NO OR PARTIALLY. THE SURVEY DATA IS SKEWED BY THE SCHOOLS SELECTING THE PROGRAMS.
13. Apply measures of quality assurance to ranking processes themselves. These processes should take note of the expertise that is being applied to evaluate institutions and use this knowledge to evaluate the ranking itself. Rankings should be learning systems continuously utilizing this expertise to develop methodology.
YES, AT LEAST PARTIALLY, BUT WHAT IS DONE IN THIS AREA IS NOT ALWAYS TRANSPARENT TO THE SCHOOLS BEING RANKED.
14. Apply organizational measures that enhance the credibility of rankings. These measures could include advisory or even supervisory bodies, preferably with some international participation.
YES, IF WE ASSUME THAT AACSB AND EQUIS PROVIDE THIS IN A SECONDARY WAY.
D) Presentation of Ranking Results
15. Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed. This way, the users of rankings would have a better understanding of the indicators that are used to rank institutions or programs. In addition, they should have some opportunity to make their own decisions about how these indicators should be weighted.
YES AND NO. A CHOICE OF HOW RANKINGS ARE DISPLAYED IS NOT PROVIDED, IS IT? SAME FOR WEIGHTING.
“I’d love to have a digital version of the rankings where I could put in my own weightings and create my own rankings according to what’s important for me. Create a great interface and let me do my own
rankings.” – L&D Leader, Europe
118 | P a g e
16. Be compiled in a way that eliminates or reduces errors in original data, and be organized and published in a way that errors and faults can be corrected. Institutions and the public should be informed about errors that have occurred.
YES
119 | P a g e
Appendix 16: List of Interviewees by Institution and Role
Organization Geography Type Unilever Europe L&D Tetrapak Europe L&D Carrefour Europe L&D ABB Europe L&D F Hoffmann La Roche Europe L&D Danone Europe L&D Nestle Europe L&D Citizens Bank #1 US/Canada L&D Citizens Bank #2 US/Canada L&D Unicredit Europe L&D Mirabaud Europe OE Participant GE Healthcare Europe OE Participant BAT Asia OE Participant Ernst & Young Africa OE Participant Nestlé Europe OE Participant Financial Times #1 Europe Rankings Expert Financial Times #2 Europe Rankings Expert IMD #1 Europe Rankings Expert IMD #2 Europe Rankings Expert Grenoble Ecole de Management Europe Rankings Expert US Exec Ed Organization US/Canada Rankings Expert Porto Business School Europe Rankings Expert Lorange Inst. Of Business Europe Rankings Expert Consultant #1 US/Canada Rankings Expert Consultant #2 US/Canada Rankings Expert Uppsala Universitet Europe Rankings Expert Consultant #3 US/Canada Rankings Expert EQUIS/EFMD Europe Rankings Expert AACSB #1 US/Canada Rankings Expert AACSB #2 US/Canada Rankings Expert MIT Sloan US/Canada Business School EE Ashridge Europe Business School EE IMD Europe Business School EE ITAM Latin America Business School EE American Univ in Cairo Africa Business School EE INSEAD Europe Business School EE CKGSB China Business School EE Columbia US/Canada Business School EE
120 | P a g e
Organization Geography Type HEC Paris Europe Business School EE CEIBS Asia Business School EE Skolkovo Russia Business School EE AIM WA – UWA Business School Australia Business School EE
121 | P a g e
Appendix 17: Correlation between MBA and EE Rankings?
It is interesting to consider if is there a strong correlation between the MBA and the EE rank? Could a school with a highly ranked MBA bask in that glory and have it shine on its EE programs? We looked at this for both open and custom. There is correlation, but we cannot prove causality.
Using 2014 data, we created a scatterplot for both open and custom rankings vs. their respective MBA rankings from the FT. There was a correlation, with an R2 of 0.39 in the case of open and 0.30 in the case of custom. We did this same exercise for previous years and found similar numbers. Only four schools in the EE open program top 20 were not in the MBA top 20. Conversely, seven of the top 20 MBA schools were outside the EE open program top 20.
In the custom ranking, there were seven schools that were in the top 20 of the FT MBA rankings, as well as the FT EE custom program ranking. There were also seven that were in the top 20 of EE custom programs, but were not in the top 20 MBA programs, and six schools that were in the top 20 MBA programs, but not in the top 20 EE custom programs.
IMD Chicago Booth HEC Darden Insead
HBS Iese Stanford Esade
Ross LBS
Oxford: Saïd Kellogg Olin ESMT Wharton UCLA Rotman Ivey
Columbia Cranfield St Gallen Ceibs MIT SDA
IE Vlerick Schulich
Sauder AGSM EMLyon Indian Institute
Tias
R² = 0.392
0
10
20
30
40
50
60
70
020406080100120
2014
FT
Ope
n Ra
nkin
g
2014 FT MBA Ranking
Open – Relationship Between FT Open and MBA Rankings – 2014
122 | P a g e
HEC IESE IMD Esade Cranfield Stanford LBS UNC
SDA Mannheim IE Chicago
Georgetown HBS INSEAD
Oxford Olin Wharton
Kellogg Carnegie Mellon MIT Darden Ross CEIBS
UCLA Columbia Vlerick EMLyon Toronto St. Gallen Boston Univ. Ivey
Emory ESMT
York Austin AGSM Imperial Manchester
GSB Tias
Rotterdam
R² = 0.298
0
10
20
30
40
50
60
70
80
020406080100120
2014
FT
Cust
om R
anki
ng
2014 FT MBA Ranking
Custom – Relationship Between FT Custom and MBA Rankings – 2014
123 | P a g e
Appendix 18: Comparison between FT EE and MBA Rankings
As we mentioned in our introduction, the proliferation of rankings for business schools has contributed to a somewhat confusing ranking marketplace. Clearly, MBA rankings have been around longer, and there are many more of them. Given their prevalence, we think it is important to compare and contrast how EE rankings are different from MBA rankings. To keep things simple, we compared the FT EE rankings with its own MBA rankings. Per the table below, they are actually quite different.
Methodology FT MBA Ranking FT EE Rankings International accreditation? Yes Yes Other eligibility requirements? 4 years of offering MBA; at least
30 students graduate every year $2MM in revenue by category
Main inputs and weighting Alumni surveys 59% School metrics 31% Research rank 10%
Surveys 80% School metrics 20%
Min Response Rate 20% of students + min 20 alumni 20% for Open + min 20 responses 5 Custom Clients
Individual Criteria Weighting Set by FT. Note that “Salary” and “Salary Increase” represent 20% each (40% combined)
Set by participants. Individual criteria weightings range between 7 and 9%.
# Survey Criteria 8 10
# School Metric Criteria 11
5 for Open 6 for Custom
Multi-year weighting? Yes Yes Z-scores? Yes Yes Auditing of school data? Yes, KPMG annual Data checks done by FT One of the main differences is what the rankings measure. Whereas the FT MBA ranking is based upon 3 main components (alumni surveys, school-provided metrics, and a ranking of research, which is based upon number of faculty publications), the EE rankings are based upon two main components (the surveys and the school-provided metrics). The EE rankings have no such significant research component, although to be fair, we know from looking at the 2015 report that there are questions that touch upon research, such as “integration of research,” that are then rolled up into the criteria (see Appendix 10: 2015 Graphic Showing Topics/Questions behind FT EE Criteria).
Another significant difference is how criteria weightings are determined. For the FT MBA ranking, the weights are set by the FT and are heavily weighted toward alumni salary and salary increases – the two of which combined represent 40% of the ranking. For the EE criteria, the weightings are determined by the survey respondents. Interestingly, no single criteria is greater than ~9% (from our review of the data, this has been true over the years). The MBA ranking also includes a much larger number of school-provided metrics (11 vs. 5).
124 | P a g e
Also of note, the school-provided MBA rankings data are audited annually by KPMG. To our knowledge, no such audit takes place for the EE Rankings, but the FT did stress to us that they run extensive data checks, as stated earlier.
125 | P a g e
Appendix 19: Variability in Position
Listed in order of spread, from greatest to lowest.
Variability in Custom Program Ranking, 2010 – 2014 School Min Max Spread Std. Dev. Rotterdam School of Management, Erasmus University 32 72 40 15.1 Boston University School of Management 5 44 39 15.6 Insper 13 52 39 14.5 Stanford Graduate School of Business 8 42 34 14.5 UCLA: Anderson 26 57 31 11.6 MIT: Sloan 31 60 29 13.5 ESCP Europe 27 56 29 11.4 Politecnico di Milano School of Management 37 65 28 12.6 AGSM at UNSW Business School 31 59 28 10.8 ESMT – European School of Management and Technology 23 51 28 11.3
University of Virginia: Darden 24 50 26 9.8 SDA Bocconi 11 37 26 10.0 Nova School of Business and Economics 48 73 25 17.7 Ceibs 34 59 25 12.4 Fundação Dom Cabral 3 27 24 9.4 University of Michigan: Ross 33 55 22 10.0 Northwestern University: Kellogg 18 40 22 8.4 Carnegie Mellon: Tepper 30 51 21 14.8 Tias Business School 46 67 21 9.2 London Business School 9 30 21 9.1 IAE Business School 18 39 21 7.8 Grenoble Graduate School of Business 53 74 21 8.2 University of North Carolina: Kenan-Flagler 8 29 21 8.5 Columbia Business School 17 37 20 7.9 University of Toronto: Rotman 42 62 20 8.3 York University: Schulich 37 57 20 8.8 Universidad Adolfo Ibanez 56 75 19 9.7 Essec Business School 13 32 19 9.1 University of Texas at Austin: McCombs 40 58 18 6.5 BI Norwegian Business School 61 79 18 7.5 Católica Lisbon School of Business and Economics 43 61 18 7.3 Western University: Ivey 27 45 18 7.5 Georgetown University: McDonough 17 34 17 12.0 Washington University: Olin 24 41 17 12.0 University of Cape Town GSB 49 65 16 7.6
126 | P a g e
University of Pennsylvania: Wharton 10 26 16 6.7 Melbourne Business School, Mt Eliza 36 51 15 6.0 Aalto University 41 56 15 6.1 Harvard Business School 3 18 15 6.3 Insead 8 22 14 6.5 EMLyon Business School 27 41 14 6.1 Nyenrode Business Universiteit 63 76 13 6.0 Vlerick Business School 38 51 13 5.1 Henley Business School 38 50 12 4.3 Irish Management Institute 57 69 12 5.5 Iese Business School 3 15 12 5.2 Babson Executive Education 7 19 12 4.4 Stockholm School of Economics 29 40 11 4.6 University of Oxford: Saïd 12 23 11 4.2 Universidad de los Andes 47 58 11 7.8 University of St Gallen 40 51 11 4.5 University of Pretoria, Gibs 42 53 11 6.1 USB Executive Development 57 68 11 4.0 Kelley Executive Partners at Indiana University 25 35 10 4.6 Emory University: Goizueta 47 56 9 6.4 IE Business School 10 19 9 3.6 Ashridge 11 20 9 3.8 Incae Business School 54 63 9 4.9 Cranfield School of Management 7 15 8 3.6 Edhec Business School 19 26 7 3.5 University of Chicago: Booth 15 22 7 3.2 Macquarie Graduate School of Management 58 65 7 4.9 Ipade 11 18 7 2.7 Eada Business School Barcelona 59 66 7 3.3 Queen's School of Business 46 52 6 4.2 Center for Creative Leadership 4 10 6 2.3 Porto Business School 59 65 6 2.6 Thunderbird School of Global Management 19 24 5 1.8 NHH 67 71 4 2.0 Esade Business School 18 22 4 2.2 Warwick Business School 43 45 2 1.4 IMD 5 7 2 0.9
127 | P a g e
Digging into this issue, we looked at the average range as well as the average absolute deviation for schools with 5 years of consecutive data. We compared the FT EE Rankings to its own MBA Ranking. And interestingly, the variability for custom was similar to the MBA’s while the variability for open was less.
Average Range and Absolute Deviation among Schools with 5 Years Consecutive Data
Measure MBA Custom Open Average Range 16.8 16.9 9.8 Average Absolute Deviation 5.4 5.2 3.2
Additional research by experts with background in related mathematical modeling and statistical analysis would be helpful to further understand this dynamic.
128 | P a g e
Appendix 20: Z-scores Explained
(The following background was provided by one of our expert sources and is being used with permission.)
The standard score (more commonly referred to as a z-score) is a very useful statistic because it (a) allows us to calculate the probability of a score occurring within our normal distribution and (b) enables us to compare two scores that are from different normal distributions. The standard score does this by converting (in other words, standardizing) scores in a normal distribution to z-scores in what becomes a standard normal distribution.
How the Z-score is used
The z-score is associated with the normal distribution and it is a number that may be used to: • tell you where a score lies compared with the rest of the data, above/below mean. • compare scores from different normal distributions
Let's take a closer look at each of these uses.
Z-score and distance from the mean, and Z-table
The Z-score is a number that may be calculated for each data point in a set of data. The number is continuous and may be negative or positive, and there is no max/min value. The z-score tells us how "far" that data point is from the mean. This distance from the mean is measured in terms of standard deviation. We may make statements such as "the data point (score) is 1 standard deviation above the mean" and "the score is 3 standard deviations above the mean,” which means the latter score is three times further from the mean.
Comparing Scores from Different Types of Questions – Example. The score for each student of a quiz is entered into a spreadsheet and a histogram created from the data. Suppose the histogram is bell-shaped, symmetrical about mean of 50 and st. dev. of 10. What can we say about someone with a score of:
• 50? The z-score is (50-50)/10 = 0. Interpretation: student score is 0 distance (in units of standard deviations) from the mean, so the student has scored average.
• 60? The z-score is (60-50)/10 = 1. Interpretation: student has scored above average – a distance of 1 standard deviation above the mean.
• 69.6? The z-score is (69.6-50)/10 = 1.96. Interpretation: student has scored above average – a distance of 1.96 above the average score.
Now, you might say after these examples that the z-score has not told you anything you cannot see without doing any calculations. But the z-score can tell you a bit more. Because not only can it say whether a score is above or below the mean by so many st. dev., but it can tell you what proportion of the data is below or above a particular score. In other words you can make statements such as "so and so got X marks in a quiz. The corresponding Z-score is Y, and we can say that only Z per cent of students scored higher". And this is where the z-table comes in. Looking back the final example, the z-score was
129 | P a g e
1.96. Now, if we look in the z-table, we find that 2.5 per cent of the scores are above 1.96. So we can say that in the population this student did better than 97.5 per cent of students, or that only 2.5 per cent of students scored higher.
130 | P a g e
Appendix 21: FT Criteria for Inclusion – Executive Education Programs
(The following is being included with permission from the Financial Times)
Financial Times Ranking of Executive Education Programmes
Criteria for inclusion
The following are the main criteria schools must meet in order to participate in the annual Financial Times rankings:
Open Enrolment Programmes
1. All participating business schools should be AACSB or Equis accredited. 2. Participating schools should have had an income (revenues) of US$2m or more in
NON-DEGREE open enrolment programmes between January 1 and December 31 in the year preceding the ranking.
3. Each school must submit up to two general management programmes AND up to two advanced management programmes for the rankings. We require a minimum of 100 participants overall – 50 from each programme type. The school must provide the email addresses of all the participants on the selected programmes that ran during the previous year.
Please note: for a school to be included in the final ranking table, we require a response rate of at least 20 per cent.
The selected general programme(s) should be mid-level and non-functional i.e. general management development. These programme(s) should be at least three days in length. The more senior programme(s) should be AMP or equivalent and should be at least five days in length.
Online questionnaires are available in French, German, Spanish, Portuguese or Italian. Organizations accredited by neither the AACSB nor Equis may be able to participate in both rankings at the discretion of the FT, provided they clearly provide services that directly compete with business schools in executive education.
© Copyright The Financial Times Limited 2011. "FT" and "Financial Times" are trademarks of The Financial Times.
131 | P a g e
Customised programmes
1. All business schools should be Equis or AACSB accredited. 2. Participating schools should have had an income (revenues) of US$2m or more in
NON-DEGREE customised programmes between January 1 and December 31 in the year preceding the ranking.
3. Each school should be able to supply a list of at least 20 different custom clients from the previous year, who will be asked to complete an online survey.
4. No client should be surveyed more than once. If you have fewer than 20 clients or wish to survey a client more than once, please contact us at the email address below. Please note: for a school to be included, we require a minimum response rate of at least 5 clients. The selected programmes should be at least three days in length. Online questionnaires are available in French, German, Spanish, Portuguese or Italian. Organizations accredited by neither the AACSB nor Equis may be able to participate in both rankings at the discretion of the FT, provided they clearly provide services that directly compete with business schools in executive education.
© Copyright The Financial Times Limited 2011. "FT" and "Financial Times" are trademarks of The Financial Times.
132 | P a g e
Appendix 22: FT Interview Transcript
Financial Times: Della Bradshaw & Laurent Ortmans
October 31, 2014
Interview by: Jim Pulcrano
1. Are you happy with the way the FT business school rankings have been received by the world? Della: I think we are. When I think of the EE rankings, yes, I am. And people obviously respond to what we do, so I know we’re noticed. We get regular inquiries… why are we here, why not here… we have an impressive program and the customers say so, but we’re ranked low….
Laurent: We’re judging the delivery of the program.
Della: The delivery of the program and not just the program.
Laurent: All the schools in the rankings are accredited, so they’re all good.
Della: Degree program audience is different; it is individuals. EE is companies. Some of the people looking at our rankings aren’t even going themselves. Sending people, or approving their attendance. There isn’t the same level of scrutiny by individual readers. For a 2-year MBA, YES! But if it’s an HRM, and his CEO has just told him he needs to create or find a program, he just says, where do I start, OK, let’s go for the top 10… they don’t really look closely at the data….
(prompt from JTP on whether they get backlash from readers): With EE, never. But with degree programs it is very emotional… my school deserves to be at the top….
2. Do the rankings fit into the mission of the FT, and, if so, how? Della: Yes, very much so. Every reader of the FT will at some point be in contact with a business school, either to recruit, send people, check the background of someone they’re hiring or attending themselves. The rankings fit perfectly with our mission.
3. When you step back and look at the business school world, what changes have you noticed these past 16 years?
Della: Oh yes! From the very beginning I remember people saying that distance-learning was going to become a bigger and bigger part of education, and that it would be a bigger part of their purchase…. Took many years, but it is happening.
There has been a big change in who decides on Exec Ed programs. Used to be that HR or someone in the company was heavily involved, whereas now it is much more likely to be the individual.
And with that, a shift in who pays. Much more often the individual now. Even for open programs, not just Executive MBAs.
133 | P a g e
And we’ve seen more use of custom programs over the years, moving from open to custom.
And location has become a more important issue. More people are working outside their home region and more likely to choose a school that isn’t necessarily in their home town. The international aspect is more important. The participants have more mobile jobs and expect their programs to be mobile too.
Also, it used to be that the company went to the school (speaking of custom programs), but today we see more and more of the school going to the company.
And then there’s geography, talking India and China, schools need to go where the business is…
4. Has the FT considered using the world-class expertise you’ve built up with the business school ranking to pursue other industries? Even other kinds of professional schools?
Della: We do a listing of LLMs, law degrees. It’s a listing, not a ranking. To be honest, it’s never taken off. Jurisdiction is so important in law. It’s very local. The Times owns this market in the UK. Now looking at combined law and business degrees, like the Insead/Sorbonne masters. That ranking would fit better with the FT’s mission.
5. What challenges do you have with doing the MBA and EE rankings? Della: Our biggest challenge is getting enough response data for the EE custom rankings. Schools need to have at least 5 clients to be included, and if it is low, then there can be a lot of variability in the rankings. This tends to smooth out if the school has a large number of clients. And when you have a $2 million cut-off for minimum revenue, a recession can cause a lot of schools to drop out. Our other challenge is the amount of data checking that we need to do, and cleaning up of the EE data. The MBA is simpler and clearer and you almost always have plenty of responses. (prompt from JTP, does anyone look at both rankings and react to this?) No, nobody seems to be comparing MBA and EE. The schools are probably siloed and don’t even talk across the departments about how they did in the rankings. 6. If you could have a magic wand, is there anything that you would have done differently over the
years as regards the rankings? Della: Would really like to have more responses from custom program participants. We always get low response rates for custom programs. We’ve tried twice, but the results were not good. We’d like to find a way to do this, to improve the validity of the custom program rankings, but we don’t know what we can do to improve the response rate.
7. If you did not have your legacy, and could start from scratch, what would the perfect ranking look like and how would it work?
Della: Much prefer to do rankings with objective data. You can question it, find errors. The subjective data works OK with EE, but poorly with degree programs, as the participants are younger, more emotional and more easily influenced by the fact that they did their degree there.
134 | P a g e
Culture also has an impact on subjective data that we have not figured out how to remove. The Germans are going to give you a 7 when they think you’re very good. The Latins will give you a 10 and the Dutch a 6. Even seasonality is an issue that we don’t know how to reduce…the Swedes don’t like the fact that we do EE in February when everyone in Scandinavia is depressed…. 8. For a manager, a perspective business school candidate, what advantages do the FT rankings offer to
him or her over the Business week rankings? The FT is every year. The FT has better criteria. The FT has a larger volume of data on every point. There are at least 3 questions behind every ranking criterion. 9. How do you think executive education offices can best (most effectively) use the rankings to improve
their programs? Della: Use the data instead of the rankings. The rankings tell you where you are compared to other schools, but the data will give you a better picture of how to improve. Look at yourself compared to other schools, but also your own data from year to year. And look across the data. Improvements or declines in one area might be having an impact on something else.
Laurent: You’ll probably learn more looking at the areas where you did not score well, rather than where you came in high.
10. What feedback do you hear from business school deans? Della: Not a lot, but we do spend one day a week on average responding to requests, meeting with business schools, speaking at events to explain the rankings. Those who come to see us want to know, inevitably, why they dropped in the rankings, etc. We can’t really tell them more than what is in the data, which they have complete access to.
It still seems that many schools do not understand the rankings. Even with all our efforts to be transparent.
Laurent: With EE you cannot calculate your ranking yourself, just from your data. But the degree rankings you can.
11. What feedback do you hear from reader-managers and/or reader-HR managers? Della: Not a lot. Can’t really recall much.
12. From your perspective, do you feel that schools with lower Exec Ed revenues are at a competitive disadvantage? (separate from rankings).
135 | P a g e
Della: Yes, they have less faculty and a less well-known brand. Fewer alumni. The reality is that a small number of business schools have large revenues and dominate everything. This applies to the MBAs as well when you look at the size of the HBS MBA program.
13. As regards the rankings, what advice do you offer to HR managers, heads of corporate universities or heads of talent development?
Della: The rankings are a starting point, not an end point.
14. As regards the rankings, what advice do you offer to readers who may be considering attending a business school program?
Della: Same. Do your homework.
15. As regards the rankings, what advice do you offer to business school deans? Della: Our criteria are good. If you run a good school and good programs you will do well, but that includes everything, including getting great participants into your programs. And listening to your participants. We were very pleased when we saw career services starting to improve at schools, as this was one of the biggest criticisms from MBA alumni who were surveyed.
16. What is next for the FT rankings? Do you foresee changes? Della: For some time we’ve been collecting and measuring entrepreneurship data in our survey, but we haven’t used it yet. Not yet sure how to use it. In the alumni survey we ask if they have started their own business, whether that is fulltime or a hobby, whether they’ve raised money, who from (VCs, friends, family, etc.). This isn’t about entrepreneurship programs. We wondered if this data might help in the normal degree program rankings. For example, we ask for salaries three years after graduation, but presumably if someone starts a company he or she will have a lower salary at that point, so this would make the school look worse in the rankings, but if that school had a higher % of people creating companies, maybe we’d have to find a way to compensate them….
17. Is there anything I should have asked? No. Very complete.
136 | P a g e
Appendix 23: Number of Schools by Region and Years in Ranking
Years in the Ranking EU US/Canada Asia LatAm Mid East Oceania Africa Total<=5 Years 6 4 4 2 1 17
1 1 1 3 52 1 13 3 1 1 1 64 1 1 25 2 1 3
6-10 Years 11 2 4 3 206 4 1 3 87 1 1 1 38 2 29 1 2 310 3 1 4
>=11 Years 16 15 1 3 2 1 3811 1 112 1 1 1 1 413 1 114 1 3 415 4 3 716 1 2 1 1 517 9 6 1 16
Grand Total 33 19 7 9 1 2 4 75% of Grand Total 44% 25% 9% 12% 1% 3% 5%
Open Programs - Number of Schools by Region and Years in Ranking (Based upon 2015 Open Ranking)
Years in the Ranking EU US/Canada Asia LatAm Africa Oceania Grand Tota<=5 Years 9 4 6 5 2 1 27
1 3 6 1 1 112 3 1 1 53 2 2 44 1 1 25 2 1 1 1 5
6-10 Years 10 3 1 3 3 206 1 17 4 1 58 1 1 1 1 1 59 1 2 310 4 2 6
11-17 Years 17 16 3 2 3811 2 1 1 413 2 214 4 415 2 2 1 516 3 5 1 917 8 6 14
Grand Total 36 23 7 11 5 3 85% Grand Total 42% 27% 8% 13% 6% 4%
Custom Programs - Number of Schools by Region and Years in Ranking (Based upon 2015 Ranking)
137 | P a g e
Bibliographic Essay
There is a fairly substantial literature on business school and university rankings exploring the implications and challenges these create. The literature takes on the fact that rankings generally are not measures of quality (1), that they generally do not use the right criteria (1) which sometimes leads schools to invest in the wrong things (1, 2), that they create comparisons of different programs (1) or attempt to draw significant differences among similar programs. (1, 3) There is a general sense in this literature that rankings have negative impacts on business schools (1), one author calling them “competitiveness in saddest manifestation” and a “wrongheaded approach we have all approved.” (4)
(1) AACSB International (2005), The Business School Rankings Dilemma, A Report from a Task Force of AACSB International’s Committee on Issues in Management Education
(2) Adler, N. J. and A.-W. Harzing (2009). "When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings." Academy of Management Learning & Education 8: 72-95.
(3) Dichev, I. D. (1999). "How Good Are Business School Rankings?" Journal of Business 72: 201-213. (4) Giacalone, R. A. (2009). "Academic Rankings in Research Institutions: A Case of Skewed Mind-Sets
and Professional Amnesia." Academy of Management Learning & Education 8(1): 122-126.
Robert Zemsky, a professor of education at Penn State, entered the ranking conversation in 1990 claiming news magazine rankings are meaningless, confusing and dangerous and that they encourage the kind of competition that puts higher education at risk. (5) He was given an opportunity to comment again in 2008, a year after the US News & World Report rankings celebrated its 25th anniversary, arguing that rankings “cannot possibly measure what they propose to measure— that immeasurable quantity: academic quality” (6) and instead are a measure of market position (8, 9), and that a better ranking system would be a measure of customer satisfaction (6). He also argued that faculty should treat them as one of a number of diagnostic tools. (6) A response by DeNisi acknowledges that rankings are “inherently imperfect” (7) and offers that the “solution” is to educate the customer. (7, 8) In this set of articles the point is also made that business school deans are complicit in the rankings process and that the overall process tends to preserve status quo by rewarding a relatively static set of performance metrics. (8)
(5) Zemsky, Robert, & Massy, William. (1990). “The Rain Man Cometh.” AGB Reports, 32(1), 21-23. (6) Zemsky, R. (2008). The Rain Man Cometh-Again. Academy of Management Perspectives, 22(1), 5-
14. (7) DeNisi, A. S. (2008). "Rain, Snow, and Sleet Are Just Different Types of Precipitation." Academy of
Management Perspectives 22(1): 15-17. (8) Glick, W. H. (2008). "Rain Man or Pied Piper? Moving Business Schools Beyond Media Rankings
with Mass Customization and Stakeholder Education." Academy of Management Perspectives 22(1): 18-22.
(9) Jain, D. C. and M. Golosinski (2009). "Sizing Up the Tyranny of the Ruler." Academy of Management Learning & Education 8(1): 99-105.
138 | P a g e
Complicity in the rankings is also a theme in a recent article published by Luca and Smith that explores how business schools disclose their rankings. (10) They point out that top schools are least likely to disclose, and mid-ranked schools are most likely. Schools also “coarsen” information to make it more favorable and, in fact 72% of schools shroud details.
(10) Luca, & Smith. (2015). “Strategic disclosure: The case of business school rankings.” Journal of Economic Behavior and Organization, 112, 17-25.
Acknowledging the challenge of rankings, others argue that because rankings drive reputation, it is important to continue “playing the game” (11)
(11) Peters, Kai. (2007). “Business school rankings: Content and context.” Journal of Management Development, 26(1), 49-53
While many of the general articles engage with several rankings, European Management Review also published a series of articles on business school rankings and focused specifically on the Financial Times rankings (primarily the MBA ranking). This began with a seminal article by Elizabeth Wedlin where she makes the point that rankings are a type of “classification system” that codifies an “organizational template” for business schools, thereby establishing recognition of “belonging together” while allowing for variation and organizational identity. (12) In the year following, also coinciding with the 25th anniversary of the US News rankings, further analysis of the Financial Times rankings was published by Devinney et al. illustrating that rankings are driven in large part by structural factors that many schools cannot change, the ranks of the top schools are quite stable over time, and the ranks of the bottom schools are quite dynamic. (13) Commentaries focused on the need to provide future students with more useful sources of information. (14) Dichev jumps into this debate making the point that the rankings hide important variations in the structural factors, including the fact that top schools are much more similar, whereas lower ranked schools have more variation. By putting these “into uniformly spaced ordinal rankings, [it is] obscuring the fact that ranking differences mean very different things for top schools and lower-ranked schools.” (15)
(12) Wedlin, L. (2007). “The role of rankings in codifying a business school template: Classifications, diffusion and mediated isomorphism in organizational fields.” European Management Review, 4(1), 24-39.
(13) Devinney, T., Dowling, G., & Perm-Ajchariyawong, N. (2008). “The Financial Times business schools ranking: What quality is this signal of quality?” European Management Review, 5(4), 195-208.
(14) Hopwood, Anthony G. (2008). “The rankings game: Reflections on Devinney, Dowling and Perm-Ajchariyawong.” European Management Review, 5(4), 209-214.
(15) Dichev, I. (2008). “Comment: The Financial Times business schools ranking: What quality is this signal of quality?” European Management Review, 5(4), 219-224.
139 | P a g e
Note that Wedlin’s dissertation on Financial Times business school rankings was published in book form in 2006, and more recently she has written about global rankings and their implications. (16, 17)
(16) Wedlin, L. (2006). Ranking Business Schools: Forming fields, identities and boundaries in international management education. Cheltenham, UK, Edward Elgar.
(17) Wedlin, L. (2011). Going global: Rankings as rhetorical devices to construct an international field of management education. Management Learning. US, Sage Publications. 42: 199-218.
Related to Wedlin’s work, Enders explores the idea that rankings do “institutional work” creating a “sub-field” of “world-class universities” and taking part in distributing symbolic capital within the field. Hazelkorn points out rankings have the greatest effect on university leaders and politicians and that they establish a framework for change that will not lead to improvement. (19) Enders argues: “Differences between universities thus become a matter of better or worse within a pre-defined space of performance, a value statement that excludes non-hierarchical alternatives. Qualities that cannot be expressed in quantities disappear, are marginalized and become de-valued. What is not countable does not count. Rankings produce what they measure: an imagined world-class university that can be calculated according to standardized norms of excellence.” A similar point is made by Rasche et al. who argue rankings help to create an “artificial order” within business schools, which defines categories of knowledge that become self-justifying. (20) These authors are less supportive of the idea that the template allows for diversity, arguing that rankings contribute to “the erosion of differentiation” and “potentially threatens to flatten out diversity among organizations.” (18)
(18) Enders, J. (2014) “The Academic Arms Race: International Rankings and Global Competition for World-Class Universities” in “The institutional development of business schools,” eds. Pettigrew, A., Cornuel, E., Hommel, U., Oxford University Press
(19) Hazelkorn, E. (2011). Rankings and the reshaping of higher education: the battle for world class excellence, Basingstoke, Palgrave Macmillan, 2011, 259
(20) Rasche, A., Hommel, U., Cornuel, E. (2014) “Discipline as Institutional Maintenance: The Case of Business School Rankings” in “The institutional development of business schools,” eds. Pettigrew, A., Cornuel, E., Hommel, U., Oxford University Press
The role rankings play in relation to identity is also explored by Elsbach and Kramer, who argue that rankings are disliked because they are a threat to organizational identity by calling into question perceptions of highly valued core identity attributes of schools, and challenging beliefs about schools' standing relative to other schools. (21) Sauder and Espeland frame rankings as a process of “surveillance and normalization.” (22) The relationship between rankings and accreditation, and the related limits this creates for schools, including threatening to make them more similar to one another, are explored by Wilson and McKiernan and Thomas (23, 24).
(21) Elsbach, K. D. and R. M. Kramer (1996). "Members' Responses to Organizational Identity Threats: Encountering and Countering the Business Week Rankings." Administrative Science Quarterly 41: 442-476.
140 | P a g e
(22) Sauder, M., Espeland, W. (2009). “The Discipline of Rankings: Tight Coupling and Organizational Change,” Americal Sociological Review, 74(1):63-82
(23) Wilson, D., McKiernan, P. (2011). “Global Mimicry: Putting Strategic Choice Back on the Business School Agenda,” British Journal of Management, 22: 457-469.
(24) Wilson, D. C. and H. Thomas (2012). "The legitimacy of the business of business schools: what's the future?" Journal of Management Development 31: 368-376.
Lest we think MBA rankings are only of interest to Deans and social scientists, operations research experts (25, 29) take them on as well. Those with a background in statistics and advanced math might choose to wander into the literature that takes a deep look at the numbers the rankings generate and that thinks about how performance metrics work, including several comparative studies of rankings. In part, these can suggest how schools might improve their rank (26). A comparison of six rankings of MBA programs concluding “there is little empirical evidence that difference in the individual rankings reflect real differences in value.” (27) Another study comparing Financial Times, Business Week and US News MBA rankings suggests that each system does have some limited validity, but they could be improved and they capture different things. (28)
(25) Chang, Allison ; Rudin, Cynthia ; Cavaretta, Michael ; Thomas, Robert ; Chou, Gloria. “How to reverse-engineer quality rankings.” Machine Learning, 2012, Vol.88(3), pp.369-398
(26) Han-Lin, L., Y. Chian-Son, et al. (2011). "Organizational Change and Management Science: Implications from Ranking and Grouping Business Schools on Spheres." International Journal of Business & Information 6: 1-34.
(27) Halperin, M., Hebert, R., & Lusk, E. (2008). “Comparing the Rankings of MBA Curricula: Do Methodologies Matter?” Journal of Business & Finance Librarianship, 14(1), 47-62.
(28) Iacobucci, D. (2013). “A Psychometric Assessment of the Businessweek, U.S. News & World Report , and Financial Times Rankings of Business Schools’ MBA Programs.” Journal of Marketing Education,35(3), 204-219.
Jessop takes on the challenges of the weighted criteria model used by the Financial Times specifically, arguing in part, “cognitive limits lead to inconsistency in preference judgements so that weights may be subject both to uncertainty and to bias. It is proposed that choosing weights to minimize discrimination between alternatives (not weights) guards against unjustified discrimination between alternatives.” (29) Köksalan et al. argue “neither the data nor the weights or the aggregation model itself is precise enough to warrant a complete ranking” and they present a model for a more flexible ranking that allows for criterion weights to vary in specific ranges. (30)
(29) Jessop, A. “A portfolio model for performance assessment: the Financial Times MBA ranking.” Journal Of The Operational Research Society, 2010 Apr, Vol.61(4), pp.632-639
(30) Köksalan, Murat, Büyükbaşaran, Tayyar, Özpeynirci, Özgür, & Wallenius, Jyrki. (2010). “A flexible approach to ranking with an application to MBA Programs.” European Journal of Operational Research,201(2), 470-476.
141 | P a g e
Proudlove claims to have “cracked” the rankings, and shows how the FT methodology can be used to reconstruct the underlying calculations to a great degree. (31)
(31) Proudlove, N. (2012). “Cracking the rankings Part (i): Understanding the Financial Times MBA rankings.” OR Insight, 25(4), 221-240.
And, an analysis of six ranking systems from a social science perspective suggest that current ranking systems do not follow best practices and rather “recycle reputation.” (32)
(32) Marginson, S. (2014). “University Rankings and Social Science.” European Journal of Education, 49(1), 45-59.
There is a much smaller but still worthwhile literature from the perspective of the rankings where critiques are addressed. These tend to acknowledge that rankings are not objective (34) and are not measures of quality, and yet, at the same time, play a crucial role in educating consumers of business school education and for that reason can be a positive force for change (33). This literature has also pointed out that in spite of criticizing rankings, business schools are also active in using and disseminating them (34). One interesting Business Week article cautions MBA applicants on using the rankings, warning reliance on rankings can “ruin a business school application.”(35)
(33) Baty, P. (2011). Say it loud: I’m a ranker and I’m proud. Times Higher Education, March 11. http://www.timeshighereducation.co.uk/story.asp?storycode=415470
(34) Bradshaw, Della. (2007). “Business school rankings: The love-hate relationship.” Journal of Management Development, 26(1), 54-60.
(35) Meglio, F. D. (2011). "Making the Most of Business School Rankings." BusinessWeek.com: 4-4.
A few of the articles that discuss the Financial Times ranking touch on executive education rankings specifically (5, 23), but the majority of the articles and books cited here focus on the MBA rankings. That said, there are many lessons and perspectives that are also applicable to the executive education ranking. There are, however, also some relevant differences. First, for the most part, executive education is a secondary or tertiary activity, not a primary one and, as such, has less centrality in “institutional identity.” Also, as noted in this report, executive education is an interesting case because satisfaction surveys are so prevalent in the industry and are broadly used for program improvement.
The FT’s editor in charge of business education, Della Bradshaw, has been quite active in speaking about FT’s rankings in the press, primarily about the MBA rankings. She has gone on the record recognizing that the differences between schools can be small (36) and urging people not to take rankings “too seriously” (37). She has also addressed some of the nuts and bolts behind how the rankings work and the FT’s international focus (38).
(36) Bickerstaffe, George, “Top Rank: Della Bradshaw Interview,” European Foundation for Management Development’s Global Focus, Volume 3, 2009.
142 | P a g e
(37) Editors, “Interview with Della Bradshaw, Editor of the Financial Times Business School Rankings,” Selections, The Magazine of the Graduate Management Admission Council, Fall 2001.
(38) Ouezman, Stéphanie, X, “Financial Times rankings, how do they work? Della Bradshaw answers,” Espace Prépas, Sep 15, 2014
UNICON has published a number of reports that are relevant to these debates in that they provide a deeper context for understanding business school executive education programs and customers. Specifically, there are a number of reports in recent years that have looked deeply into what customers are looking for, especially custom clients of executive education. These have a great deal of material that could be used to revise criteria and approaches to ranking.
(39) Eiter, Marie. “Investigating our Custom Clients’ Evolving Needs,” UNICON Research Study published June 2009
(40) Eiter, Marie and Robert Halperin. “Investigating Emerging Practices in Executive Program Evaluation.” UNICON Research Study published October 2010.
(41) Lloyd, Frank R. and David Newkirk. “Strategies and Choices: University-Based Executive Education Markets and Trends.” UNICON Report dated August 15, 2011. (Also see Lloyd, Frank, and David Perryman. “The Case for University-Based Executive Education: A white paper surveying current research.” SMU Cox Executive Education, Fall 2006)
(42) Maybar-Plaxe, K., Allen, M., Renaud-Coulon, A. (2014). “Minding Their Business By Flexing Our Minds: A Guide To Corporate University Partnerships.” UNICON research publication.
(43) Pulcrano, J. (2013). “Serving the Future Needs of Business in Management Development and Retention.” Research sponsored by AACSB, EMBAC and UNICON. http://uniconexed.org/2013/research/AACSB_EMBAC_UNICON_October_2013_Pulcrano.pdf
(44) Voller, Shirine and Sue Honoré (Ashridge), “Innovation in Executive Development: A case-based study of practice in international business schools.” UNICON report published March 2008.
(45) Weldon, Elizabeth (2014). Designing And Delivering Customized Programs with Impact, UNICON Research Report, Fall 2014
(46) Eiter, Pulcrano, Stine, Woll, “Same Solar System, Different Orbits: Opportunities and Challenges in Executive Education and Corporate University Partnerships,” UNICON Research Report, Fall 2014.
143 | P a g e
About the Authors
Tom Cavers is a market research and strategy consultant. His work in executive education includes in-depth market surveys into MOOCs, open enrollment programs, custom programs, executive coaching, and executive search and assessments. As a “classically trained” marketer who works regularly with academia, Tom focuses on driving impact by translating research findings into clear and tangible action. In addition to his work in executive education, Tom is also a partner with Green Flash Consulting, a boutique market research and strategy firm focused on consumer products and services. Prior to joining Green Flash, Tom held marketing leadership roles with Clorox, Del Monte, Hershey and Safeway. He holds an MBA from U.C. Berkeley and graduated from Occidental College with Distinction in Philosophy.
Jim Pulcrano is an independent consultant, teacher and business coach. His current work includes teaching in Lausanne, Moscow, Grenoble, Silicon Valley and Boston, and various strategy, networking, leadership, customer-centricity and innovation mandates with multinationals in Europe and the MiddleEast.
Jim joined the management team of IMD in 1993 as Director of Marketing. During a 20-year career at IMD, Jim had direct responsibility for IMD’s clients across a wide part of the world, was a member of the Executive MBA teaching team (focused on entrepreneurship and IMD’s work in Silicon Valley), ran IMD’s annual startup competition, and had responsibility for IMD’s worldwide alumni network. Jim stepped down from IMD in August 2013 as Executive Director. He is the former Chairman of UNICON. Before joining IMD, Jim was Managing Director of a Swiss startup in the medical device industry. Previous to his MBA degree, he worked as an engineer in the USA and Africa for Schlumberger, a leader in the petroleum services industry. He has contributed substantially to six startup companies over the past 25 years in the US and Switzerland. The Swiss federal government chose him in 2003 to join its strategy committee for the “Swiss Houses” in the United States, Singapore and Shanghai, under the Swiss Secretariat for Education & Research. He is also a jury member for Switzerland’s VentureLab entrepreneurship program and serves on the Board of AISTS, the International Academy of Sports Science & Technology. He completed his doctorate at the Grenoble Ecole de Management in 2012 focused on entrepreneurship and networking. He received his MBA from IMD in 1984 and he has a BSc in Mechanical Engineering from the U. of Missouri-Columbia.
Jennifer Stine is an independent consultant, teacher, and innovator. She is an expert in the development of world-class executive and professional programs, with over a decade of leadership experience at Harvard and MIT. Her current focus is developing university-corporate partnerships that lead to innovative, co-created educational experiences. She also conducts research in executive and professional education, and she is an instructor at Harvard Extension School where she teaches organizational behavior, leadership, and teamwork.
144 | P a g e
From 2008 to 2012 she served as Managing Director of Professional Education at Harvard University’s Graduate School of Education, where she was responsible for professional and executive education programs at the school, including K-12 Institutes, The Principals’ Center, Harvard Institutes for Higher Education, and WIDE World online courses.
From 2002 to 2008, Stine led executive and professional education for the School of Engineering at MIT. Her accomplishments at MIT included significant expansion of a portfolio of custom and open enrollment certificate programs including the award-winning BP Projects Academy, developed jointly with the MIT Sloan School of Management, the global Accenture Solutions Delivery Academy (now the Accenture Technology Academy) certification program, and the MIT Career Reengineering program.
Prior to joining MIT in 2002, she worked as an entrepreneur in technology consulting and served as documentation team lead and training co-lead on a major Oracle implementation at Caltech. Stine holds three degrees from Stanford University, including a Ph.D. in history.
The authors can be reached at:
Tom Cavers: [email protected]
Jim Pulcrano: [email protected]
Jenny Stine: [email protected]
145 | P a g e
About UNICON
UNICON is a global consortium of 110 business-school-based executive education organizations. Its community of member organizations is engaged in accelerating the development of leaders and managers, thereby enhancing performance in public and private organizations globally through executive development initiatives. UNICON’s primary activities include conferences, research, benchmarking, sharing of best practices, staff development, recruitment/job postings, information-sharing, and extensive networking among members, all centered on the business and practice of executive education. To learn more, please visit http://uniconexed.org/.
To see a list of UNICON schools, please see Appendix 2: UNICON Member Schools and Their 2015 Open & Custom Program Rankings.
This report was sponsored by UNICON; the materials here are the opinions of the authors only.