evaluation of user experience across all ryanair digital platforms

Ryanair Digital Evaluation 1

Project Title: Evaluation of User Experience across all Ryanair Digital Platforms

Presented by: Shane Hickey

Course: Certificate in Digital Marketing

Module: Technology and User Experience

Lecturer: Dermot Bradfield

Date: 10/12/2014


Table of Contents

1.0) Executive Summary

2.0) Ryanair

2.1) Digital History

2.2) Ryanair Digital Labs

2.3) Website Redesign

2.4) App Relaunch

3.0) Method of Evaluation

3.1) Nielsen’s Heuristics

3.2) Hypothesis

3.3) Scenario

3.4) Persona

3.5) Test Methods

4.0) Test Results

4.1) Summary of Desktop Evaluation

4.2) Summary of Mobile Evaluation

4.3) Summary of App Evaluation

4.4) Conclusion

4.5) Criticisms of the Report

Bibliography

Appendix A Desktop Site Evaluation 1

Appendix B Desktop Site Evaluation 2

Appendix C Desktop Site Evaluation 3

Appendix D Mobile Site Evaluation 1

Appendix E Mobile Site Evaluation 2

Appendix F Mobile Site Evaluation 3

Appendix G App Evaluation 1

Appendix H App Evaluation 2

Appendix I App Evaluation 3


1.0) Executive Summary

Ryanair is one of the world’s largest and most successful airlines. They have enjoyed huge growth

through a low cost model based largely on the Southwest Airlines approach in the US. Although

offering customers fantastic value in terms of pricing they were viewed poorly in terms of customer

service both online and offline.

It is fascinating in many ways how the company has experienced phenomenal year on year growth

based largely on its price point while at the same time being one of the most controversial and

arguably hated brands of all time.

Their most recent strategy has been to improve customer experience and a large part of this has

been to improve the digital experience of their customer. This has resulted in a number of initiatives

including a website redesign and also a new and improved App that allowed airline ticket purchase

and downloading of boarding passes onto smartphones.

All of these initiatives have been developed at the new Ryanair Digital Labs which is striving to

provide a world class online travel platform.

The purpose of the report is to therefore assess the quality of the user experience on the new

Ryanair website and also to determine whether the quality of this user experience is replicated

across the new mobile site and App.


2.0) Ryanair

Ryanair was established in 1985 and using a low cost model has grown to become Europe’s favourite

airline (Ryanair, 2014). Michael O’Leary, the Ryanair CEO, has himself admitted to replicating the

business model of Southwest airlines in the US, the low cost carrier, in Europe (CNBC, 2013).

This low cost model coupled with the deregulation in the Aviation industry legislation under the

“Open Skies” policy have been the two greatest contributors to Ryanair’s success (RTE, 2014).

In 2013 Ryanair was both the largest European airline by scheduled passengers carried (Huffington

Post, 2013). It also came fifth in terms of the busiest International airline by passenger numbers,

carrying over 80 million passengers last year. (Wikipedia, 2014).

According to its own website they operate 1,600 flights daily from 70 bases connecting 183

destinations in 30 countries and operating a fleet of more than 300 Boeing aircraft (Ryanair, 2014). It

is also expected that the airline will carry over 3.5 million customers over the 2014 Christmas period

(Independent, 2014).

Even with all its success the company has endured a volatile relationship with its customers. The CEO

of Ryanair, Michael O’Leary, has made some infamous quotes down through the years that reflected

the company’s attitude to its customers and they included:

“People say the customer is always right but you know what - they’re not. Sometimes they

are wrong and they need to be told so” (Telegraph, 2012)


2.1) Digital History

According the Ryanair corporate website the consumer site, www.ryanair.com, was launched in

2000 and within the first three months the site was taking over 50,000 bookings a week (Corporate

Ryanair, 2014). Although thousands of customers choose to fly with Ryanair given its pricing

structure the Ryanair website has come under heavy criticism for its poor user experience and gaudy

appearance (Harbison, 2012).

One of the major criticisms of the site had been the number of click through’s which a customer

needed to make before they eventually purchased their ticket, through the old website this had

been measured at 15 clicks (The Marketing Institute, 2014). Customers were offered car hire, hotels

and travel insurance which was very much in keeping with the Ryanair mantra of trying to sell to you

at every opportunity.

http://www.ryanair.com/


Another infuriating feature of the site was the Captcha which asked users to rewrite text that

appeared in a box as a form of a security question. It often had very difficult to read text or had

obscure messages like “I fancy you” (Barry, 2013).

There have also been numerous accusations of Ryanair advertising rates at one price and then

having additional charges on the website resulting in customers booking fares which far exceeded

the price they thought they would pay. This resulted in Ryanair having continuous difficulties with

Advertising authorities and consumer protection associations (BBC, 2010).


Ryanair were also guilty of using UX dark patterns according to a blog called the Digital Tonic (Digital

Tonic, 2014). Essentially dark patters are where users are tricked into booking additional features or

signing up to something to which they have no knowledge of.

This is particularly evident on the Ryanair website where the default setting is that you purchase

travel insurance and you have to click on a drop down menu to opt out of this purchase. Also the

“please don’t insure” wasn’t listed as a first option or placed in alphabetical meaning the user has to

scroll down to find this option.

2.2) Ryanair Digital Labs

In order for Ryanair to redesign their digital presence and improve user experience they were going

to have to outsource this or hire an internal technology team to assist with this project.

Ryanair Digital Labs was launched with the intention of creating a start of the art digital and

innovation hub in Dublin and according to their spokesperson Robin Kiely “Ryanair changed air travel

in Europe forever – now we want to change the world of online travel and were going to do that

through Ryanair Labs” (Kiely, 2014).


2.3) Website Redesign

Ryanair launched its new look website in April 2014 with a host of new features and a promise of

improved customer experience. There were a number of hitches in the launch of the new Ryanair

website though with a botched rename meaning that the website disappeared from many Google

flight searches albeit the company argued that the brand is so strong that customers search under

the company as oppose to searching for flights (Topham, 2014).

According to an article published in the Telegraph the main features of the new website are:

Airline ticket purchase has been reduced to five clicks.

Removal of many products which were previously offered.

‘My Ryanair’ feature where users can store their personal information.

Eradication of flash banners and improvement on design and colour scheme.

Passengers will be encouraged to share information on social media.

Improved mobile site.

(Paris, 2013)


2.4) App Relaunch

The original Ryanair mobile app was launched in March 2012 but was only officially unveiled about

six months later and carried a cost of €3 to download it (May, 2012).

The latest update to the App was in July 2014 and is now available for free in both iPhone and

Android formats through the iTunes App store and the Google Play Store (Technology.ie, 2014).

There are a number of new features with the new App including additional language versions such as

Dutch, French, Polish and German and it also allows for IOS Passbook integration (Financial Mirror,

2014).


3.0) Methods of Evaluation

There are a number of methods available when looking to determine user experience. The options

can be listed as follows (Wikipedia, 2014):

Implicit UX Methods (focus on what the users do not express verbally, examples include eye

tracking and electroencephalography).

Explicit UX Methods (explore what the users are consciously aware of, examples include

expression and verbal self-reporting).

Creative UX Methods (where users are allowed to co create with designers examples include

creativity workshops)

Longitudinal UX (where the user experience is measured over a period for example diary

methods)

Implicit methods normally involve having appropriate software or hardware at your disposal, to

track heart rate for example, and were therefore ruled out of this evaluation. Explicit methods often

focus on emotion evaluation and creative workshops would not be possible as the users will not sit

down with the engineers and designers of the Ryanair website.

It was therefore decided to use longitudinal user experience evaluation to test user’s reactions to

the platforms over a prolonged period, in this project, over a twenty minute period.

In terms of the actual method of evaluation to be used it was decided to use Heuristic Evaluation as

the most appropriate form of measuring user experience across the three Ryanair platforms. The

Heuristic evaluation methodology to be used is that of Nielsen’s Heuristics.

The main reasons for choosing Nielsen’s Heuristics are similar to those outlined by Macefield in his

article “An overview of expert heuristic evaluations” (Macefield, 2014)

Cheap and easy to perform.

Takes advantage of evaluations from experts in their field, which the users are, as Digital

marketing students.

Heuristics are used to move away from opinion and more toward measurement.

Alternatives to Nielsen’s Heuristics that could have been used were Gerhard Powell’s cognitive

engineering principles and the Weinshenk and Barker classification model (Wikipedia, 2014).

Heuristic Evaluation (Nielsen and Molich, 1990; Nielsen 1994) can be defined as:

“A usability engineering method for finding the usability problems in a user interface design process.

Heuristic evaluation involves having a small set of evaluators examine the interface and judge its

compliance with recognised usability principles (the “heuristics”) (Nielsen, 1995).


3.1) Nielsen’s Heuristic’s

The following Heuristics will be tested:

H1: Aesthetic and minimal design

H2: Match between system and the real world

H3: Recognition rather than recall

H4: Consistency and standards

H5: Visibility of system status

H6: User control and freedom

H7: Flexibility and efficiency of use

H8: Help users recognise, diagnose and recover from errors

H9: Error prevention

H10: Help and Documentation

A rating scale of between 0 and 4 will be used to measure each heuristic.

In this particular case nine evaluators will be used so there will be a minimum of three evaluators

per platform. Each platform will then be tested individually by three evaluators. Each evaluator will

be asked to score each platform and add written context where appropriate.

Essentially the evaluators will be asked to comment on the heuristic rating so as to provide some

quantitative research to add to the heuristic scoring of the user experience.

The evaluations will take place unaccompanied and the suggested time each evaluator spends on

this is between fifteen and twenty minutes.


3.2) Hypothesis

The aim of the report was to essentially the hypothesis that the quality of user experience is

replicated across the three digital platforms currently being used by Ryanair which are:

Desktop site

Mobile site

App

As mentioned earlier the company made a huge investment into its digital presence in order to

mainly benefit customer experience and of course ultimately to increase flight bookings. Given the

very poor website experience that customers traditionally associated with Ryanair the first task of

this report was to evaluate the user experience of the new Ryanair website, mobile site and App.

Secondly we can then assess whether the user experience score is replicated across all three

platforms or if there is a deviation in any of them.

This will be achieved by asking users to do three things on each platform:

1) To book a flight through the assigned platform as a guest.

2) To email the company with thoughts on the newly designed platform in question, through their

customer services section.

3) To explore the platform and assess the user experience.

Please see below a screenshot from the App where the evaluator will choose to continue without

registering.


3.3) Scenarios

Scenario 1

A number of decisions were taken when formulating the scenario to test individual elements of the

booking experience. They included booking a return trip that spanned across two months and did

not include the month that we are currently in. This would demonstrate how easy or difficult it

would be to use the Calendar. It would also indicate how comfortable the user experience is in terms

of adjusting the default settings which are normally given based on time and location.

It was also important to give the scenario a specific arrival and departure airport as this meant the

same booking was being tested across all the platforms and again the locations given were those

that would not include the airport given in the default settings. Essentially the departure flight was

to be from Malaga and not from Dublin which the Ryanair site being accessed by an Irish IP address

would default to.

To book a return ticket from Malaga to Dublin for one adult and one child departing on December

23rd and returning on January 7th with no checked baggage or any additional bookings such as car

hire or hotels. The user is to go through the whole experience up until the point where they have to

click on the final purchase icon after submitting all personal and payment information.

Scenario 2

Users were originally going to be asked to create a ‘My Ryanair’ profile and then to rebook the same

flight details that were given in scenario 1 but credit card information needed to be added which

was outside the scope of this project so instead users will be asked to locate the Ryanair customer

services details and email the company. The contact us form on the App can be seen below.

Scenario 3

Users will also be given a few minutes freedom to explore the site to assess the UX across the site.


3.4) Persona

The Personas to be used will be participants in the 2014/2015 Digital Marketing certificate course at

the National College of Ireland. The reason for this is that they are considered experts in this field.

Although we are using experts it is recognised that experts don’t necessarily see usability problems

which non experts might see (Macefield, 2014). There will be no distinction made between the sexes

of the evaluators as it is felt this will have no bearing on the results. Nine evaluators will be used in

total.

3.5) Test methods

The evaluators will be asked to go through the system twice, the first pass to test the general flow

and the second pass will allow evaluators to test specific interface elements (Nielsen, 1995).

In the case of each platform the user will need access to the following technology:

Desktop site (PC with internet access)

Mobile site (Smartphone with internet access)

App (Smartphone with internet access)

In terms of the mobile and App site, the mobile device that the user is accessing these sites with

should be recorded as this may have a bearing on the results. The App will is free but users will need

to download it from iTunes or the Google Play store depending on the operating system on their

phone.

The evaluators will be asked to allow 20 minutes for their evaluation.

Please see below the landing page of the mobile site where the App evaluators can download the

App on their smartphone.


4.0) Test Results

The results of the test will be broken into each broken into a summary on each individual platform.

4.1) Summary of Desktop Evaluation

The results from the evaluators regarding the desktop site were very positive from a Ryanair perspective. The total score on the Heuristics was 18. The three evaluators had an average score of 6 per evaluator which is very low and indicates a very satisfactory user experience. Only one heuristic scored over 2 which was from Evaluator B who gave H8, help users recognise, diagnose and recover from errors a 3. Contrary to the historic site the new site is easy to navigate, has a very attractive design and colour scheme and offers a very positive and enhance user experience. In particular all of the issues outlined in the beginning of the report have been addressed. The following have been either improved or removed:

Improved flight booking process (reduced number of clicks).

Improved navigation.

Improved design and colour scheme.

Removal of captcha’s.

Price transparency.

Ancillary offers are presented better. There are still some issues though with the Ryanair desktop site and that was highlighted by the evaluators. For example, you still have to opt out of the travel insurance which is listed quite far down on the drop down menu. Also the live chat facility was not working which some who have had previous experience of Ryanair customer service was unsurprising. The offers of the car hire and hotels are still present through the booking process. Suggestions from the evaluators were that they should come after you conclude your ticket purchase and not before to improve the user experience further or be a separate tab. It is obvious to see the work of the Ryanair Digital Labs team in terms of the user experience on the desktop site and have been key to digitally transforming a brand and website that was actually hated by many even if they continued to use Ryanair being motivated purely by the price point.


4.2) Summary of Mobile Evaluation

The mobile site did not receive the glowing review that desktop site received although the mobile site is in fact an exact replica of the desktop site which is in itself a criticism. Ryanair should really have a mobile optimised site. The results from the evaluators regarding the desktop site were very positive from a Ryanair perspective. The total score on the Heuristics was 30. The three evaluators had an average score of 10 per evaluator which indicates a reasonable user experience but highlights some UX issues. In terms of why it did not receive a glowing review the evaluator’s heuristics scores and comments tended to focus on the load speeds of the pages as well as the responsiveness. Areas that were highlighted included the Calendar and the Route Map. Positives though much like the desktop site included the wealth of information that was available to the user. Heuristics that need to be examined as they scored quite poorly in the evaluation included H3, recognition rather than recall, and H6, user control and freedom and H10, help and documentation. H10 received a score of 4 from Evaluator E indicating a serious user experience problem. Areas of the site which received praise included the flight booking process and customer service options in particular. The Youtube video on the “Online Checkin Guide” was highlighted in particular by one of the evaluators. 4.3) Summary of App Evaluation

One important feature of the App was that first of all it was free and not charged for as it was historically a €3 download which was a throwback to Ryanair’s policy of charging customers at every opportunity. The results from the evaluators regarding the desktop site were very positive from a Ryanair perspective. The total score on the Heuristics was 30 from the three evaluators. The three evaluators had an average score of 10 per evaluator which is a reasonable score but does indicate some user experience issues. The App only had a reasonable score from the evaluators which was slightly disappointing given the investment made in the Digital Labs and the resources available to Ryanair. The heuristic which scored most poorly was H5, visibility of the system status and this referred to at times the frustrating lack of responsiveness which at times gave no information on the status of the request or action. The drop down menus were also highlighted as having poor load speed. In terms of interesting features on the App the evaluators liked the flying aeroplane loading icon, the colour scheme and the ease of navigation which all added to the user experience. The Calendar was also highlighted as being easy to use and this feedback was very interesting given that part of the scenario given to the evaluator was to book flights in different months to test the Calendar.


4.4) Conclusion

Ryanair has transformed the digital experience of user particularly on the desktop site. The mobile

site and the App also scored quite highly in terms of user experience but areas have been

highlighted through the report that are in need of improvement so the user experience has not been

replicated yet across all the platforms.

Ryanair has embarked on the long journey of transforming the online experience for its customers

and through the work of their Digital Labs team can now set the goal of offering the best user

experience as well as the cheapest flights which would be a very strong business model.

4.5) Criticisms of the Report

One criticism that could be levelled at the report is that Ryanair as a major global brand and a

dominant player in the world aviation industry would obviously offer an excellent user experience

through its various platforms.

This is a debatable point though and was certainly not true up until the launch of the new website.

The historic site was certainly an effective website and one supported the Ryanair sales strategy but

there is no doubting that it offered an unbelievably poor record in terms of user experience. Given

this I think it was fair to choose Ryanair as a company and to test their user experience and

particularly across all platforms.

We can see from our findings that the new desktop site is excellent in terms of the user experience

offered but the mobile site and App still have a lot of improvement so it is still surprising to see

global brands launching digital products that have not undergone a rigorous testing procedure.

A second criticism of this report is in terms of the evaluations and the scenario producing real

findings. Unfortunately given the resources at hand it was not possible to ask evaluators to go

through the whole booking process as payment information would have had to be included to

complete the process.

This caused difficulties in the sense that the boarding passes could not be downloaded onto the

particular devices and the “My Ryanair” profiles could not be filled in as they require payment

information. It would have been interesting to test how the booking process was affected using “My

Ryanair” as this function works as an autofill. Also to have seen the soft copy of the downloaded

boarding passes would have been very interesting.

Lastly in terms of the desktop site they were really not that many negative findings as many of the

initial bugs and difficulties that had been there when the site launched had been since eradicated so

the report did not produce many errors on that particular platform.


Bibliography

1) Ryanair (2014) ‘History of Ryanair’ [Online] Available from: http://corporate.ryanair.com/about-

us/history-of-ryanair/ [Accessed 24th November 2014].

2) CNBC (2013) ‘Ryanair CEO: We’re Europe’s Southwest airlines’ [Online] Available from:

http://www.cnbc.com/id/100839583# [Accessed 24th November 2014].

3) RTE (2014) ‘Ryanair to focus on core European market for growth – commercial chief’ [Online]

Available from: http://www.rte.ie/news/business/2014/1017/653097-ryanair-growth-europe/

[Accessed 24th November 2014].

4) Boscamp E. (2013) ‘the world’s busiest airlines’ [Online] Huffington Post, June 18th, Available from:

http://www.huffingtonpost.com/2013/06/18/busiest-airlines-world_n_3460451.html [Accessed 24th

November 2014].

5) Wikipedia (2014) ‘World’s largest airlines’ [Online] Available from:

http://en.wikipedia.org/wiki/World%27s_largest_airlines [Accessed 24th November 2014].

6) Ryanair (2014) ‘Welcome to Ryanair’ [Online] Available from: http://www.ryanair.com/ie/about/


7) O’Conghaile P. (2014) ‘Ryanair to carry 3.5 million passengers over Christmas’ [Online] Irish

Independent, 20th November, Available from: http://www.independent.ie/life/travel/travel-

news/ryanair-to-carry-35-million-customers-over-christmas-30760661.html [Accessed 24th

November 2014].

8) Telegraph (2012) ‘Michael O’Leary’s most memorable quotes’ [Online] Telegraph, September 5th,

Available from: http://www.telegraph.co.uk/travel/travelnews/9522319/Michael-Olearys-most-

memorable-quotes.html [Accessed 24th November 2014].

9) Corporate Ryanair (2014) ‘History of Ryanair’ Available from: http://corporate.ryanair.com/about-

us/history-of-ryanair/ [Accessed 24th November 2014].

10) Harbison N. (2012) ‘Imagine what Ryanair could do online if it really cared about user experience

and design’ [Online] Tnooz, October 1st, Available from: http://www.tnooz.com/article/imagine-

what-ryanair-could-do-online-if-it-really-cared-about-user-experience-and-design/ [Accessed 24th

November 2014].

13) The Marketing Institute (2014) ‘Has customer experience soared at Ryanair?’ [Online] Available

from: https://marketinginstituteofireland.wordpress.com/2014/08/18/has-customer-experience-

soared-at-ryanair/ [Accessed 24th November 2014].

14) Barry A. (2013) ‘No more captchas for individual bookings on Ryanair.com’ The Journal,

September 20th Available from: http://businessetc.thejournal.ie/ryanair-website-changes-1092189-

Sep2013/ [Accessed 24th November 2014].

15) BBC (2010) ‘Ryanair reprimanded for misleading advertising’ [Online] Available from:

http://www.bbc.com/news/10626652 [Accessed 24th November 24th].

http://corporate/

http://www/

http://www/

http://www.huffingtonpost.com/2013/06/18/busiest-airlines-world_n_3460451.html

http://en/

http://www/

http://www.independent.ie/life/travel/travel-news/ryanair-to-carry-35-million-customers-over-christmas-30760661.html

http://www.independent.ie/life/travel/travel-news/ryanair-to-carry-35-million-customers-over-christmas-30760661.html

http://www.telegraph.co.uk/travel/travelnews/9522319/Michael-OLearys-most-memorable-quotes.html

http://www.telegraph.co.uk/travel/travelnews/9522319/Michael-OLearys-most-memorable-quotes.html

http://corporate.ryanair.com/about-us/history-of-ryanair/

http://corporate.ryanair.com/about-us/history-of-ryanair/

http://www/

https://marketinginstituteofireland.wordpress.com/2014/08/18/has-customer-experience-soared-at-ryanair/

https://marketinginstituteofireland.wordpress.com/2014/08/18/has-customer-experience-soared-at-ryanair/

http://businessetc.thejournal.ie/ryanair-website-changes-1092189-Sep2013/

http://businessetc.thejournal.ie/ryanair-website-changes-1092189-Sep2013/

http://www.bbc.com/news/10626652


16) Digital Tonic (2014) ‘Ryanair’s new website – still hiding some mean tricks with dark patterns’

Available from: http://www.digital-tonic.co.uk/digital-tonic-blog/ryanairs-new-website-still-hiding-

mean-tricks-dark-patterns/ [Accessed 24th November 2014].

17) Kiely R. (2014) ‘Ryanair launches digital lab to develop travel platform’ [Online] Breaking News

Available from: http://www.breakingnews.ie/business/19ikiped-launches-digital-lab-to-develop-

travel-platform-635608.html [Accessed 24th November 2014].

18) Topham G. (2014) ‘Ryanair drops out of top Google flight search results after website overhaul’

[Online] The Guardian, 17th April, Available from:

http://www.theguardian.com/business/2014/apr/17/ryanair-website-drops-out-top-google-flight-

search-results [Accessed 24th November 2014].

19) Paris (2013) ‘Ryanair unveils simpler website’ [Online] Telegraph November 19th, Available from:

http://www.telegraph.co.uk/travel/travelnews/10460167/Ryanair-unveils-simpler-website.html


20) May K. (2012) ‘Ryanair officially unveils first mobile app, six months after first launching it’

[Online] Tnooz, September 13th, Available from: http://www.tnooz.com/article/19ikiped-officially-

unveils-first-mobile-app-six-months-after-first-launching-it/ [Accessed 24th November 2014].

21) Technology.ie (2014) ‘Ryanair’s digital boarding pass app gets a mixed welcome’ [Online]

Available from: http://technology.ie/ryanairs-digital-boarding-pass-app-gets-mixed-welcome/


22) Financial Mirror (2014) ‘Travel: Ryanair releases latest mobile app update’ [Online] Available

from: http://www.financialmirror.com/news-details.php?nid=33439 [Accessed 24th November

2014].

23) Wikipedia (2014) ‘User Experience Evaluation’ Online Available from:

http://en.wikipedia.org/wiki/User_experience_evaluation [Accessed 24th November 2014].

24) Macefield R. (2014) ‘An overview of expert heuristic evaluations’ [Online] Available from

http://www.uxmatters.com/mt/archives/2014/06/an-overview-of-expert-heuristic-evaluations.php


24) Rohrer C. (2014) ‘When to use which user experience research methods’ Available from:

http://www.nngroup.com/articles/which-ux-research-methods/ [Accessed 24th November 2014].

25) Wikipedia (2014) ‘Heuristic Evaluation’ [Online] Available from:

http://en.wikipedia.org/wiki/Heuristic_evaluation [Accessed 24th November 2014].

26) Nielsen (1995) ‘How to conduct a Heuristic Evaluation’ [Online] Available from:

http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/ [Accessed 24th November

2014].

27) Nielsen (1995) ‘How to conduct a Heuristic Evaluation’ [Online] Available from:

http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/ [Accessed 24th November

2014].

http://www/

http://www/

http://www.theguardian.com/business/2014/apr/17/ryanair-website-drops-out-top-google-flight-search-results

http://www.theguardian.com/business/2014/apr/17/ryanair-website-drops-out-top-google-flight-search-results

http://www/

http://www/

http://technology.ie/ryanairs-digital-boarding-pass-app-gets-mixed-welcome/

http://www/

http://en/

http://www.uxmatters.com/mt/archives/2014/06/an-overview-of-expert-heuristic-evaluations.php

http://www.nngroup.com/articles/which-ux-research-methods/

http://en.wikipedia.org/wiki/Heuristic_evaluation

http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/


Appendix A

Platform being tested (Desktop, Mobile or App): Desktop

Device being used (Make and Model) to access the platform: Toshiba Laptop

Can you please do the following:

1) To book a return ticket from Malaga to Dublin for one adult and one child departing on

December 23rd and returning on January 7th with no checked baggage or any additional

bookings such as car hire or hotels. Go through the whole experience up until the point

where you have to click on the final purchase icon after submitting all personal and payment

information.

2) Email customer service in Ryanair with your thoughts on their newly designed platform.

3) Spend a few minutes exploring the site and assess the user experience.

Can you give each heuristic a score between 0 and 4 in the box below?

In the case where you have scored a heuristic less than three can you explain why in comments?

Also can you give some general feedback and any recommendations in the comments box?

Thank you kindly for your participation.

H1 Aesthetic and minimal design 0

H2 Match between system and the real world 0

H3 Recognition rather than recall 0

H4 Consistency and standards 0

H5 Visibility of system status 0

H6 User control and freedom 0

H7 Flexibility and efficiency of use 1

H8 Help users recognise, diagnose and recover from errors 0

H9 Error prevention 0

H10 Help and Documentation 2

Total 3

Comments:

Ryanair have made a very impressive website giving an excellent UX. It could not be more

different in many ways from the old website.

Some criticisms though are the Live Chat facility had no agents available and the travel

insurance, car hire and hotel offers still need to be removed from the flight booking

process.

Improved features include simplified pricing, the removal of the captcha’s, a quicker

booking process and an appealing site design and colour scheme.

The Route Map is also much improved as it used to be painful.


Appendix B


Device being used (Make and Model) to access the platform:






information.







H1 Aesthetic and minimal design 2 Comments: H1; gave a 2 as it’s very busy and not immediately obvious where to go. However it’s not a 4 as it’s not impossible or even difficult just a level of effort above that which most businesses are asking of their customers ( supermarkets etc and their continuous attempts to “guide” the customer with as little thought from the customer as possible ).

H2 Match between system and the real world

0 H2; simple language – all calls to action/buttons. Logical order.

H3 Recognition rather than recall 0 H3; the continuous forward motion of the system is logical and seems easy to navigate

H4 Consistency and standards 0 H4; consistent look and feel through the booking process, and importantly, from the home page to the first conversion page – there’s no disconcerting change to the environment

H5 Visibility of system status 1 H5; not confusing but with so much information to elicit from customers and so many add-on options It takes focus and effort to complete. On the technical side it passes well – all options are covered with explanation/escape options.


H6 User control and freedom 1 Back arrows and forward ARROWS CAN CAUSE SERVER ERROR, THOUGH IT’S EASY TO EXIT THAT. Returning to start can lose information put in already.

H7 Flexibility and efficiency of use 0 MYRYANAIR option available to speed up future interactions

H8 Help users recognise, diagnose and recover from errors

3 Server error not explained – though this is an aesthetic problem rather than a functional one, as no action on the part of the customer is required.

H9 Error prevention 0 Very robust as I bashed around the site. No confirmation dialogue boxes on booking.

H10 Help and Documentation 0 Useful info one of three top buttons spanning site. That leads to comprehensive faq section and searchable on keywords. Well-spaced with images to guide which way to navigate.

Total 7


Appendix C


Device being used (Make and Model) to access the platform:






information.
















Total 8

Comments:

I scored quite highly as the user experience was very straightforward, did not take up much

time to book flights in comparison to their previous website.

One area that is misleading is the Insurance – and the way it is in a list ‘Don’t insure me’ –

it is quite hidden.

Also there are too many Hire Car options, my suggestion is this should be a menu tab.

The password is a little annoying as you have to enter one upper case and one number,

saying that on the plus side you don’t actually have to register at that early stage.

I found the Hotels very helpful with Ratings and costs, as allows you to do a price

comparison on other websites. The site could also benefit from a separate Hotel tab.


Appendix D

Platform being tested (Desktop, Mobile or App): Mobile

Device being used (Make and Model) to access the platform: Apple Iphone 4






information.

















Total 11

Comments:

The mobile site is an exact replica to the desktop site as oppose to being designed for

mobile.

Page loads can be slow as is responsiveness in particular on the calendar.

The load time for the Route Map was over 15 seconds as an example.

The site is very easy to navigate though and provides a wealth of information.

The landing page on the site redirects to the App.

Useful sections included special mobility assistance, flight disruption and how to check in.

YouTube online check in guide was very informative.


Appendix E


Device being used (Make and Model) to access the platform: Apple Iphone 5C






information.















Total 12

H3 – If I went back to look at other flight times and then proceeded with my booking; the

system did not remember me or my child’s details.

H5 – I found that this was executed extremely well. It told you it was loading with a small

plane going around in a circle. It was aesthetically sound and to the point.

H6 – If I was midway through a booking, there is no way of selecting to go straight back to

the main menu. I must keep hitting the back button.

H7 – No remember my details or autofill option, however, this generally isn’t available on

mobile devices to my knowledge.

H8 – I found that this worked quite well. If I forgot to fill in a field, it told me how many

errors I made and highlighted them in red.

H10 – The help and information is there but it is presented terribly in my opinion. It is one

big blurb of information with so many options. In my opinion, there should be one easy to

find FAQ section, one e-mail and one contact number.


Appendix F








information.






Desktop version on mobile











Total 22

Mobile Version











Total 7


Comments:

When you first visit the website it offers you a choice of visiting the mobile web site or the desktop

web site. The home page gives you a choice of viewing the mobile website or the desktop website.

The Ryanair desktop website via mobile phone offers a poor and frustrating user experience. The

look and feel of the page is clustered, busy and difficult to navigate.

The website is not responsive in its design. When you are looking at booking flights the calendar box

does not appear properly on the mobile phone’s screen. The user has to move across the screen as

highlighted below.

This is incredibly frustrating and time consuming.

Once you have managed to find the correct flights after painstakingly swiping across the screen you

come upon step two of booking your flights. The page highlights the cheapest flights and the

business class flights. The page is not too bad but may prove difficult to those with any form of visual

impairment as the screen text is quite small.

The third page is irksome and very busy. There are lots of options to get through. Firstly you have to

enter your passenger details.

It then asks you if you would like insurance or not. You have to click into a drop down menu which

zooms in on the page as highlighted below. This means you have to zoom back out of the page to get

continue on to the next step. You can also choose your seats on this page, check a bag on, book car

parking, rent sports/baby/musical equipment, book a transfer to and from the airport and receive an

SMS of your flight details.


The fourth step asks you f you would like car insurance. It is not displayed very well if you would like

to rent a car as you have to scroll across the screen to see all the images. If you are not interested in

car insurance, it is a straight forward scroll to the bottom of the page.

The fifth step offers you a choice of hotels and again is not mobile friendly as you have to scroll back

and forward across the screen if you are interested in booking a hotel. Similar to the previous page,

if you are not interested in hotel insurance, it is a straight forward scroll to the bottom of the page.

The sixth step asks the user if they would like to register with my Ryanair which again is not mobile

friendly as it zooms in on the text and distorts the overall text on the page as demonstrated in the

visual example below.


It gives the user the option to skip registration and proceed to step seven. Step seven is the final

purchase page. It is again clustered with information on redeeming Ryanair vouchers and offering a

chance to win free flights.

Then payment method itself is straight forward for the user and follows similar designs to other

websites in asking for your credit card details and postal address.

Mobile site

In contrast to the desktop site the mobile site is excellent. It is sleek, fast and easy to navigate. It takes

seven steps to reach the purchase page but in a much more concise manner.

Step one is to choose your flights and uses a simple and quick interface. In contrast to the desktop

version, the calendar is easy to use and is responsive in its design.

Step two offers the user the lowest fare and the business class fare which is exactly what I wanted.

Unlike the desktop version, step three asks just for your name. It not overly clustered like the desktop

version.

Step four asks for insurance and again is simple and eloquent in its design. There is no drop down

menu and it is very simple for the user to progress to the next step.

The fifth step gives the user the option to choose seats and is aesthetically pleasing in its design. It is

again simple for the user to progress to the next step.

The sixth step is straight forward and offers the user the chance to book extra bags with their flight.

Again there is no problem progressing to the step.

The final step is offers an easy way to pay for your flight and there was no issue here.

Conclusion

In my opinion, Ryanair should scrap offering the user a choice between the desktop version of the site

and the mobile version of the site. It should automatically direct the mobile user to the mobile site.

This site is far more efficient and superior in its design and concept. The only criticism I would have of

the mobile site is that can be a bit slow to upload the pages from time to time.


Appendix G

Platform being tested (Desktop, Mobile or App): APP


Can you please do the following?





information.

















Total 13

Comments:

The App offers a reasonable user experience. The look and feel of the App are very good as

is the use of the aeroplane for page loading.

The app though can be very frustrating at times in terms of responsiveness. This can lead

you to touching the screen again if you feel nothing is happening, resulting making in

touching the screen again and selecting information you may not have wanted.

The calendar though is very easy to use and the options offered to users are limited

meaning you can purchase quite quickly.

Some of the drop down menus can be slow at times.


Appendix H

Platform being tested (Desktop, Mobile or App): App

Device being used (Make and Model) to access the platform: iPhone 5s






information.






Thank you for your participation.











Total: 10

Comments:

Looks good on the app home, clear and minimal.

Clear and easy path to booking a flight.

Not good when you leave the flight booking path. Impossible to go backwards or get back

to the home page.

They make it difficult to contact them.

Hard to get any help or documentation.


Appendix I

Platform being tested (App):

Device being used (i-phone 5S) to access the platform:






information.

















Total: 7

Some comments: The design was great overall of the app but it was let down by the contact us form

which wasn't mobile optimized. This took away from the consistency of the design. I also

experienced errors putting in the countries. I selected a country but it didn't automatically update

the text box a few times – it's nothing major but something to keep an eye on. I thought the section

for help for their services and the app was organized terribly and very hard to navigate

evaluation of user experience across all ryanair digital platforms

Business

ryanair ryanair

ryanair digital evaluation

summary of app evaluation

method of evaluation

summary of mobile evaluation

evaluation of user experience

summary of desktop evaluation

ryanair digital platforms