Using multiple methods to assess the usability of credit card disclosures
Angela [email protected]
Using Multiple Methods | Angela Colter
What are disclosures?
card member agreements
solicitation letters
Why was the U.S. Congress interested?
1. Assess the need to revise disclosures so they better inform consumers of now-common penalty rates and fees
2. Find out how effectively pricing practices are disclosed to cardholders
3. Determine whether penalty charges contribute to cardholder bankruptcies
Using Multiple Methods | Angela Colter
Three evaluation methods
1. Readability analysis using formulas
2. Expert review using plain language guidelines
3. Usability test
Using Multiple Methods | Angela Colter
Why those methods?
Using Multiple Methods | Angela Colter
Strengths Weaknesses
Readability formulas
Quantitative data
Quick & cheap
Only counts what can be counted
Doesn’t tell you what to do about it
Plain language review
Predicts problems and gives guidelines for improvement
Also relatively cheap
Doesn’t mean users will have problems
Usability test Shows how people actually use an interface
Provides quantitative and qualitative data
Not quick and not cheap
Small sample size can be a hard sell
Readability analysis
• Used FOG, SMOG, and Flesch Grade Level
• Took multiple samples
• Carefully prepped the text (removed URLs, trailing periods and hard returns)
• Used Readability Analyzer software
• Explained strengths and weaknesses of readability formulas in the final report
Using Multiple Methods | Angela Colter
Readability analysis findings
Using Multiple Methods | Angela Colter
Flesch Grade Level
SMOG
FOG
Average grade level of cardmember agreements
12
Readability analysis findings
Using Multiple Methods | Angela Colter
Flesch Grade Level
SMOG
FOG
Average grade level of cardmember agreements
12
Average grade level of annual percentage rate info
17
Readability analysis findings
Using Multiple Methods | Angela Colter
Flesch Grade Level
SMOG
FOG
Average grade level of cardmember agreements
12
Median reading level of U.S. adults8
Average grade level of annual percentage rate info
17
Expert review
Used a plain language guidelines document developed by the U.S. Securities and Exchange Commission in 1998
Deals with organization, content and design
Using Multiple Methods | Angela Colter
http://www.sec.gov/news/extra/handbook.htm
Expert review findings
Using Multiple Methods | Angela Colter
The disclosures violated many of the guidelines. The report listed each guideline and gave examples where disclosures followed or violated them.
Expressing a complicated, multistep
process in paragraph form makes it
difficult to understand the relationships
between steps.
By using bullet points, it’s much easier
to see what the steps are and when they
are applied.
Guideline: Use bullet points
Expert review findings
Using Multiple Methods | Angela Colter
At times, gave suggestions for improvements
Original (101 words)
Rewrite (48 words)
If at any time during any rolling consecutive twelve billing cycle period you fail to make two Minimum Payments on a timely basis or exceed your Credit Limit twice we may elect to increase your Purchase, Cash Advance and/or Balance Transfer APRs to the Penalty APRs. All Penalty APRs will remain in effect until, in a subsequent rolling consecutive six billing cycle period, you do not exceed your Credit Limit at any time and you make all of your required Minimum Payments on a timely basis when, in your next billing cycle, all Penalty APRs will no longer apply.
If you pay late or go over your credit limit twice in a year, the interest rate you pay on most things goes up to the default rate. It will go back down when you pay on time and don’t go over your credit limit for six months.
Guidelines: Eliminate jargon
Use short words
Use short sentences
Usability test
Tested the material with 12 users
Gave them two sets of tasks using two different documents: one solicitation letter, one card member agreement
Topics covered:• Annual percentage rates
• Grace period
• Minimum payment
• Late fee
• Over limit fee
Using Multiple Methods | Angela Colter
• Cash advance fee/APR
• Balance transfer fee/APR
• Finance charges/balance computation method
• Changes to terms
• Payment allocation
Usability test plan
Can they use disclosures to locate information?
Do they interpret the info correctly once found?
Using Multiple Methods | Angela Colter
Question Correct answer
Usability test findings
Gave an overview citing task success rate
Followed with a full explanation of outcome, observations and participant quotes
Using Multiple Methods | Angela Colter
Task 8: Finance Charges / Balance Computation Method
Solicitation Letters: Only 1 of 12 located the information and seemed to know the significance of the term average daily balance. No one knew what two-cycle ADB meant.
Cardmember Agreements: All were able to locate the information, but only one was able to correctly interpret the information she read.
This was, by far, the task with the lowest success rate.
Only one out of the 12 participants located the name of the balance computation method used by the credit card in the solicitation letter. Two participants said things like “I know it’s based on the average daily balance, but that's not what it’s called.” Part of the problem may have been the label used on the Schumer box: Balance Calculation Method for Purchases didn’t seem to convey “this is how we calculate the interest you owe.”
Only one person gave a plausible explanation for the definition of what Average Daily Balance Computation Method means. No one who was asked what Two-Cycle Average Daily Balance Computation Method means even tried to guess.
Using Multiple Methods | Angela Colter
Usability test findings
Question P# Doc Time Correct? P’s Answer Analysis
This card uses the “two-cycle average daily balance” method to figure out how much interest you owe. What does it mean?
P6 SL #1 0:15 N -- You pay interest on this month’s and last month’s balance.
It means if you made a partial payment last month, you are paying interest on principle you have already paid off.
No one got it right.
P8 SL #1 0:50 N “I can’t understand it”
P7 SL #2 1:35 N “I really don’t know”
P9 SL #2 -- N “You got me”
P1 SL #3 1:10 N Described averaging
P3 SL #4 0:16 N “No clue”
Detailed information and performance metrics were included in the report’s appendix.
Complimentary approach
Using Multiple Methods | Angela Colter
Findings Weakness
Readability formulas
12th grade reading level Doesn’t tell you what to do (but P.L. guidelines do)
Plain language review
Docs don’t provide hierarchy, group related information, use markers (TOC, meaningful headings), or use simple words and sentences
Doesn’t mean users will have problems(but usability testing shows where they do)
Usability test Users had trouble locating info and often didn’t understand it once they had located it.
Disincentives for reading terms.
Small sample size can be a hard sell(but video puts a face on problems; findings from formulas and P.L. review found some of the same issues identified in testing)
Outcomes
Using Multiple Methods | Angela Colter
Communicating concepts wasn’t the main problem, unfair practices were.
The U.S. Congress and Federal Reserve passed laws and regulations that:
• ban universal default and “two-cycle” billing
• require high interest balances be paid first
• require opt-in to over limit fees
• disclose consequences of making only the minimum payment on monthly bills, not just in disclosures.
Further reading
Using Multiple Methods | Angela Colter
GAO Report: CREDIT CARDS Increased Complexity in Rates and Fees Heightens Need for More Effective Disclosures to Consumers
http://www.gao.gov/new.items/d06929.pdf
A Plain English Handbook: How to create clear SEC disclosure documents
http://www.sec.gov/news/extra/handbook.htm