when too many is just enough - jeff pass at uxcamp dc 2013

Hey, I’m Jeff Pass.

It starts with a guy introducing himself…

I’m the one that mentioned male impregnation during my presentation at last year’s UX Camp DC… remember me?

Then he makes his pitch…I’m working on a presentation for the IA Summit.

It’s called, “When Too Many Is Just Enough: Citizen Engagement and Federal Government Websites.”

I’d like to workshop part of it with you.

Here’s the (boring) summary:

Statistical significance is an important consideration in usability studies. You need a certain minimum sampling to ensure valid results, but at the same time, too large a sampling creates more work without increasing significance or validity. Nevertheless, there is a case to be made for casting a wide net and engaging a larger audience in order to actively engage users.

Then he gets to the point...So today I want to talk about online card sorts.

Open, closed, reverse, it really doesn’t matter… any kind of card sort you like as long as it is online.

More to the point, I want to talk about how many participants you want/need.

So, you’ve got yourself a card sort…

How many participants do you need? (1)

According to Optimal Workshop, for an open card sort you need:

How many participants do you need? (2)

According to Optimal Workshop, for a closed card sort you need:

How many participants do you need? (3)

And there are lots of other opinions…

So how many did we do?

???

Over 1109 (1110 to be precise)

But why? What were you thinking?The 2012 Digital Government Strategy calls for large-scale citizen engagement.

Increased participation doesn’t increase validity (and can complicate analysis and reporting), but it does allow us to directly engage the wider public in usability studies and give them a voice in organizing the (federal) websites that are meant to serve them.

And how did you engage them?Directly…

Social media was our recruiterA blog post was our screener

OptimalSort and TreeJack were our test vehicles*

*And yes, we did include a free text comment option and yes, analyzing the results was onerous, but the additional effort was well worth it because while quantitative data (the sort results) had pretty well stabilized by 25-30 participants, every comment provided valuable qualitative data.

So what do you think?Share your thoughts and experiences about large-scale usability studies and direct user engagement

But enough about me; what do you think about me?

Err… I mean… what do you think about large-scale usability studies, direct user engagement, and too many being just enough?

Thanks for your time & participation!

Jeffrey Ryan PassLead User Experience ConsultantAquilent (www.aquilent.com)

jeff.pass@aquilent.com@jeffpass

UX Camp DC 2013