Knight Commission – Usability Testing

Knight Commission – Usability Testing

All writing, prototyping, images, video content produced by Cody Johnson.  Access the functioning demonstration of the prototyped alterations by clicking here.  


I developed the idea for my project with a fortunate combination of timing, opportunity, and my shifting interests with information and sports media.  I spent the summer of 2014 in Miami, Fla., for an internship with Knight Foundation, a nonprofit that funds media innovation and community engagement projects.  While my interest and appreciation of the work of Knight Foundation drew me to the nonprofit, my interest in one of its funded information projects compounded my desire to work with Knight Foundation.  The Knight Commission, funded by Knight Foundation and founded in 1989, is a committee of current and former administrators in higher education dedicated to encouraging academic and financial integrity in college athletics.  The Knight Commission conducts research and audits and produces reports for university administrators and conference and NCAA leadership to reference and use for college athletic reform.


The Knight Commission produced a database that provides university athletics spending data.  The data reveals raw numbers and spending trends of resources dedicated to academics and athletics.  The database allows users to explore data profiles of conferences and individual institutions, as well as the option to create customized reports that compare specific variables of particular institutions.

While I was with Knight Foundation, I had the opportunity to speak with Knight Commission Executive Director Amy Perko.  I intended my conversation with Perko to focus on the details of the work from the Knight Commission and future projects and initiatives.  At one point during the conversation, Perko acknowledged that one of the problems the committee recognized and plans to address is the lack of use of the database, particularly from journalists.  I was familiar with the database, both from personal interest and research conducted through a previous class.  I proposed the idea to Perko of identifying why journalists are not using the database could be a very appropriate project for me to pursue.  Perko was encouraged by the idea and responded that the Knight Commission would consider implementing changes or suggestions I propose at the conclusion of my project.

Before I began the process of conceptualizing my usability tests, I first had to first understand as much about the internal objectives of the Knight Commission.  I had the opportunity to have a conversation with Scott Hirko, an assistant professor and director of the sports management program at Central Michigan University and an associate of communications and research for the Knight Commission.  Hirko manages the web analytics for the Knight Commission spending database, reports key numbers to leadership, and implements decisions on behalf of the Commission.  Our conversation focused on current usage by journalists and the initial plans the Knight Commission might have to address this problem.

Hirko said journalists mainly use the database to drill down to spending patterns of specific schools or a particular region.  While Hirko did admit some of the backend analytics were not sharable at the moment of our conversation, he did reveal that the Knight Commission database was referenced in 60 news articles in 2013.  Hirko said the main objective for the Knight Commission with the spending database is to “keep the database on the front burner” and to encourage the media to engage with it for news stories.  The Knight Commission wants more people to know the power of the database, and how to get people to use it to impact policy.  Hirko listed journalists, college administrators and policy makers as the main constituents of the database, but decided to focus on journalists as the best way to encourage policy makers to enact policy that suits the mission of the Knight Commission.  Hirko elaborated that journalists are important stakeholders because they can applyconsistent pressure on decision makers.


After my conversation with Hirko, I was able to hone the focus of my question with more accuracy.  As the minimal amount of press mentions verified the lack of engagement with journalists, what tweaks can I make with the Knight Commission spending database to make the information more dynamic and shareable for journalists?

The direction of my usability tests that will generate insights for this question focused on two issues: customization and understandability of information.  I began the process with my own preconceived assumptions of what needs improving, but I must remember that, according to Steven Krug in Rocket Surgery Made Easy, that the purpose of usability testing is not to prove anything, but to gather insights that will enable me to improve what I am focusing on (Krug 19).  In order to keep my project within a workable context, I also made sure to focus on one of Krug’s reasons as to why usability tests work: most of the serious problems tend to be easy to find (Krug 21).  Easy to find and easy to fix are two different things.  If my usability tests revealed significant problems that I would be able to fix with my level of design skills, I should be able to provide strong suggestions for the Knight Commission.

Because Hirko stated the problem that the Knight Commission wants to see more use of the database from journalists, it was obvious that journalists must be my primary source of users for the tests.  Because the database focuses on financial data of college athletics, I made sure that the users I recruited to participate in the tests had a background in sports or financial reporting.  Krug notes the domain knowledge of the target audience will reveal problems more representative of actual users (Krug 46).  The users I ultimately recruited for testing range from award-winning student journalists to journalism school faculty to professional sports reporters from the Reno Gazette-Journal.


After I conducted my usability tests, I was ready to begin implementing tweaks from the various insights.  Because I was not building a website, I could skip the wireframing stage and go directly to creating visual treatments of particular sections of the database.

One of the problems I noticed many of the users were experiencing was the time spent on finding information between the two major sections of the database.  The first section, labeled “Option 1,” gives users the option to explore data by typing in the name of a school in the search bar.  The second section, labeled “Option 2,” encourages users to build customized data sets.  As users began using the database, many did not venture between the two major sections as they attempted to answer various questions.  Perhaps the wording of “option” forced users to make a decision about which data options to use, and stay within the realm of that decision throughout the process.  For my tweak, I decided to replace the “option” wording with the title of data option.

before and after - custome reporting Before and after - splash screen

Another problem users encountered was the timeliness of the data.  Since the users were journalists who work with telling stories of new and updated information, some had reservations about the value of data from 2012 when the calendar year is about to close on 2014.  As this is more an issue about upkeeping the database and not so much about design and usability, it is still important to disclose the data capture and upkeep process for journalists who wish to understand the disconnect.  I included a disclaimer at the bottom of the two search options on the splash page so users have a better understanding of why the most updated data only extends to 2012.

When talking about usability design of a database for journalists, we are talking about people who typically work on deadline and need to find information and make comparisons of trends as quickly as possible.  While the customizable data options are great for journalists to compare data profiles of institutions or conferences, there are a few extra customizable features some of the users wanted to see.  Data profiles are displayed in alphabetical order by default when displayed as a list.  When journalists want to reorganize the lists for things such as “highs” or “lows,” there is no feature that allows users to do this outside of manually finding such a number.  I decided to add a feature where users will click on a particular variable of the list, such as a year, and the data will be reorganized from high-to-low based on the other variable (such as academic spending).  Users can click again to reorganize the data from low-to-high.  The easier it is for journalists to customize and reorganize the data to find trends and dynamic data points, the more likely journalists will reference the Knight Commission database for future work.

before and after - custome sorting

Another major roadblock for the users who wanted access to the data further than the website was the download process.  Many of the users did not like having to fill out information in order to download the data, both for the time and the disclosure of personal information.  Some users said if the Knight Commission wants journalists to download and reference data in their work, it might have to be willing to offer the use of its data without asking for any information in return.  Based on those insights, I decided to redesign the download pop-up screen to offer users different ways to download the data.  An embed code for the different types of visualization, as well as different file formats of the data to download are options for users.

before and after - data profile

Although not directly related to the usability of the database as it currently functions, some users did comment they would like to see a deeper breakdown of some of the finances.  Multiple users acknowledged the desire to see the financial information for sports other than football.  If financial data is available by sport and by gender, journalists would have access to numbers that could generate a wide variety of story ideas.  Another suggestion would be to include data reports or suggested story ideas with an occasional newsletter.  If the Knight Commission presented journalists with new ways to interpret and synthesize data, journalists can tailor the ideas to the local, regional or national beat they are responsible for covering.  These two ideas are presented as suggestions outside the primary purpose of the test because these are more concerned with strategy and content, not so much usability design.  Still, multiple test subjects spoke to the suggestion, so it is worth noting.


For my overall analysis of the degree of usability of the Knight Commission spending database for journalists, a few tweaks to make some of the data organization more revealing of key trends and eliminating steps for journalists to download and share data are important for encouraging more use of the database.  Many of the users commented on the usefulness of the information and the importance of the visualization in the form of graphs and charts.  Although most of the users were not familiar with the database prior to the tests, many commented on the appealing design and interaction.  Some even bookmarked the database on the spot.  Journalists need quick and unburdensome access to download and sharing features.  They might understand some of the jargon associated with athletics in higher education, but it is unwise to design based on assumption.  Journalists need to understand the source and definition of the data points, otherwise there is too much hesitation to reference the database at all.  Journalists want to analyze number sets from their own lens.  Providing clickable features that allow users to reorganize data lists to reveal more dynamic trends will lend to greater reference for story ideas.  These major insights discovered through the process of usability testing, if addressed by considering some of the suggested tweaks, could lend to solving the problem of increasing use of the Knight Commission database by journalists.


Click here to view a live demonstration of the prototyped alterations.