Mass.gov/courts Usability Study

An in-depth usability evaluation of the Massachusetts Court System section of mass.gov, focused on the experience of jurors and self-represented litigants.

Problem

Mass.gov/courts is the visited by 4.7 million users annually. The web team at mass.gov was aware that the users' experiences could be improved for self-represented litigants and citizens responding to a jury duty summons.

Process

Expert review: Each team member conducted an independent usability evaluation using the Structured Heuristic Evaluation of Online Documentation by Kantner et al. (2002). We then combined our findings, identified themes, and wrote a full report and presentation for our counterparts at mass.gov/courts.

Usability testing: Based on the expert review findings, we developed a within-subjects, formative usability study, including a test plan, screener, and moderator guide. 14 participants completed eight tasks in a 60-minute, moderated, in-person session. Half of the tasks focused on self-represented litigants, and the other half on responding to a jury summons. We tallied cumulative results and identified qualitative trends, as well as task completion and difficulty metrics.

Findings

Summary

• We documented 11 usability issues, the most critical being overwhelming and broken PDFs
• Participants were able to respond to a jury summons with minimal effort
• Juror tasks including “what to do when you lose your jury summons” and “juror attendance” had a 25% success rate and required a moderate level of effort

PDFs are Overwhelming and Sometimes Didn’t Open

Of all the usability issues identified, the most critical issue is difficulty with PDFs. PDFs used throughout the site were hard to search and navigate because they were optimized for print. Users expressed frustration and were unable to complete tasks related to finding information in a PDF, with some users giving up completely. Additionally, users expressed frustration when PDFs wouldn’t open.

Recommendations:

• Avoid PDFs when possible; display information on the website
• Prioritize navigation paths to webpages (not PDFs)
• Fix technical errors causing some PDFs to not open

Information Architecture Did Not Match Users' Mental Model

We also found instances of confusion with the site’s information architecture stemming from information scattered across multiple pages and organized in non-intuitive ways, requiring participants to read and consume information they did not always find relevant.

Recommendations:

• Have fewer links that direct users to the Massachusetts Juror Service Website as the portal requires login credentials
• Move contact information from the middle of the page to the bottom
• Add a “Divorce” quick link on the “Family Matters” block on the self-help page
• If “Divorce” is frequented enough, consider having its own block on the self-help page
• On the Divorce page, develop more relevant headings that map to what the content will be about
• When nuanced rules apply to laws, offer users a series of questions to guide them toward answers that are applicable in their situation
• When practical, consolidate information on one topic or task into one page

Next Steps & Reflection

Our findings presentation to the mass.gov/courts web team was well received and provided recommendations for both quick wins and long-term issues to address. While issues related to broken PDFs stemmed from larger technical problems, they immediately implemented some of our recommendations around content organization. From here, we recommend they do a card sort study to further clarify information architecture issues.

Access to the court system is an inalienable right as a resident of Massachusetts. Throughout my work, I want to have an impact in ways that matter and improve people's lives, and this project fulfilled that.

© 2021 Rebecca Mayfield. All rights reserved.