Author: makinguxbetter

Almost a year ago, I experienced a salient moment that began to crystallize my thoughts around a growing challenge in our industry. This thought had been a hankering feeling that there was a gap (now I know it is more like a chasm) between the rigorous academic research world (Big “R”) and the much more practical UX research world (little “r”). I now feel strongly that both populations desperately need to bridge it and have a lot to offer each other in doing that. We have a lot of answers to each other’s questions and need to come together for the betterment of user experiences everywhere.

It was July of 2017 and I was standing at my poster at the Veteran Affair’s Health Services Research & Development (HSR&D) National Meeting. Our team had been conducting some very important user research, which we were documenting and sharing in the form of a set of journey maps. The work hadn’t been accepted as a talk, but I was excited to share it as a poster. [1] While not my first poster session as a presenter, it was my first academic conference and, now that I look back, I did not appreciate the differences in the audience.

So, here I am, standing by my poster as the conference attendees start pouring into the session. It doesn’t take long before I have my first visitor standing in front of my poster, beverage in hand, taking a gander at our beautiful and exciting “research.” I give her a few minutes to read and digest the content, waiting for her to formulate a question before diving into a discussion. I see her nose start to crinkle and a classic inquisitive expression begins to spread across her face. She’s formulating a question! I start to get excited thinking about what she might ask and how the conversation might evolve into a deep and thoughtful discussion about the work. She pauses for a moment and then says, “What exactly is a ‘Master Human Factors Engineer’ and where did you go to school for that?” 

I stood dumbfounded for a few seconds, an amazing occurrence for those of you who know me personally. I am rarely at a loss for words and I was flabbergasted.

Once I regained my composure (and stifled my deep egotistical anger), I politely answered her question. As we spoke, I realized that most of the people in this audience had no idea what a “journey map” was or why UX’ers create them. So, I was placed in my first cognizant situation where I had to actively bridge the Big “R” – little “r” gap. And, I had to do it thinking on my feet.

The first thing I did was stop referring to “journey maps.” Instead, I called them “timeline based visualizations of qualitative data.” Just changing the language helped begin to build a bridge. The next step was to begin highlighting the level of evidence that we had collected and applied to the maps, which in UX circles was extensive for a journey mapping effort. Gaining some respect, I was able to have some very meaningful conversations explaining the power of these types of visualizations in disseminating research findings and enabling operational partners to take action on our research to make an impact. I ended up exchanging business cards with quite a few researchers who visited my poster. Since then, several of them have reached out to me for copies of the poster and for follow up discussions on applying journey mapping to their data. One reached out 7-8 months after the conference, which indicated to me that the poster had made an indelible impression. I feel like that experience, while jarring, was a good lesson in bridge building.

This memory has, and probably will always be, stuck in my mind. It suggested to me that many people in the academic research community and UX research community may not even be aware that this “gap” exists. This is deeply troubling to me because this “gap” is where the bulk of the meaningful work happens. This is the area that most of us need to live in, but we are struggling to live there for different reasons. Academic researchers are clearly struggling to make their work “efficient”, “innovative”, and “impactful.” UX researchers are clearly struggling with making their user research “generalizable”, “rigorous” and “credible.” As I pontificated over this for several months, an image started to form in my mind.

At one end of the spectrum, we have the most rigorous of academic user researchers (BIG “R”). These are cognitive scientists, psychologists, ergonomic scientists and other human factors professionals producing “research” that furthers our understanding of how the human brain and body functions within systems of technology. At the other end of the spectrum, we have hardcore UX’ers (little “r”). These are interaction designers, information architects, service designers and other user experience professionals who employ “research” to understand user needs within a very specific context in order to produce wildly innovative solutions to practical real-life problems.

Each population excels at their end of the bell curve, executing work that requires an advanced level of expertise in each context. But, most of the work doesn’t require that level of execution. Most of the work is in the middle of this bell curve and both populations have challenges inherent in their working environments that make it hard for them to get “the work” done in this gap so that it will be valued by other people. Each population is pushing hard to try to live in “the gap,” to improve the usefulness and longevity of their work. And, each population has the answers to the other population’s challenges. We need to partner together and help each other.

For example, prior to my poster presentation, I observed a discussion thread during the conference I was attending. Keep in mind that this was an academic conference. So, in this context, “research” meant the effort and results of scientists.

Throughout the conference, I heard discussions about how challenging it is to make “research” actionable and responsive to the needs of the organization. I repeatedly heard it throughout the sessions that I attended, including throughout the HSR&D State of the Service plenary by the organization’s director. [2] As someone coming from a UX “research” background, I found this discussion thread surprising and confounding. I mean, isn’t that the driving force behind why we do “research”? Aren’t we always laser-focused on how our work will help the organization make better design and investment decisions and, at the VA, improve care delivery?

“We need to know what our leadership expects of our research. [The Secretary] is not a patient person and that is what you need in these days… He is ready and looking for action when there is an important research finding. They want metrics for progress and impact. They clearly want faster results. They want bolder goals… greater innovation… they want implementation of our findings into Veteran care in ways that we can see.” – David Atkins, M.D., M.P.H., Director of HSR&D; Acting Deputy Chief Research and Development Officer [2]

I have to be honest. This truly made me feel like an outsider, in a curious and disturbing kind of way, sort of like one of those early anthropologists sticking out from the natives with my safari hat and multi-pocketed vest. Having worked with a handful of team members with academic research backgrounds, I thought I understood their constraints. I mean, I had worked in a lab as an undergraduate and understood the “publish or perish” pressures. I became familiar with the IRB process through my more recent work and am even familiar with the relatively recent development of a scientific replication crisis. But, this conference provided me an up close and personal look at the academic research culture and I realized just how foreign it was to me. As I observed the conference presentations and networking conversations, I started to have a much better understanding of the impacts of the history of academic research, their value systems and their current working constraints, which produce the challenges that they are facing today.

The nearly singular focus of UX researchers is to develop innovative solutions to user needs by being efficiently responsive to the needs of the business. We are extremely adept at providing the data to support business decisions. We understand that those business decisions are going to be made with or without our data. So, we must get the data to the decision-makers in a timely fashion. I couldn’t articulate it at the time, but during that experience, I couldn’t shake the thought that the challenges being faced by the academic community are the strengths of the UX community and vice versa.

In a complementary example, shortly after the conference, I saw this tweet. It was like the breath of fresh air that I didn’t know I was looking for, reinforcing the existence of “the gap” from the UX community.

This article demonstrates a level of rigor not currently common in the UX community. [3] Kath, whom I deeply respect for her practicality in her approach to “research” in the UX context and her current knowledge of the academic literature, aptly pointed this out.

In the April of 2017, just a few short months before my academic poster presentation experience, I had the pleasure of participating in Kath’s workshop Research in Practice: Studies that UX/CX pros should know (2017) at the annual conference of the Washington DC chapter of the User Experience Professionals Association (UXPA DC). During this talk, she provided a wonderful brief of foundational material from the academic community for current UX design principles. Many of the attendees had internalized design best practices through their practical work but were not aware of the nuances of the “research” from the academic community which provided the basis for these practices. The questions from the audience were incredibly interesting. Among other topics of discussion, we commiserated at how often we don’t have that academic “research” at our fingertips when our customers question our design decisions. We also lamented that, at times, we are forced to “reinvent the wheel” on a ridiculously tight time frame with contextual studies of subpar rigor.

Kath’s tweet brought me immediately back to her workshop and aligned that experience with my academic poster session experience. That little voice in my head got louder. Despite the available resources, those of us in the UX community often don’t maintain a focus on responsibly applying mathematical constructs (such as statistics or metrics). Many UX’ers also don’t have the depth of knowledge to select the right methods when designing research studies or to execute those methods accurately. A symptom of this is the reoccurring debate about the minimum number of participants (spoiler: it depends on your research question, method and needed level of confidence). The academic community has some answers for us too and, in many cases, it’s incredibly well documented. We just need to harness it more efficiently and effectively.

There has been another hankering thought in my mind. As evidenced by my experience presenting my poster and other personal experiences as a practicing UX’er amongst academics, I have also become concerned that there is a growing mutual disdain contributing to “the gap.” I fear that academic researchers see UX’ers as uneducated, sloppy and unqualified, while UX researchers see academics as impractical, pretentious and slow.

“In the past I’ve taken a rather relaxed approach to my research. With the understanding that it was important to just get cracking, get out into the world and start talking to users…” – Simon Hook in his blog post review [4] of  Just Enough Research by Erika Hall

This is incredibly unfortunate because I believe that we have a lot to offer each other to overcome our unique challenges, but we may not be talking to each other effectively or working together to reach a common ground based on mutual respect. I was relieved to find, as I began reading and networking to explore this further, that there are people in both communities who recognize the gap and have ideas about how to address it. I don’t believe the conversation has made it mainstream and I’m hoping that this blog post might help by connecting some of these sources.

One of my first calls was to my colleague, Dr. Jason Saleem, who co-authored a paper on this topic a couple of years ago. [5] I re-read Jason’s article several times before he and I had an opportunity to catch up over a virtual coffee. He commiserated with me and reassured me that I am not alone in this concern about “the gap.” Apparently, this is something that he thinks about quite a bit. I’ve enjoyed my experience working with Jason and value his opinion as an academic striving to live “in the gap” with me.

“Although the HIT research community has strong dissemination networks, as evidenced by relatively high citation counts, research often has little impact on or relevance to actual operations and development of HIT.” – Saleem et. al. [5]

If you haven’t had the opportunity to read this paper, I highly suggest getting a copy and pulling out your highlighter. Please reach out to me for coffee or beers if you’d like to pontificate. There were several key takeaways for me:

• Performance criteria for “Big R” is tied directly to publication (see “publish or perish“). But, performance criteria for operational partners (the business offices) is rarely tied to publication. This contributes to the fact that many real-world, pragmatic advancements and lessons learned never make it into research literature.
  • In my opinion, this is not just because of lack of attachment to performance criteria. Could it also be because the research literature actively discriminates against this work due to “lack of rigor”? As a result, often times, pragmatic advancements are published in books (not journals) or blogs, which are often seen as “gray literature” by academic researchers and not given the same attention or consideration.
  • I’d also like to put out a personal plea to all UX’ers to begin citing sources in the mass “gray literature” that we produce. It is important to your credibility as a professional and to the development of the knowledge of your reader. It doesn’t much matter to me what citation format you follow, just make it easy for people to find your source. And, if you don’t have a source and the idea didn’t originate from your brain, seriously consider whether or not you should include it (especially if it is a statistic!).
• Business stakeholders have a more community-based reward system where achievement is attributed to organizational or team goals. “Big R” research goals may need to be subsumed into operations goals for mutual success.
  • I am wondering, though, about the reluctance of “Big R” researchers to share their raw data and the culture of mistrust that researchers feel that their findings will be “stolen” by other researchers or by business stakeholders. There seems to be this underlying distrust fueled by the publication performance measure that actually counteracts or negatively impacts dissemination. “little r” researchers don’t feel that. We openly share. We blog “now” and then publish in a book again later, if its good enough to stand the test of time. “Big R” researchers live in a different world where that data can only be published once, so they wait. True?
• For projects focusing on business needs, sharing of the results is often discouraged or prohibited. The concept of “academic freedom” that many researchers enjoy does not exist in these business contexts. “Big R” researchers need to understand how to present findings in a sensitive and professional manner, without compromising proper interpretation of the data.
  • Boy, don’t I know about the business constraints for sharing your “research”! This is something that UX’ers are painfully familiar with (see my post on The Handcuffed UX Portfolio). That said, often times “little r” researchers work around that by freely and quickly share their work via blogs and books without exposing the identity of the business (with the permission of the business of course). This is a common practice. But, it has to be done with an incredible amount of finesse. UX researchers have developed that skill over time, and often through a lot of blood and tears. But, UX’ers could probably use some pointers from academic researchers on presenting the data without compromising proper interpretation.

Aside from my own takeaways, based on our discussion, Jason advocates most strongly for academic researchers to structure their studies very carefully to ensure timely and responsive interim results for business stakeholders. In this fashion, academics can still pursue their larger publishable research aims while also supporting business decisions with customized short-term study/research objectives designed to meet business needs.

I offered that UX researchers have a strength in working with business stakeholders to uncover what those objectives should be. It is our bread and butter to be able to understand the business context and help determine how research can support the business decisions. Perhaps academic researchers can foster working relationships with UX researchers to develop and practice these valuable skills. Business needs also often pivot. Once those short-term objectives are formed, they will change. Academic researchers could benefit from working with UX researchers who have a lot of practice pivoting. Together, we could build stronger long-term academic research projects that are responsive to business needs as they pivot.

UX researchers also have a lot to gain from working partnerships with academic researchers. In addition to exposure to more rigorous study designs and responsible method selection, UX researchers have research questions that they can’t (or shouldn’t) answer in the timeframes that they work within. With small sample sizes and highly specific contexts of use, UX’ers aren’t positioned to contribute generalized foundational findings to the body of knowledge. But, we are painfully aware of what the industry needs are for this information. UX’ers could help feed some of the foundational and generalizable research questions into the pipeline for academically rigorous exploration. Because industry often leads education in the integration of research with design, this feeding of research questions from UX’ers could help academic researchers stay relevant.

“The current situation is marked by the fact that practice leads education in the integration of research with design. The integration is going well. The biggest challenges are the incompatibilities between how design research is done in practice and how research takes place at the university. With the rise of sponsored projects in academia, the need for integration becomes imperative.” – Sanders [6]

This brings me to another little gem that I came across. In May 2017, the same timeframe when I was initially having my epiphany, Dr. Elizabeth Sanders‘s paper on this topic was accepted and published. [6] In this paper, she describes her 35 years of experience simultaneously working at both ends of the spectrum. She has acted simultaneously as a rigorous “Big R” researcher and a practical “little r” researcher. Reading her paper was incredibly validating for me as she provided compelling evidence that “the gap” not only exists but there is a growing interest in it. She also adeptly articulates the urgency to close the gap. Interestingly, rather than working to bridge the gap, she advocates for the development of a new “design research” practice at this crossroads of education and practice.

This article was inspired by Elizabeth’s participation as a Steering Committee Member for the newly formed North American Design Research Organization (NADRO).  I hope to see the foundation of more new organizations to support the development of the “new thing in the middle.” In the meantime, it is refreshing to see some of the original UX masters producing educational and certification programs designed to help bridge and/or fill the gap. These include the NN/g UX Certification program, HFI’s usability certifications, and Jared Spool’s Center Centre school. Many universities are also now offering UX certificates. Budding UX researchers now have options for academic style training that was not available just a handful of years ago. And, those of us with decades of UX research experience now have certification options to help bring credibility to our experience.

Certification, certificates and formalized education will be critical for UX’ers as we respond to the increasingly savvy businesses who expect more rigorously responsible research and as we partner with academia. That said, our current educational system does not meet the needs of industry or the needs of the next generation of employees. It will be critical for academia to reinvent their model. [7] Hopefully, changes to the academic system and growing academic recognition of UX will help to encourage the cultural changes needed for academic researchers and UX researchers to build a framework of mutual respect.

“Until journals require scientists to submit notebooks, and until sharing your work and your data becomes the way to earn prestige or funding, people will likely just keep doing what they’re doing.”  – James Somers from The Scientific Paper Is Obsolete. [8]

I believe that we, as UX’ers and academic researchers, are presented with this challenge during a fundamental change in the culture of research. In support of addressing the replicability crisis and accelerating academic development of knowledge, there seems to be a growing interest in revolutionizing the publication of scientific findings. Groups of people and technological products are emerging to enhance and facilitate the sharing of raw data. UX’ers and academic researchers can come together as the scientific dispositioning environment reinvents itself to become part of the design of the solution.

There’s a lot of work to do. But, this is a very exciting time to be a “researcher,” whatever that means to you and your work. I hope to break this discussion wide open and get to work in bringing “Big R” and “little r” research together for the common good. As we do this, I would like to encourage all of us to avoid meaningless distractions (like the semantics of language) and take on the hard work of designing a solution for ourselves where we can collaborate vigorously to build better user experiences for everyone. What will you do today to jump into the gap?

References

1. Cook, A. & Nebeker, J. (2017).  Applying User Experience Journey Mapping to the Modernization of VHA Health Information Technology Solutions. Presented as a poster at HSR&D/QUERI National Meeting.
2. Atkins, D. (2017). HSR&D State of the Service: Ensuring Success in an Uncertain Environment. Retrieved from https://www.hsrd.research.va.gov/meetings/2017/streamed_sessions/davidatkins.cfm
3. Linowski, J. (2018). Insignificant Significance: Why Reverb.com Rejected A Winning A/B Test. Retrieved from http://ht.ly/XlUD30inhuM
4. Hook, S. (2018). Just Enough Research by Erika Hall. Retrieved from https://medium.com/@hookster81/just-enough-research-by-erika-hall-6aa2519ecb96
5. Saleem, J. J., Militello, L. G., Russ, A. L., & Wilck, N. R. (2016). The need for better integration between applied research and operations to advance health information technology. Healthcare, 4(2), 80–83.
6. Sanders, E. (2017). Design Research at the Crossroads of Education and Practice. She Ji, 3(1), 3–15.
7. Belkin, D. (2018, April 8). One Year of “College” With No Degree, But No Debt And a Job at the End. Wall Street Journal. Retrieved from https://www.wsj.com/articles/one-year-of-college-with-no-degree-but-no-debt-and-a-job-at-the-end-1523288219
8. Somers, J. (2018, April 5). The Scientific Paper Is Obsolete. The Atlantic. Retrieved from https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/