Response to Pirolli's Critique of MAX model
Chris Tilt, cet@webcriteria.com
CTO, WebCriteria, Inc.
In "A Web Site User Model Should at Least Model Something About Users", Pirolli makes editorial comments on a WebCriteria product press release, a paper "The Max Model: A Standard Web Site User Model" (Lynch, Palmiter, Tilt, HFWEB '99), and the conference review process. We are disappointed by his style of comment and with the conclusions he draws. His inflammatory comments, speculation, and editorial tone are inappropriate given his stature as a scientist and his presumed respect for the juried editorial process.
I would like to address the two main issues that Pirolli brings up: the nature of press releases and WebCriteria's Max Model.
First, in regard to the press release, it is common to make product statements without supplying all of the supporting arguments. This is customary across the industry, including Xerox, and does not typically lead to the assumption that claims are unsupported or untrue. For example we let the readers use inductive reasoning to conclude that a software agent produces repeatable results. Our release made no claims based solely on the Lynch, et al. paper.
Second, we must not underestimate the utility of a "Standard Observer" of web site usability. Pirolli claims "Max is not a psychologically real model of users." We agree since no model is a psychologically real model of users. However, there is great utility in an approximation for both engineering (commercial) and scientific (research) purposes.
For example, GOMS requires non-errored behavior modeling, which is clearly not a real attribute of humans browsing a web site. Thus, Max does not use the Back button. Another assumption we made was that Max was tireless and would pursue all goals. This is also unreal, but required to measure pages deep within web sites. These assumptions make the model less psychologically real, but allow for an approximate model to be built that ultimately provides utility to web site owners.
Lynch, et al. presented the concept of a "Standard Web Site Observer" that formed the basis of automated measurement of web site characteristics, a set of results of human experience in seeking information on web sites, and an initial comparison of how the accessibility metric would compare web sites relative to how the people actually performed on the web site. No claim was made that Max "predicts" actual user time. He/She (Max or Maxine) describes the time it takes her/him to navigate to the target. Comparisons between target accessibility (within a site or across a site) can then be made.
A standard observer is meant to facilitate comparison and repeatability of measurements. It is descriptive and not predictive. The Lynch, et al. paper mentions two historic standard observers, for color measurement and for sound. The Max model was created using GOMS to provide an evolutionary path for this class of standard observers so that additional cognitive processing and information seeking strategies may be added in the future.
In his editorial Pirolli discusses only the psychological model of the Max model human and does not list the discussion of the cognitive portion of the model discussed in the Lynch, et al., (1999) paper. In the section of the paper describing the cognitive modeling used, the authors stated:
"The selection rules and behaviors that the Max Model employs are drawn from the emerging body of literature and research concerning user behavior while using the web and from previous research concerning display layout."
The authors went on to describe examples of the GOMS modeling approach used.
Pirolli, in his editorial, used our data in a statistical correlation that has nothing to do with the Max metrics. The Max metrics are only a comparative measure, as stated throughout the Lynch, et al. (1999) paper and in the press release that Pirolli cites.
" Another encouraging result is that the Accessibility Values in 8 out of 10 comparisons aligned with user data. This is a promising and intriguing result, but not yet conclusive. "
More research needs to be done. In fact, a larger, more robust study is already underway with sixty (60) participants performing twenty-four (24) total tasks on six (6) different web sites. The results will "…validate the Accessibility metric which is an abstraction of the Max Model" (Lynch, et al. 99).
Pirolli's assessment does not refute the conclusions presented about user experiences. To quote our Max paper,
" Currently, the main conclusion to be drawn from this study is that use data can be consistently and reliably measured about the differences in finding particular tasks on competing web sites. The data is highly correlated (internally consistent) and conclusive that a set of homogeneous users presented with the same task will have similar results in terms of time and subjective experience when searching for that task on sites that were designed to support those tasks."
We maintain the importance of a "Standard Observer Model," the findings of the research, and the conclusions related to the actual implementation and use of the model. We continue to invite a studied dialogue on behavioral modeling of web site usability with the ultimate goal of improving web sites for users. To this end, we acknowledge and respect the foundational work done at Xerox PARC on GOMS, and invite them to further a scalable solution to modeling website behavior.
© Internet Technical Group
Last update: April 30, 2000
URL: http://www.sandia.gov/itg/newsletter/mar00/response_critique_max.html
hosted by Sandia National Labs
Disclaimer: Neither Sandia Corporation, the United States Government, nor any agency thereof, nor any of their employees makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately-owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by Sandia Corporation, the United States Government, or any agency thereof. The views and opinions expressed herein do not necessarily state or reflect those of Sandia Corporation, the United States Government or any agency thereof.