Outcomes Measurement 2.0
The outcomes measurement methods employed by the ACORN community rely on the use of sophisticated statistical methodologies to assure that the questionnaires provide a high "yield" of useful information while recognizing the constraints on time and money in real world clinical settings. In order to achieve a high degree of measurement "efficiency", the psychometricians and statisticians involved in the analysis of data from ACORN questionnaires are employing measurement methods that are unfamiliar to most clinicians who have been exposed to research based on tradition outcomes research methodology, which we refer to as "Outcomes Measurement 1.0." This term refers to a tradition of measurement that started with Francis Galton and intelligence testers such as Terman and Wechsler, extending to the present in the huge catalogue of copyrighted tests available for purchase. Tests in this tradition were generally static, with updates every few years, and they were "one size fits all," with the same one or two forms that everyone had to use. Tests were distributed in a fee-based proprietary model like book publishing.
The arrival of the 21st century with ubiquitous computers is changing many things, and the measurement of client progress is unlikely to be an exception. We call the old tradition "Measurement 1.0" and what seems to be coming, "Measurement 2.0," following the popular terms "Internet 1.0" and "Internet 2.0." Like what has come to be called the
Web 2.0 (of which
http://en.wikipedia.org is a prime example), Outcomes Measurement 2.0 depends upon the ability of large communities of clinicians and researchers to collaborate in developing measurement methods. Such natural communities of collaboration are able to generate data, perform analyzes, and share information on a scale unheard of a decade ago. For example, the organizations utilizing the ACORN COMMONS currently generate data on between between 30,000 and 40,000 psychotherapy sessions monthly (a number that is steadily rising) using a wide variety of patient self report questionnaires.
The
National Institute of Health's Patient-Reported Outcomes Measurement System is an exciting example of measurement 2.0 at work.
Following are comparisons and contrasts between Measurement 1.0 and 2.0 methods across key domains of activity. Comments and contributions are welcome!
Warren Lambert & Jeb Brown
Questionnaire ownership and licensing
| Measurement 1.0 |
Measurment 2.0 |
| Reliance on copyrighted and published questionnaires |
Item bank and resulting questionnaires belong to community of users. See CopyRightCopyLeft |
| Copyright holder may charge fees for the use of questionnaires |
No fees for questionnaires constructed from items in the shared item bank |
| Copyright holder may place conditions or restrictions on the use of questionnaires |
Each organization is responsible for their own measurement methods and determines the appropriate content and use of questionnaires |
Questionnaire development
| Measurement 1.0 |
Measurment 2.0 |
| A pool of items are tested in various samples |
A pool of items is tested in various samples |
| Item analysis used to select items for final questionnaire |
Item analysis used to select items for multiple versions of the questionnaires, depending on the needs of the users |
| Questionnaire validated, usually in correlation studies with questionnaires measuring the same construct |
Construct validity determined by factor analysis and comparison of results to known constructs. Various organizations may conduct correlations studies to satisfy internal skeptics. |
| Questionnaire published in final form |
Questionnaires are constantly evolving as data accumulates and measure needs change |
| Manual published |
Online manual constantly updated as data accumulates and needs of users dictate |
| Questionnaire published in peer reviewed journal |
Information on questionnaires shared among community of users and the interested public via a web site |
| Several years pass before a new version is published |
Multiple versions available, with the community of users determining which versions offer the greatest utility |
Normative data
| Measurement 1.0 |
Measurement 2.0 |
| Normative data collected by test developer |
Normative data collected by community of users |
| Test developer may or may not continue to have access to data from wide spread field use of the questionnaire |
Community of users continuously generate data from the field use of the measures |
| Normative information provided in manual and remains static |
Normative information available through online manual and is updated continuously |
| Measurement 1.0 |
Measurement 2.0 |
| Manual may include some information on norms for different categories of patients |
Online manual includes information requested by users |
| Proprietary software products associated with the proprietary questionnaires may include case mix adjustment algorithms |
Case mix adjustment calculations performed dynamically using the General Linear Model and related methods. This is made possible through use of enterprise level data warehousing and analytic software such as products from SAS Institute |
| Case mix adjustment algorithms are static, until next version of software is released |
Algorithms are constantly evolving based on continuous analysis of a growing number of cases data repository |
| Measurement 1.0 |
Measurement 2.0 |
| Questionnaire vendor may offer proprietary benchmarking reports |
Community of users collaborate on development and implementation of benchmarking methodologies |
| Benchmarking methodology (if it exists) tends to be static |
Benchmarking method is dynamic, based on continuous revision to case mix adjustment models as the data dictates |
| Offers benchmarks against reference samples external to the organization |
Offers benchmarking based on both external and internal reference samples, i.e. normative samples generated by the organization or community of users using the questionnaire |
Comments
Primary topics:
You may also view all topics in alphabetical order in
WebTopicList.
ACE
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.