Toolkit usage and improved outcomes

Introduction

We have been able to count the total number of times each user has logged into the Toolkit since May of 2009. As data accumulated, we monitored the correlation between the frequency of Toolkit usage and treatment outcomes, using the Severity Adjusted Effective Size as the outcome metric. The Severity Adjusted Effect Size is calculated for all cases with intake scores in the clinical range and two or more assessments in a treatment episode.

Research evidence suggests that we should find a relationship between the frequency of Toolkit usage and outcomes, namely that more frequent monitoring of data results in better outcomes. However, data in the Toolkit data warehouse is not collected under controlled circumstances, and the probability of clinicians using the Toolkit is influenced by a number of factors.

For example, it seems plausible that clinicians with greater confidence in their outcomes are more likely to log into the Toolkit, and that clinicians with better outcomes may log in more frequently because they like what they see. When we first started to measure usage, this certainly seemed to be the case. The clinicians most likely to log in were those with the best outcomes.

As time goes by, we have been able to also look at changes in outcomes. Which clinicians have improved? Does Toolkit usage correlate with improvement over time?

The following analysis which looks at clinicians' improvement in outcomes between prior years and 2010 suggests this may well be the case.

Description of Sample

A sample of clinicians was selected from those participating the the Regence outcomes informed care program. The mean effect size for all patients treated by clinicians participating in this program has trended upwards for 3 years in a row. The sample of clinicians was selected based on 2 criteria:

• at least 35% of cases with multiple assessments
• at least 15 clinical range cases with change scores in both the baseline period (prior to 2010) and in 2010.

A sample of 87 clinicians meet this criteria. The mean number of cases per clinician was 84 cases during the base line period and 71 cases in 2010.

The clinicians varied considerably in the number of times they logged into the Toolkit, ranging from 0 to 244 logins between May 2009 and December, 2010.

Method

The mean Severity Adjusted Effect Size (SAES) was calculated using hierarchical linear modeling (HLM) is specified in the calculation of the ACORN Criteria of Effectiveness. The use of HLM has the advantage of controlling for differences in sample size.

Since regression to the mean would predict that clinicians with very poor outcomes at baseline would improve over time while those with very good outcomes at baseline would tend to show a decrease. In 2010, we controlled for regression artifacts by using a general linear model to calculate the residuals of the change in the mean SAES for each clinician from the baseline period to 2010. By using the residualized change scores we are able determine if a clinician improved more or less than other clinicians with a comparable SAES during the base line period.

Results

The mean Severity Adjusted Effect Size for these clinicians (averaged at the clinician level) was 0.82 at base line and 0.84 during 2010. Within this group, the average change in effect size between baseline and 2010 ranged from -0.32 to 0.49.

Two variables were found to correlate significantly with improvement in outcomes: improvement in the percentage of cases with multiple assessments and the number of times the clinician logged onto the Toolkit. (p<.01 in both cases). However, the Toolkit login count and the percentage of cases with multiple assessments were not significantly correlated during 2010.

The correlation between login count and increase in effect size was 0.32 (p=.01). After controlling for the percentage of cases with multiple assessments, the number of times the clinician logged onto the Toolkit, remained a significant predictor of improvement (p<.01).

-- JebBrown - 29 Dec 2010

Comments