How can we help you?

When an assessment is taken by multiple team members, the presence of a few outliers could impact the overall Agility Index Score of the assessment, which can impact the team’s current state (Pre-Crawl, Crawl, Walk, Run, Fly). In some instances, customers have raised concerns about team assessment results showing the team is in a Run/Fly state while believing the team is actually in a Walk state. 

To combat this situation, we created a feature called High Variance Treatment that is applied to maturity competencies where assessment responses exhibit high variance and a few outliers (this is not applied to performance competencies). This feature adjusts the Agility Index Score and dimension scores using statistical outlier treatments algorithmically as per industry standards. The adjusted score will be automatically reflected across the platform, including APIs, growth journeys, radar results pages, dashboards, benchmarking, and reports.

Note: High Variance Treatment (HVT) can be applied to the TeamHealth 4.0 Assessment. Other assessments may be configured to support HVT in the future.

Watch the video below to learn more:

How does it work?

The High Variance Treatment will be applied for team assessments in the following steps:

1. Outlier Treatment:

   • Upper outliers are removed across maturity competencies using the Interquartile Range (IQR) method.

2. Statistical Adjustment:

   • Maturity competency scores will be adjusted if the Coefficient of Variation exceeds 30%, using a reduction factor based on user settings:

     • 0.50 (Optimal Reduction Factor: Higher the reduction factor, greater the adjustment)

     • 0.75

     • 1.00

   Note: Coefficient of Variation measures the distance between the mean (average of member scores) and each individual member score.

Please refer to the PDF at the bottom of this article for the detailed methodology of how maturity competency scores could be adjusted with the High Variance Treatment. You can also download a PowerPoint presentation about both High Variance Treatment and Unified Calculations.

Where do the changes take place?

These adjusted scores will be automatically reflected across the platform (ie., APIs, growth journeys, radar results pages, dashboards, benchmarking, and reports).

On the team's radar results page for their assessment, the red line displaying the average of the team's responses will be lowered in the competencies that meet the above criteria, the red line will never be raised. The team member's actual responses (the black dots) will remain in place. Outlier responses will not be removed from the radar.

Here are the assessment results before the High Variance Treatment is applied:

Here are the same assessment results after the High Variance Treatment is applied:

Company Admins, Business Line Admins, and Team Admins will see a checkbox allowing them to see the original and adjusted scores. 

The first two boxes in the Analytics section also reflect the adjusted scores. The last two boxes will always reflect the original score. 

All users will see a new Overall Score section below the Analytics section on a team's radar results page displaying the team's updated Agility Index score/state and adjusted score/state in each dimension.

On the Enterprise Agility Dashboard, the team's adjusted score may put the team in a different state. For example, the team's overall score may be adjusted enough to show in the Agility Index widget as in the Walk state instead of the Run state. The Maturity Metrics scores will also be adjusted accordingly and displayed in the appropriate widget.

Additionally, when HVT is enabled for your company, Company Admins can now view the High Variance Treatment Analysis report in the Reports section of our platform—this report includes an analysis of each team's original score and how adjustments were made using HVT.

What methodologies are used?

Z-score and Interquartile Range (IQR) are two statistical methods used for identifying and handling outliers in data. Outliers are data points that differ significantly from other observations, which could be due to variability in the measurement or it could indicate experimental errors. We use both methods to determine if the maturity competency scores meet the criteria for the application of the high-variance treatment.

Z-score
The Z-score, also known as the standard score, measures how many standard deviations an element is from the mean of the dataset. The formula to calculate the Z-score of a value is:

Z=(X−μ)/σ

where:

X is the value,
μ is the mean of the dataset, and
σ is the standard deviation of the dataset.

Values with a Z-score above or below a certain threshold (commonly 2 or 3) are considered outliers. This method assumes that the data follows a normal distribution, and it's particularly sensitive to outliers because the mean and standard deviation are easily influenced by extreme values.

IQR (Interquartile Range)
The IQR is a measure of statistical dispersion or variability based on dividing a data set into quartiles. Quartiles divide a rank-ordered data set into equal parts. The values that divide each part are known as the first quartile (Q1), the second quartile (Q2, or the median), and the third quartile (Q3).

The IQR is calculated as:

IQR=Q3−Q1

Outliers can then be identified by using the IQR. A common rule of thumb is that a data point is considered an outlier if it is:

Below Q1 - 2 x IQR or
Above Q3 + 2 x IQR

This method does not assume a normal distribution of the data and is, therefore, more robust for data with an unknown distribution or when the data is skewed.