How can we help you?

When an assessment is taken by multiple team members, the presence of a few outliers could impact the overall Agility Index Score of the assessment, which can impact the team’s current state (Pre-Crawl, Crawl, Walk, Run, Fly). In some instances, customers have raised concerns about team assessment results showing the team is in a Run/Fly state while believing the team is actually in a Walk state. 

To combat this situation, we created a feature called High Variance Treatment that is applied to maturity competencies where assessment responses exhibit high variance and a few outliers (this is not applied to performance competencies). This feature adjusts the Agility Index Score and dimension scores using statistical outlier treatments algorithmically as per industry standards. The adjusted score will be automatically reflected across the platform, including APIs, growth journeys, radar results pages, dashboards, benchmarking, and reports.

Watch the video below to learn more:

How does it work?

The High Variance Treatment will be applied for team assessments that have:

• Maturity competencies that are answered by at least 4 team members, the average score is > 50 units and those responses have 50% or higher variance (ie., the difference between the minimum and maximum member score for that competency); and
• After the outlier treatments are applied, there are maturity competencies that still have a score of > 50 units and > 50% variance.

If both of the above criteria have been met, the maturity competency scores will be pulled down by:

• 10 units if the remaining variance is still between 50-60 units.
• 20 units if the remaining variance is still between 60-70 units.
• 30 units if the remaining variance is still above 70 units.

Please refer to the PDF at the bottom of this article for the detailed methodology of how maturity competency scores could be adjusted with the High Variance Treatment. You can also download a PowerPoint presentation about both High Variance Treatment and Unified Calculations.

Where do the changes take place?

These adjusted scores will be automatically reflected across the platform (ie., APIs, growth journeys, radar results pages, dashboards, benchmarking, and reports).

On the team's radar results page for their assessment, the red line displaying the average of the team's responses will be lowered in the competencies that meet the above criteria, the red line will never be raised. The team member's actual responses (the black dots) will remain in place. Outlier responses will not be removed from the radar.

Here are the assessment results before the High Variance Treatment is applied:

Here are the same assessment results after the High Variance Treatment is applied:

Company Admins, Business Line Admins, and Team Admins will see a checkbox allowing them to see the original and adjusted scores. 

The first two boxes in the Analytics section also reflect the adjusted scores. The last two boxes will always reflect the original score. 

All users will see a new Overall Score section below the Analytics section on a team's radar results page displaying the team's updated Agility Index score/state and adjusted score/state in each dimension.

On the Enterprise Agility Dashboard, the team's adjusted score may put the team in a different state. For example, the team's overall score may be adjusted enough to show in the Agility Index widget as in the Walk state instead of the Run state. The Maturity Metrics scores will also be adjusted accordingly and displayed in the appropriate widget.

Additionally, when HVT is enabled for your company, Company Admins can now view the High Variance Treatment Analysis report in the Reports section of our platform—this report includes an analysis of each team's original score and how adjustments were made using HVT.

What methodologies are used?

Z-score and Interquartile Range (IQR) are two statistical methods used for identifying and handling outliers in data. Outliers are data points that differ significantly from other observations, which could be due to variability in the measurement or it could indicate experimental errors. We use both methods to determine if the maturity competency scores meet the criteria for the application of the high-variance treatment.

Z-score
The Z-score, also known as the standard score, measures how many standard deviations an element is from the mean of the dataset. The formula to calculate the Z-score of a value is:

Z=(X−μ)/σ

where:

X is the value,
μ is the mean of the dataset, and
σ is the standard deviation of the dataset.

Values with a Z-score above or below a certain threshold (commonly 2 or 3) are considered outliers. This method assumes that the data follows a normal distribution, and it's particularly sensitive to outliers because the mean and standard deviation are easily influenced by extreme values.

IQR (Interquartile Range)
The IQR is a measure of statistical dispersion or variability based on dividing a data set into quartiles. Quartiles divide a rank-ordered data set into equal parts. The values that divide each part are known as the first quartile (Q1), the second quartile (Q2, or the median), and the third quartile (Q3).

The IQR is calculated as:

IQR=Q3−Q1

Outliers can then be identified by using the IQR. A common rule of thumb is that a data point is considered an outlier if it is:

Below Q1 - 2 x IQR or
Above Q3 + 2 x IQR

This method does not assume a normal distribution of the data and is, therefore, more robust for data with an unknown distribution or when the data is skewed.