Predictive Validity Estimations for the JobMatchTalent Instrument
Bengt Jansson1,2, Klaus Olsen2, and Trevor Archer1,2*
1Department of Psychology, University of Gothenburg, Sweden
2Job Match Talent, Job Match Sweden AB, Sweden
Submission: January 28, 2020; Published: February 18, 2020
*Corresponding author: Trevor Archer, Department of Psychology, University of Gothenburg, S-40530, Job Match Talent, Job Match Sweden AB, Skårs Led 3, 40020, Gothenburg, Sweden
How to cite this article:Bengt Jansson, Klaus Olsen, Trevor Archer. Predictive Validity Estimations for the JobMatchTalent Instrument. Psychol Behav Sci Int J. 2020; 14(3): 555889. DOI:10.19080/PBSIJ.2020.14.555889.
Abstract
Predictive validity was investigated through executive self-reports of work performance and type of work carried out by employees, 258 of these assessed by 94 executives who were recruited using the JobMatch Talent (JMT) recruitment instrument. Three dimensions (criteria) formed the basis of these estimations: (i) Productivity and Motivation, (ii) Quality and Work Structure, and (iii) Cooperativeness. Ten main scales of JMT were applied as predictors of the criteria. Pairwise correlational analysis (Pearson) provided the basis for predictive validity. The 30 possible correlations were assigned a priori according to theoretical formulations expected (middle or higher) and non-expected (zero or weak), respectively, association. The combined results of both studies provided marked support for the assumed association. Predictive validity for the JMT-test instrument was adjudged to be good, particularly in view of being entirely comparable with published meta-analyses wherein associations of almost 0.40 were considered highly impressive. In future, it is likely that those estimations of predictive validity from JMT, involving larger, more accumulated samples for these three as well as other arising categories, are expected to provide greater levels of association than those reported here.
Keywords: Predictive validity; Job Match Talent; Work performance; Recruitment, Correlations, Confidence intervals; Behavioural repertoire; Regression analyses.
Introduction
Criterion or ‘concrete’ validity expresses the extent to which a measurable entity may be related to an outcome (or result). For example, in a study of potential benefits arising from application of a simple, valid tool derived for assessing broadly the corporative positive wellness climate, particularly due to climate contribution to employee wellbeing, it was shown that a confirmatory factor analysis indicated that a 9-item measure has good model fit (RMSEA=0.06, CFI=0.91), with an inter-item consistency of 0.74, and a mean Rwg(j) of 0.87, with the new instrument demonstrating a significantly positively correlation with physical health and wellbeing, and a negative correlation with substance use behavior [1], implying that the 9-item instrument possessed good reliability, construct, and criterion validity. Although concurrent and predictive criterion validity share similarities, the former refers to a comparison between the measurement that is under observation and an outcome that is estimated concurrently. The latter is assessed typically through comparison with a gold standard test.
In a replicative study based upon executives’ estimations of work performance at recruitment, using the Job Match Talent (JMT) database [2,3] administrators’ estimations of employees’ performance and ‘work-ability’, as assessed by the JMT recruitment instrument [3]. The study involved 258 recruited employees, who had responded to the JMT instrument, consisting of three main scales wherein each scale was comprised of three sub-scales, at recruitment and 94 executive-administrations. As the basis for these estimations, three dimensions (or criteria) have laid the foundation: (i) “productivity and motivation”, (ii) “quality and work-structure”, and (iii) “cooperativeness”. The formulations of the validation questions resemble those instruments utilised in other studies examining predictive criterion validity, Spector (2003); Viswesvaran et al. (1996). Thus, the purpose of the present study was to estimate the predictive validity of the JMT recruitment instrument through applications of pairwise correlational analyses based upon the above three dimensions.
Methods and Materials
Predictive validity, based on uni-variate pairwise correlations (r), was compared to multi-variate relationships. The following three predictive criteria were included: Productivity, Quality of work, and Ability to cooperate.
Expected convergent and divergent correlations
![Click here to view Large Table 1](images/PBSIJ.MS.ID.555889.T001.png)
An acceptable coefficient for convergent validity was set to 0.30 (Pearson r) as a ‘rule of thumb’ criterion. According to Cohen's guidelines, this value corresponds to a medium correlation, Cohen (1988). Criteria for the analogue divergent coefficient was either an uncorrelated relationship (r <.10), or a weak correlation (.10 r <.30) , Cohen (1988). The expected relationships among predictors and criteria are shown in Table 1 below.
Results
Predictive validity, based on uni-variate pairwise correlations (r), was compared to multi-variate relationships. The following three predictive criteria were included: Productivity, Quality-of-work, and Ability-to-cooperate. In both studies, the results were mainly in accordance with expected relationships. For the 18 divergent relationships (‘non-correlated', or 'weakly correlated'), there were no correlations were higher than the expected values (.30).
For the first study (N=258), the mean value for convergent correlations was 0.37, and –0.071 for divergent. See further details on Table 2 below.
![Click here to view Large Table 2](images/PBSIJ.MS.ID.555889.T002.png)
Note: Convergent correlations are marked (yellow). The mean value was 0.37 for convergent correlations, and –0.071 for divergent.
![Click here to view Large Table 3](images/PBSIJ.MS.ID.555889.T003.png)
Note: Convergent correlations are marked (values =0.30 are yellow, and values <0.30 are light blue). The mean value was 0.40 for convergent correlations, and –0.062 for divergent.*p<.05, **p<.01
For the follow-up study carried out about four years after the first, the mean value for convergent correlations was 0.40, and –0.062 for divergent. See Table 3 below, for details.
The difference between convergent and divergent correlations was also tested for statistical significance by use of confidence intervals (CI). For both studies, this difference was significant (i.e., the 95% CI was >0). See Table 4 below.
![Click here to view Large Table 4](images/PBSIJ.MS.ID.555889.T004.png)
Notations CV: Convergent validity; DV: Divergent validity; CV_DV: Mean differences between CV and DV (here expressed as absolute value); [#.##]: Denotes a 95% asymmetrical confidence interval (CI) based on Fishers’ Z-transformation of a Pearson correlation.
Discussion
Despite the restrictiveness intrinsic to the test model, implying that only the main scale could influence criterion measures, the applications of independent pairwise associations in the present study was shown to possess utility for the assessment of criterion evaluation among the main scales of JobMatchTalent (JMT), implying that predictive validity was observed markedly. The purpose of deriving optimal predictability of the criterion measures provided a multivariate approach through which a result was obtained that was associated with the theoretical implications obtained from the JMT manual as well as the relevant scientific notions of psychometry in recruitment. The previously arranged associations between the criterion measures and predictors (JMT main scales) were confirmed on the grounds of the generally significant correlations as well as the situation that the relatively high levels for the median value derived for executives (r = 0.48) exceeded the estimated value pertaining to an explained variance of 15% (r = 0.39), which has been reported as an expected highest value from a meta-analysis of criterion measures and personality psychometry [4,5]. Furthermore, these analyses provided convergent and divergent conditions, respectively, thereby implying that the expected association gave broad support to the analyses and median values. The exploratory factor analysis exploratory study confirmed the present findings comprehensively despite possessing a lesser extent of predictive power thereby enhancing markedly the ‘combined’ predictive validity of the JMT-test instrument.
The predictor, “Stress Index” (C) showed throughout low levels of association over the personality categories and was excluded but nevertheless discussed here fleetingly. The finding that this main scale failed to be predictive of the criterion measure “Ability to Co-operate” may be open to several interpretations. For example, stress levels experienced by an employee may be exceedingly difficult to assess since the myriad of stressors affecting individuals induce, not only psychological and behavioural reactions but biological, physiological and epigenetic reactions as well [6-9]. These types of ‘internal’ and ‘extreme’, respectively, responses/reflexes present aspects of individuals’ behavioural repertoire that one attempts, with greater or lesser levels of success, to mask, strategically-tactically, at the place of work. Thus, as to be expected from these notions, the ‘true’ estimation of an individual’s stress level or expression becomes more apparent than real. Even applying the focus upon analyses differentiating executives from non-executives, there emerged interesting aspects of the results pertaining to similarity and stability from several smaller personal categories, such as specialists, administrators and HR-orientated. It appears likely that future estimations of predictive validity from JMT, involving larger, more accumulated samples for these three as well as other arising categories, are expected to provide greater levels of association than those reported here.
Conclusion
High levels of pairwise correlations indicate sufficient, indeed impressive, extents of predictive validity inherent to the JMT instrument.
Limitations
It is possible that another analysis employing regression analyses may offer further evidence of JMT predictive validity.
References
- Reynolds GS, Bennett JB (2019) A Brief Measure of Organizational Wellness Climate: Initial Validation and Focus on Small Businesses and Substance Misuse. J Occup Environ Med 2.
- Archer T, Jansson B, Olsen K (2015) Effects of age upon leadership attributes from recruiment instrument: a selective developmental trajectory. Clinical and Experimental Psychology 1:1.
- Jansson B, Olsen K (2018) En replikativ studie av prediktiv kriterievaliditet hos Job Match Talent genom chefsskattningar. [A replicative Study of predictive criterion validity of the Job Match Talent instrument through Executive Estimations]. Unpublished Manuscript.
- Humphrey SE, Nahrgang JD, Morgeson FP (2007) Integrating motivational, social, and contextual work design features: a meta-analytic summary and theoretical extension of the work design literature. J Appl Psychol 92(5):1332-1356.
- Nahrgang JD, Morgeson FP, Hofmann DA (2011) Safety at work: a meta-analytic investigation of the link between job demands, job resources, burnout, engagement, and safety outcomes. J Appl Psychol 96(1): 71-94.
- Cassidy-Bushrow AE, Sitarik AR, Johnson-Hooper TM, Phillips JM, Jones K, et al. (2019) Prenatal pet keeping and caregiver-reported attention deficit hyperactivity disorder through preadolescence in a United States birth cohort. BMC Pediatr 19(1): 390.
- Van Bortel T, Martin S, Anjara S, Nellums LB (2019) Perceived stressors and coping mechanisms of female migrant domestic workers in Singapore. PLoS One 14(3): e0210717.
- Zhou L, Yang K, Wang Z, Luo Z (2019) When Do Employees Speak Up Under Job Stressors? Exploring the Potential U-Shaped Relationship Between Hindrance Stressors and Voice Behavior. Front Psychol 10: 2336.
- Jansson B, Winge H (2013) Prediktiv kriterievaliditet hos JobMatchTalent genom chefsskattningar: En korrelationsbaserad ansats [Predictive criterion validity derived from executive estimations: a correlation-based analysis]. Unpublished manuscript.