2020 Gymnastics Judges Survey Analysis of Judges' Learning Process and Technical Challenges
In the spring of 2020, a gymnastics judges’ survey was conducted to compare judging educational systems and mental processes across disciplines, levels, and nationalities. The survey also included questions regarding sources of pressure, stress, and the judges’ perception of their own judging bias. Given the richness of data from the survey, the analysis of survey results will be broken up into multiple topic-specific reports to allow for more in-depth comparison to past research data. The first report analyzed the demographics of the 853 voluntary participants from 54 countries and is an overview of the usefulness of a variety of judges’ educational tools. This second report builds upon the evaluation of the educational tools mentioned in the first report, focusing on the judges’ learning process and examines specific technical judging challenges.
Judges' Learning Process
How Respondents Were Taught to Judge
The judges’ survey sought to understand the judges’ general learning process and provide a comparison of different educational systems used across disciplines and nationalities. Much of a judge’s education comes from their personal experience in the sport (69.7% of the respondents were former gymnasts) and individual judging practice. Judges frequently learn their craft from other judges (66.4%), however only 22.3% of judges would classify themselves as having learned from a formal mentor, and just over 100 of the respondents (11.8%) reported that they mainly taught themselves how to judge.
Guidance from experienced judges is highly valued and, as demonstrated in the analysis of educational tools in the previous report, is more effective when done in small groups with a higher-level judge helping to guide the learning process. Judges stated that it is “Important to have judging and training sessions with other judges to know you're judging correctly” (Rhy Nat’l), and “having someone there that’s been doing this for longer than I and being with them, in terms of scores, is always reassuring” (WAG Nat’l). However, there were also several comments in which respondents said the mentor relationship was difficult to establish, “Mentors are hard to come by to help with judging” (MAG Comp) or failed to have a positive impact, such as “Current judges give no support, encouragement or resources to new judges. Their attitude is generally that they did it on their own, so you should too” (MAG Nat’l). In some cases, judges cited being paired with higher ranking judges who failed to keep up with changes in rules, needing to stay “more up to date with current skills and how they are taught” (WAG Nat’l).
USA WAG judges are more likely to be taught by other judges or self-taught and less likely to have formal mentors or to attribute their learning to what was taught in classes. International WAG judges, on the other hand, are more likely to garner their knowledge from what was taught in a class environment, suggesting stronger nationally standardized judges’ education systems exist elsewhere in the international judging community. (See Figure 1)
When Judging Skills Were Learned
Gymnastics judging is a complex process (Salmela, 1978). As such, the learning and attainment of judging skills should be acquired in a structured manner to ensure there are not learning gaps. A federation’s judges’ certification structure can provide a framework for the sequence of judges’ education, however, there is great variety in the approaches to entry level certification between nations. In many European countries, novice judging certifications are restricted to performing exclusively E jury duties on a panel, therefore, sequencing the judge to learn to effectively evaluate the quality of execution of an exercise before an understanding of D jury routine construction is necessary. Some nations, such as Great Britain, have additional structures in their certification process restricting which apparatus judges are able to perform certain duties on, with judges gaining experience judging on floor and vault before expanding to other apparatus. USA WAG judges typically spend several years judging at the compulsory level before needing to learn the elements of judging optional routine construction. Such certification and learning structures lower the educational barrier for the recruitment of entry level judges and provide them with an opportunity to acquire live judging experience through meaningful competitive judging evaluations as part of an E jury panel, or in lower level competitions. Without such structure, a judge needs to be proficient in every aspect of judging before being qualified to judge, which can be an overwhelming task and a barrier in the recruitment and training of new judges. Judges’ education curriculums should be thoughtfully structured to align with the certification permissions of their nation. Figure 2 shows when judges as a general population learn various judging skills.
Of the technical judging skills, judges most often first learn to identify gymnastics elements and learn their difficulty values. Most judges world-wide (94%) also learn E jury deductions before or within their first year of judging. However, 97 USA Compulsory and National level WAG judges (38%) reported not learning FIG E jury deductions at all, as they are irrelevant to judging WAG compulsory level routines in the USA which have their own system of deductions.
MAG judges generally learned how to compute a D score earlier than WAG judges (78% MAG before or within their first year of judging compared to 44.6% of WAG judges). USA MAG judges usually learn D score skills earlier than their international counterparts with 43.1% of the USA MAG judges counting it among the very first skills they learned compared to 23.2% of the Int’l MAG population. This correlates to the increased frequency that USA MAG judges must simultaneously D and E compared to the Int’l MAG community, and is amplified by a certification structure which does not allow for E jury only certification.
While MAG judges learned D jury scoring earlier than WAG judges, they tended to learn element shorthand later. Part of the shorthand gap can be explained by the fact that many older judges did not learn element shorthand until later in their careers, as it was not widely used or standardized when they first started judging; and the use of element shorthand was standardized in the WAG community before it was universally required for MAG.
To the researcher’s knowledge, there is no internationally standardized system of execution deduction symbols (MAG or WAG). Most often (63%), judges notate only the amount of E jury deductions (1, 3, 5 etc.) and the shorthand symbol of the skill on which the deduction was taken. Fewer than 30% of judges notate the type of E jury deduction taken, only the amount. This means that there is no way for most judges to account for their E jury deductions the same way they can reconstruct the D jury elements of the routine from their shorthand after the fact. In Dallas and Kirialanis’s (2010) study of Men’s Artistic Gymnastics judges, it was noted that there is no method of deciphering if different judges’ deductions come from the evaluation of the same faults. The development of element shorthand has been indispensable in the process to rectify D score discrepancies and errors. Without a similar system to notate execution deductions, E jury scores will never be able to be justified with the same authority.
Easiest and Hardest Apparatus
Note: Questions pertaining to the easiest and most difficult apparatus were asked of a small percentage of the participants (10.8%, 92 total) These were mostly Regional/Compulsory level judges (51) who are generally earlier in their learning process, and some higher level judges who wrote in alternatives to the provided options for their judging certification level.Due to the few number of respondents for this section (12 MAG, 80 WAG) results may not be generalizable to the overall judging population.
MAG judges generally find parallel bars easiest to judge (45.5%), followed by vault (36.4%) then floor (27.3%). Pommel horse was reported as the most difficult MAG apparatus (60%), followed by high bar (20%). Speed was frequently cited as the critical factor which made an apparatus difficult to judge by novice judges. “Pommel horse - dem circles go by too fast with alot of deductions each circle,” “high bar-too fast” (MAG Comp). Salema’s (1978) and Plessner’s (1999) studies of gymnastics biases both concluded that the speed of performance on an apparatus is associated with an increase in judging errors, noting that pommel horse, high bar and vault were more prone to errors than the slower apparatus of floor, rings and parallel bars.
WAG judges differed in their perceptions of easiest apparatus according to their level and experience. Novice WAG judges tend to find balance beam easiest to judge (See figure 3), and were nearly split between vault and bars as the most difficult (See figure 4). Similar to MAG, speed was a determining factor of judging difficulty for novice judges. Of the 92 judges who were involved with this section of the survey, 41 judges (35 WAG, 6 MAG) mentioned speed when explaining why they selected an apparatus as the hardest one. Of those 35 WAG judges, 54.30% said uneven bars is hardest and 45.70% said vault is hardest. “Vault-goes too quickly and hard to separate height/distance/dynamics deductions” (WAG Comp).
The difficulty from the pace of bars is further compounded by the task of scripting the routine in shorthand while the routine is occurring, making it a challenging apparatus for even more seasoned judges. “In my early years of learning - bars went too fast and skills came up I didn't recognise and while thinking I didn't have time to script and evaluate” (WAG Nat’l). “Bars - everything happens so fast, it is difficult to keep up with all the deductions that are happening as well as writing symbols and deductions down on paper” (WAG Comp).
More experienced WAG judges had wider disparity in their responses. As explained in Ste-Marie’s (1999) study of differences between expert and novice judges, expert judges are more efficient in their analysis of biomechanical information; therefore, speed is not as great of a determining factor of difficulty for more experienced judges, as they have developed increased proficiency in overcoming mental processing limitations. Instead, other nuances of evaluation come into greater play for more experienced judges, such as precise evaluation of angles and the ambiguities surrounding awarding connection. Each apparatus has certain evaluation challenges as captured in these comments by WAG Nat’l judges:
“Beam, because of the connections and specific rules for the side jumps. It also has artistry/composition deductions... too many things to look for at one time… so many ways to judge the same routine very differently from another judge.”
“Floor, it is easy to see the difference between the presentation of gymnasts, but hard to fit it into the categories available”
“I feel like vault is tricky to judge and apply height/length penalties accurately as this can be so dependent on the size of the gymnast.”
Ease and Difficulty of Identifying Twists
Being able to easily identify the number of twists a gymnast has performed is an essential judging skill for MAG and WAG judges across multiple apparatus. A mis-identified twist is an embarrassing error for a judge and is a differentiator between lower and higher-level judges. 678 MAG and WAG judges rated the ease and difficulty they have in identifying twists with 1 indicating “super easy” and 5 indicating “extremely difficult.” Only four judges gave themselves a rating of 5 saying they find identifying twists “extremely difficult”. Unsurprisingly, higher level judges report less difficulty identifying twists. These differences are statistically significant at alpha=0.05. Figure 5 shows the breakdown of how judges characterized their ease or difficulty of identifying twists
In the continued exploration of the benefits for judges to have been former gymnasts (see the Demographics and Educational Tools report in this series), it would appear that the ease of identification of twists carries weight. Judges who competed at higher levels as a gymnast report less difficulty identifying twists. In particular, judges who did not do gymnastics at all report significantly more difficulty than judges who did gymnastics at any level (alpha=0.05). (See figure 6)
However, it isn’t necessarily the ability to perform a twisting element which is the differentiator, but the frequency that one sees elements with twists performed live. It stands to reason that gymnasts who have trained at higher levels have spent more time in the gym seeing other gymnasts perform these elements alongside them at practice. There was a similar correlation of ease of twist identification with judges who have coached (alpha=0.05) (See figure 7), and judges who spend more than 20 hours in the gym per week. (See figure 8)
Judges emphasized in their comments: “Being a coach really helps here” along with watching “podium training at big meets” (MAG FIG) are things which have facilitated their ease in identifying twists. Judging experience alone does not appear to help with identifying twists. The correlation between a respondent’s years of experience and their difficulty identifying twists is -0.05, which not statistically significant (alpha=0.05). The ease of identification is more closely correlated to the factor of time spent seeing twists live in the gym.
Mental Process for Identifying Twists
The majority of MAG and WAG judges report that they count twists in their mind, along with many who reported that they just know. Of the judges who said they find it “super easy” to identify twists (difficulty = 1 of 5), 71.01% said they “just know” how many twists were performed and also used such techniques as: “simultaneously noting the height, time spent in the air and speed of the twist” (WAG Nat’l). Judges who count the number of twists often mentioned that they anchor their focus on one part of the gymnast’s body, such as the head, shoulders, chest, or back, notating the direction they are facing and position of the feet at the start and end of the twists, particularly for “multiple connection situations” (MAG FIG). Judges who are not as confident in their ability to identify twists are likely to rely more heavily on using the gymnast’s ability level, routine composition, and the speed of the twist as guides, as explained by this MAG Nat’l judge: “On floor, I also look at the other passes when I am unsure. i.e., if it could be a double or triple, but they do a triple later, I can 1) compare how it looked in my mind to the other one ("ah yes that was definitely more, so the other one was a double") and 2) know that they likely did not repeat an element.” Methods judges use to identify twists are shown in figure 9.
Time Spent Preparing for Assignments
As we have seen, gymnastics judging requires much autonomous study and preparation. In addition to the general education questions, survey participants were asked: “How much time do you spend preparing for each judging assignment?” Responses are captured in figure 10.
There is not a large difference in time spent preparing between judges who have been judging for more than ten years and those who have been judging for less time. This could be a factor of judges having judging assignments which match their ability level. As mentioned earlier, entry level judges are restricted to judging either certain levels, roles, or judging on limited apparatus. Such structures permit a novice judge to gain practical experience judging competitions without an overwhelming amount of preparation.
Many judges wrote in clarifications that the amount of time they spend preparing varies greatly according to:
Level: “The higher the levels the more I review prior to competition,” “Less time for compulsory, much more time for NCAA and JO optionals”,
Timing of the competition in the season: “Longer hours early season, less mid-season, then more again at the end and post-season”
Level of comfort and experience judging that apparatus: “If I have the same assignments three times in a row it is unlikely that I am going to spend a whole lot of time on # 2 and 3”.
Conclusions and Limitations
An analysis of the mental processes of judges reinforces the observations made in the analysis of the educational tools. Time spent in the gymnastics gym observing live gymnastics has a positive impact on facilitating the ease and proficiency of gymnastics judging. The time in the gym can be a by-product of coaching experience, having been a gymnast, or just a concerted effort by the judge to watch live practices of gymnasts.
Entry-level certification structures, which frame the educational process for judges, vary greatly between nations and disciplines. By examining and considering alternatives, associations can find solutions to address difficulties in recruitment and retention of novice judges. Associations which have a certification level with simpler judging requirements, such as E jury only, ease the amount of knowledge a judge must have mastery over in order to start judging. Of course, this is also dependent on the judging panel structure at competitions. It is useless to have an E jury only certification if judges are always required to perform D and E at competitions. The size and variety of judging panel formats will be examined in a future report along with how it relates to pressures judges.
This was a self-reported survey. As such, the questions regarding a judge’s ability to identify twists was measuring the ease they felt in identifying twists: “How difficult is it for you to identify twists?”, not their perceived accuracy in twist identification. Therefore, it is possible that there are judges who feel as if identifying twists comes easily to them but are inaccurate in their evaluation.
I would like to thank all of the judges who participated in the survey, particularly those who helped with follow-up interviews and the people who gave the survey a wide distribution within their organizations. I would also like to give a special thanks to Brina Seidel at Score for Score for partnering to analyze the data, Danae Rupp for their insight and eye for presentation, and Darby Summers for her editorial assistance.
Address correspondence to Kathi-Sue Rupp. E-mail: FlippedDecisions@gmail.com
Dallas, G., & Kirialanis, P. (2010). Judges' evaluation of routines in men artistic gymnastics. Science of Gymnastics Journal, 2 (2), 49-57.
Plessner, H. (1999). Expectation biases in gymnastics judging. Journal of Sport & Exercise Psychology, 21, 131-144.
Salmela, J. H. (1978). Gymnastic judging: A complex information processing task, or (Who's putting one over on who?). International Gymnast, 20(7,6), 54-56, 62-63.
Ste-Marie, D. M. (1999). Expert-novice differences in gymnastic judging: An information-processing perspective. Applied Cognitive Psychology, 13, 269-281.