Is AI judging the future of gymnastics or just a surveillance tool?

henWhen Simone Biles saluted the judges and stepped onto the mat to vault at the Sportpaleis in Antwerp, Belgium, it seemed like every camera in the packed arena was trained on her. People in the audience pulled their smartphones to record. The photographers zoomed in from their media perches. One TV camera tracked her run on a high-speed dolly, all the way down the runway, as she hurdled into a roundoff onto the springboard. The spider cam, swinging above, caught the upward trajectory of her body as she turned towards the table and blocked up and off, twisting one and a half times before landing on the blue mat and raising her arms above her head. The apex of human athleticism and kinesthetic beauty had been captured.

But there were other cameras that few other people watching in the arena were thinking about as they took in Biles’ prowess on the event: the four placed in each corner of the mat where the vault was situated. These cameras also caught the occasion but not with the purpose of transmitting it to the rest of the world. These were set up by the Japanese technology giant Fujitsu, which, since 2017, has been collaborating with the International Gymnastics Federation (FIG) to create an AI gymnastics judging system. 

In its early days, the system used lidar (light detection and ranging) technology to create 3D composites of gymnasts in action. These days, it uses an even more sophisticated system, drawing from four to eight strategically placed hi-def cameras to capture the movement of the athletes, make 3D models, and identify whether the elements they are performing fall into the parameters established by the judging bodies inside the federation. 

But the computer system doesn’t make judgments itself. Instead, it is deployed when there is an inquiry from the gymnast or coaches or a dispute within the judging panel itself. The Judging Support System (JSS) can be consulted to calculate the difficulty score of an athlete’s exercise — a second opinion, rather than an initial prognosis. Currently, it is mostly used for edge cases.

The JSS wasn’t necessary to evaluate the value of Biles’ vault in Antwerp. Her performance on that vault was too emphatic to be borderline. Still, the cameras positioned at the corners of the vault podium captured her 3D likeness as they did for all of the other athletes who competed through the 2023 World Gymnastics Championships. The technology distilled the legendary athlete and her performance down to straight lines and sharp angles; it showed the distance and height she traveled in numbers. The awe and wonder one feels when watching Biles perform could now be recognized by a computer — understood, though not exactly appreciated.

heThe JSS started, according to Watanabe and Hidenori Fujiwara, as a joke. It was late in 2015, about a year before Watanabe won his first FIG presidential election, making him the first non-European to helm the international federation since its inception in 1881. He suggested that Fujitsu should develop robots to judge gymnastics.

Fujiwara, head of Fujitsu’s sports business development division, took the challenge seriously. “We developed a prototype system,” Fujiwara said, which he then showed to Watanabe, who was surprised by the progress. Watanabe clarified that what he’d said about robots had only been a joke, and yet here they were.

This origin story for the JSS was emphasized during the press conference I attended in Antwerp shortly before the start of the men’s all-around final. There was, of course, a PowerPoint. An early slide in the presentation showed a comic with robots holding up score placards, as a male gymnast swings into a scissor-like movement on the pommel horse. The caption above the image read: “Joke come true!” (I didn’t get why it was funny; I guess you had to be there.) 

It’s a “joke” that Fujitsu has spent untold amounts of money, time, and energy on. Though the company wouldn’t disclose the cost of this whole undertaking, it’s hard to fathom, after strolling through their offices in the annals of the Sportpaleis and seeing the arena setup of the technology in the field of play — and off to the side — that it was anything short of a tremendously expensive and resource-intensive endeavor. But I couldn’t help but feel like it was a lot of effort for technology that, at least as pitched by Watanabe, would only ever amount to a slightly better version of judge-assisted video replay.

Even ignoring the years of investing in R&D, the physical footprint of JSS appears expensive. During the competition, I glimpsed the backroom where there was a row of servers and another of monitors, a cluster of power packs, and tons of cable. Like so much of AI, its “magic” obscures copious amounts of energy-intensive hardware.

Out on the floor, the JSS cameras were subtle, but a lot of human effort went into calibrating them. Before the start of the day’s competition and frequently in between sessions, you could watch as technicians took to the floor, placing large orange balls similar to exercise balls you’d find at the gym, mounted to tripod-like devices, at strategic spots on or near the equipment to make sure that the cameras were properly aligned. Sometimes, they waved these balls like wands around the apparatuses. And throughout the competition, several technicians monitored the event from behind six computer screens near the media box. Nothing about this can be done cheaply.

The entire history of judging had created tragedies, Watanabe explained somewhat dramatically. But even if his remark to Fujiwara had been made in jest, the fact that FIG has doggedly pursued this venture with Fujitsu going on six years suggests that the joke hinted at something critical and true (as jokes often do): that he felt that there was something amiss in judging in the sport of gymnastics, and maybe technology could fix it.

Watanabe didn’t specify any particular instance of judging malfeasance or error that created these personal tragedies. But he didn’t really have to. The conventional wisdom around the judged aesthetic sports, such as gymnastics and figure skating, is that there are and always have been issues with the scoring. During the Cold War, when both the US and the Soviet Union fought for the top spot in the Olympic medal rankings, there was fairly widespread cheating and collusion in gymnastics judging. Back in 1988, after former University of Utah gymnastics head coach Greg Marsden’s brief foray into international elite gymnastics, he let slip to the media that, at the previous year’s world championships, there was judging collusion between the US and Romania, with the coaches exchanging scores before their athletes took to the mat. And in the years since the Cold War ended and the old judging alliances started to break down, the issues became more mundane but no less consequential. It was mostly human error, confusing rules and processes, with a dash of bias — racial, national, or both — that created most of the problems. 

Elements of subjectivity can be found in most sports, and these judgment calls can end up having major consequences when it comes to competitive outcomes. In basketball, for example, a referee might make a bad call that affects the outcome of the entire game, like this year’s Women’s Final Four matchup between UConn and Iowa that featured a controversial offensive foul call in the final seconds of the game. But in general, the way of amassing points is fairly straightforward and has remained consistent over many years. The lines on the court, except in the case of the free throw, determine the point value of any given shot, and this didn’t change when Stephen Curry started nailing deep three-pointers. A shot from well behind the three-point line is objectively more difficult — and impressive — than one made closer to the basket. But the NBA hasn’t painted another line on the court to reward the higher difficulty level of shots taken from well behind the arc. Nor did the league change the rules to make Curry’s threes harder. Players simply learned to shoot from further back.

henWhen I sat down with the Fujitsu technicians in Antwerp in a room somewhere in the bowels of the Sportpaleis, I got to see just how precise the JSS can be. I was shown recordings of the switch ring leap, a skill that was also highlighted during the press conference the day before. This element is notoriously tricky to perform and to judge. The gymnast has a lot of boxes to tick: split of the legs, the position of the back leg relative to the crown of the head (they have to be at roughly the same level), the arch of the back, and the head release. The judge has to be able to register all of that in the split second the skill appears before them on the balance beam. 

JSS looked a lot like video replay, except that the gymnast is transformed into an unclothed mannequin performing the elements. The apparatus is there, but all of the trappings of the gymnasium are gone; the rendering is set against what looks like the holodeck set on Star Trek before the computer program fills in the details, a black space, with white lines running parallel and perpendicular. To the side, you can see key measurements, such as angles, to help determine whether the gymnast met the demands of the element — all of the color and flare stripped out down to the nuts and bolts.   

In the first clip, the gymnast did not fulfill the requirements. At the apex of the leap, her back foot didn’t line up with the crown of her head. The technician applied one tool, a blue horizontal plane, which made it quite clear that her back leg wasn’t high enough. “It’s minus 40 centimeters,” she said, pointing her cursor at the upper right corner of the screen. 

Next, she played a recording of another switch ring, at normal speed. “What do you think?” she asked. I responded that I thought it was performed within acceptable parameters. Turns out I was right. Don’t give me too much credit here, though; the reason I could see it easily is because the gymnast had performed it exceptionally well. Her split was oversplit; her back foot went so high that it was well above the crown of her head.

As much fun as I had playing around with the system — and talking about the finer points of gymnastics with the experts — I wasn’t entirely convinced that the JSS, at its current stage of development, had made a compelling case for its necessity as a decision support system. It felt like a solution in search of a problem.

Steve Butcher, former head of the men’s technical committee and technical coordinator for FIG, said initially he shared my same skepticism. He knows better than most people how hard judging can be, having spent 40 years doing it. But Butcher said he was won over quickly. All it took was a short demonstration showing a gymnast doing an iron cross, a static strength hold, gripping the rings with their arms extended to the sides so they’re completely parallel to the floor. Ideally, the athlete will create a perfectly straight line across, from wrist to wrist.

“They showed me one arm, he has three degrees of deviation. And the other arm, he has one degree of deviation,” Butcher said, noting it was not perceptible by the human eye. Since that demo, he has worked with Fujitsu on behalf of FIG to help the company address the gymnastics needs and has remained a consultant on the project even though he left his full-time position with the gymnastics federation in 2022.

But was this really an improvement over plain ol’ video review? How would seeing the angles of someone’s arms to this degree — a difference of two degrees, to be specific — actually improve the judging? In the example that Butcher cited, the knowledge that the JSS provided was interesting, but it wouldn’t have changed the valuation for the gymnast: he would’ve been credited the skill because he had performed it very close to the platonic ideal. At the top end of the performance, those minute flaws, if they rise to the deductible level, would be sorted out by the execution judges. The JSS isn’t up to that particular task yet.

nIn many industries, AI has been used as an excuse to cut down on labor expenses. That’s not the case here with JSS since its implementation is strictly to support human judges. Besides, judging gymnastics isn’t a full-time career for anyone, not even at the very highest levels, so that particular objection to AI doesn’t play. But the fact that judging gymnastics events is a sporadic activity points to another issue with the JSS’s application: there isn’t a lot of opportunity to use this expensive system. It will judge even less frequently than humans do. The majority of gymnastics events are decidedly low-tech affairs. Not every competition venue will have the necessary infrastructure to support the JSS. And all meets, except the biggest ones, are a couple of days long, if that, hardly worth the time, energy, and costs that go into the setup. Fujitsu said that it took about a dozen people to set up and run the JSS in Antwerp. When asked about the next competition this much-ballyhooed system will be used at, Fujitsu didn’t answer. They said it would be decided jointly by them and FIG. 

Of course, it would be foolish to assume that it will always be this costly or difficult to set up the JSS in a competition format. The technology should improve over time and get cheaper, too. That opens up the possibility for what Butcher believes is its best use case: as a training aid. He told me that this was his first thought when Fujitsu first presented the JSS to him.  

“Somebody’s doing a triple back off the high bar but you can see that their body’s slightly skewed in the air and you can measure that angle, you can see that they [are] landing heavier on one side of their body than the other.” Being slightly off like this in the air doesn’t change the valuation of the skill. It will still be regarded as a triple-back. But in the hands of the athlete and the coach, this kind of information can prevent an almost imperceptible defect from blooming into an injury. In this example, the JSS is merely a sophisticated measuring tool. Butcher said that some national federations have expressed interest in aligning the JSS with their pre-existing video systems, which Fujitsu confirmed, adding that they plan to unveil a version specifically for training in July. Throughout the week in Antwerp, and in follow-up calls with experts, this was the most persuasive use case that I came across. 

Right after the Fujitsu press conference, I encountered Donatella Sacchi, the president of the women’s technical committee, who had been on the panel, along with her counterpart on the men’s side. She’s a compact woman, on the short side — but who isn’t in gymnastics? — with cropped hair, and speaks exuberantly, often standing to make her point and to demonstrate what she means by using her whole body. 

fIf you were in Antwerp at the world championships and wandered into the Fujitsu booth, you’d be forgiven for temporarily forgetting you were at a gymnastics competition. There was very little inside to suggest that you were even at a sporting event of any kind. Monitors were hung on the bare white walls, but they didn’t show videos of gymnasts performing routines or even single elements, overlaid by JSS analysis. Instead, they showed how the technology behind the JSS could be used for fraud and theft prevention.

Though this might come as something of a surprise, it’s not really the left turn that some might imagine it to be. There’s a long tradition of the Games being used as a showcase for new surveillance and security technology. “The Olympics are often used to be kind of a showroom,” Dennis Pauschinger, a researcher at the University of Neuchâtel, told me in 2019 when I was working on a story about the global anti-Olympic movement.

The Fujitsu booth experience began with a simplified version of the JSS that you could play around with. I stood in front of a camera, which projected my movements onto a large screen and labeled them appropriately. It would say which hand you raised and what it was doing. “The judging system is based on what we call ‘pose estimation,’” Mike Fournigault, a Fujitsu AI architect, explained to me. “With cameras, we are able to reconstruct the pose of the body of people and to understand where are the hands, where are the arms, what are they doing with their hands, with their arms, with their legs?”

This is the kind of technology that is used for self-driving cars, with incredibly mixed results. In 2018, Uber’s self-driving car could delineate between a person walking and a person riding a bike but could not reconcile the existence of a 49-year-old woman walking her bike in Tempe, Arizona; the vehicle struck and killed her. At least the stakes for JSS aren’t life and death — though, to the athletes, it can sometimes feel that way.

nOn the morning of the final day of competition in Antwerp, I was allowed to sit in the beam judges seat while the JSS was being calibrated and the arena was being set up for the evening’s competition. The field of play was clean, not yet covered in a white, chalky film, as it would be later when the gymnasts arrived to warm up. Some athletes mark the beam with chalk as a cue for where to start their acrobatic series. All of them douse themselves in the white stuff to mop up sweat on their feet and hands, both of which they need to grip the apparatus. It’s even worse over at the uneven bars where the whole apparatus is covered in the stuff. At a gymnastics meet, magnesium is always in the air.

In person, the beam seems smaller than it does on TV. When you’re watching on television, the camera zooms in on the apparatus and athlete. It’s practically all you see. Live, the equipment and the gymnast are set against the massive arena. You don’t get a sense of that scale on your screen. Still, the action seems more impressive in person, even if everything and everyone appears smaller. The added dimension really makes a difference. And in some cases, so does the massive arena. There are gymnasts out there, like Simone Biles, who, despite her diminutive stature, seem to be able to truly fill the space.

As an exercise, I tried to imagine what it would be like to actually rigorously evaluate a routine, to look at it piece by piece, and find favor or fault with it when medals are on the line. Imagining that burden left me with a queasy anxiety. Years of watching and analyzing the sport, mostly from the comfort of my couch, qualified me to do exactly what I was in Antwerp to do — report on a gymnastics competition — and little more, my success at identifying the credited switch ring notwithstanding. 

“You cannot duplicate [that pressure] when you sit in your chair and in front of you are the best gymnasts, maybe trying to qualify for the Olympic Games,” Sacchi told me. She said that even after all of her years as a judge, she is still nervous before big events. At least the JSS can’t experience anxiety.

I get why, with so much on the line, you’d reach for a technology that promises to overcome human limitations. What the JSS offers is not only the promise of accuracy but also consistency, across rounds of competition, across several days of competition. It will not tire after a 12-hour judging day the way that human judges are wont to do. Gymnasts and coaches don’t like competing in the earliest subdivisions for a reason: the judges are fresh, and their figurative pencils — they actually use tablets — are sharp, and as a result, the execution scores tend to be lower. (The JSS doesn’t yet address the execution score, but I imagine that this is the eventual goal for the technology and would make the system more useful in the long term.)

Some of the hopes that are being pinned on the JSS, such as increased transparency, which Watanabe mentioned in his opening remarks at the conference, seem misplaced. Yes, the JSS can provide a lot of detailed information, but that is not the same thing as transparency. The FBI collects lots of information on US citizens, often through high-tech means, but no one would accuse it of being transparent. (Any journalist that has tried to get info from the FBI knows that it’s actually a black hole.) The fact that the JSS is collecting all this data doesn’t mean it will be shared with the gymnastics community. Ultimately, transparency is not a question of technology but of policy.