Expert in computer science drives computer-security spinoff

One illustrious career in computer sciences at the University of Wisconsin–Madison can be traced to an anxious mother, a cocktail party conversation, and a “dead boring” job — plus a fascination with low-level machine code, a subject that many computer scientists disdain.

Professor Thomas Reps recently became the first Rajiv and Ritu Batra Chair in Computer Sciences. The chair is named for Rajiv Batra, a 1983 alumnus who founded Palo Alto Networks, a network security company in California. Reps is also president of the computer security company GrammaTech, where he puts his academic research into action.

But back in 1977, having graduated from Harvard University, Reps was home in Ithaca, New York, looking for something to do — as long as it did not involve grad school.

Reps recalls that his mother attended a cocktail party and talked with an acquaintance who was a professor of computer science at Cornell University. “He asked what I was doing, and she unloaded her frustrations on him. ‘Send Tom up to me,’ the professor said, and a few days later he hired me to fix bugs in a piece of software his group had developed. The first day, I sat down and started to read two big manuals. It was dead boring and I fell asleep.”

Obviously uninspired, he was “passed to an untenured assistant professor who was looking for help.” This was Tim Teitelbaum. “Tim was presumably about to be fired, having conducted precious little research in five years at Cornell, so he didn’t have any students at that time,” explained Reps. “As a consequence, I had a private tutor for a year. We did an incredibly interesting project that year and it completely changed my life.”

At the end of the year, Reps enrolled at Cornell to pursue a master’s degree. “Things went much better than I expected and, relatively quickly, I had a couple of novel results in hand. At that point, Tim had me scrap my planned master’s thesis and turn it into a doctoral thesis.” Reps’ thesis went on to receive the 1983 Doctoral Dissertation Award of the Association for Computing Machinery, the main professional society for computer scientists.

Reps and Teitelbaum, who is now emeritus at Cornell, have been working together ever since. In 1988, they formed GrammaTech Inc., where Teitelbaum is CEO. GrammaTech, headquartered in Ithaca, also has a regional office in Madison. In addition to conducting advanced cybersecurity research, GrammaTech sells software that detects programming errors and security vulnerabilities in other software.

In his academic research, Reps focuses on reducing software’s exposure to hackers. “My research concerns several topics,” he says, “but they all come back to the question of how to find bugs in software.”

The focus of the company is similar. “GrammaTech’s flagship product is a bug-finding tool,” explains Reps. “CodeSonar examines software, looking for evidence of bugs, and produces an ordered report of what bugs possibly exist in the code.”

Many software examiners look at a program’s source code — the code developers write to create new applications. In contrast, Reps concentrates on looking at how the machine code for a program behaves. Machine code is the low-level code in which computer applications are usually distributed. “There are certain things you can see in the machine code that you cannot see in source code — so if you were only to examine the source code, you might say that the program is okay, when in fact it’s not,” he explains.

Software called a compiler translates the source code written by developers to machine code. “The compiler has certain freedoms to make choices,” Reps says. “When you analyze the source code, there can be uncertainties about what machine code will be produced.”

“Attackers are clever,” he continues. “An attack may depend on what sits next to what in memory. Those choices are made by the compiler, and you don’t know what choices will be made if you only analyze source code.”

As the urgency for software security grows, with cyberattacks becoming more frequent, GrammaTech has little trouble convincing its market of the need for security. The company’s biggest problem is finding talented software engineers.

For his part, Reps says, “having a connection to a company that can put my ideas into practice has been a real boon. When I apply for grants to fund my research at the university, the program managers in the federal grant-giving agencies see that I have a real path for technology transfer. Technology that my group has developed has been picked up by GrammaTech and is already in use by defense contractors, medical device manufacturers, automobile manufacturers, and many other kinds of companies. It’s wonderful to know that these ideas are getting out and helping to solve real problems.”