Siemens: Virtual Tours of the Colon

Sarnoff: Views of Breast Tumors

NEC’s Pure Research:

Corrections or additions?

Fly on the Research Wall

This article by Barbara Fox was published in U.S. 1 Newspaper on November 18, 1998. All rights reserved. Clustered around a computer terminal, reporters wearing 3-D goggles are agog as they watch an endoscope enter a human trachea for a virtual endoscopy. They have a surgeon's view -- better than a surgeon's view, actually, because the pulsating glistening walls of the trachea have been enlarged -- as the endoscope penetrates deeper into the trachea, down to where the tumor lies.

At Siemens Corporate Research on College Road, scientists are perfecting software for both virtual endoscopies and colonoscopies. Starting with the raw data of two-dimensional images derived from Magnetic Resonance Imaging (MRIs) and Computed Tomography (CTs) this Fly-Through software lets the user "fly through" three-dimensional models of the patient's organs, like a submarine making its way through an ocean of data.

Siemens was one of three firms -- Sarnoff Corporation and NEC Research Institute were the others -- to host some 100 researchers from around the world at the biannual Workshop on Applications of Computer Vision (WACV) sponsored by the IEEE Computer Society last month at the Nassau Inn. Computer vision helps the computer "understand" the content and detect, track, extract, and combine components in the video stream. Computer vision research would seem an obvious match with this area's eminence in the computer/video field, rich with investigators from Thomas Edison to David Sarnoff.

Central New Jersey is so rich in that area that it is sometimes called Video Valley. But a press tour sponsored by the WACV revealed all three firms are channeling some of their imaging efforts into another of New Jersey's technical treasure troves: healthcare research. As they expand the traditional areas of computer technology into the medical arena (known as "bioinformatics") these firms are harnessing computers to improve health care and jumpstart medical research. The tour revealed -- not just three fabulous new technologies -- but also three different business models for the business of research.

Top Of Page
Siemens: Virtual Tours of the Colon

The Siemens software now on the market can be used to train doctors in endoscopic procedures such as taking biopsies of the throat or lungs. (After all, who wants to have the first throat to be invaded by a young doctor?) It can also make a diagnosis more accurate by letting the doctor see whether an endoscopic instrument (a bronchoscope) can actually get close to a tumor, whether a particular make and model can actually fit through a narrow passage without tearing it, or what the actual tumor looks like.

"Using a computer mouse," says Ali Bani-Hashemi, Siemens project manager, "a physician can maneuver the virtual endoscope down the trachea and into the bronchi on the computer screen." At age 44, Bani-Hashemi has his PhD from the University of Michigan at Ann Arbor and has been working at Siemens for 16 years. In the real procedure, a biopsy might be taken through the bronchi and into the lung itself, rather than going through the chest cavity, which is a more invasive and costly procedure.

For some procedures, such as colonoscopies, the software can actually substitute for the real thing, and this is what kindles enthusiasm in Bani-Hashemi. "People refuse colonoscopy screenings because of the discomfort," says Bani-Hashemi. "But I hope our software will be considered as a viable tool so fewer people would have to have the procedure."

Siemens Corporate Research (SCR), one of four R&D centers for the Germany-based firm, works closely with other Siemens companies to develop technology to be used in new products. Sure enough, this software is now on the market, as introduced by Siemens Medical Systems Inc. based in Iselin. It is available now as part of the 3DVirtuoso post-processing workstation and costs $30,000 to $40,000.

SCR was established in the United States in 1977 and moved to the Forrestal Center in 1990. Thomas Grandke is the president, and about 140 people work at 755 College Road East ( (About 50 people work at another Central Jersey outpost, a warehouse for Siemens Medical Systems at Centerpoint at 8A.)

It does research in four of the company's 37 R&D areas: adaptive information and signal processing, software engineering, multimedia/video technology, and imaging and visualization. In addition to enhanced virtual reality, the imaging and visualization scientists work on pattern detection and recognition, knowledge-driven image analysis, image and data fusion, segmentation and modeling, imaging architectures, and intuitive data visualization.

Other companies offer similar virtual tour software, but Siemens' versions have the added bonus of showing three virtual reality viewing modes simultaneously on one screen:

Geometry rendering for viewing the organs from both inside and outside (virtual endocoscopy and colonoscopy) provides Three-dimensional road maps for guiding the virtual instrument to keep it from colliding with and puncturing the fragile organ. Collision detection is done by calculating the forces involved, and the program can determine whether a particular instrument model can fit through a narrow passage.

Multiplanar reconstructions may be viewed in three dimensions or in a separate two-dimensional window. This gives the surgeon real-time access to the data in raw form. For instance, if you are looking for garlic in a round chunk of bologna, you can use your virtual knife freely to get any cross section. You have "six degrees of freedom," says Bani-Hashemi. Slice on any of the three axes or rotate on any of the axes.

Interactive volume rendering lets you look from both a global vantage point (a God's eye view) and an endoscopic view. It mixes volume rendering with geometry rendering to combine virtual reality models with raw data on the same terminal screen. A surgeon can look at the same point on both the model and the raw image. It is the most versatile of the 3-D visualizations because it can show the densities of each organ. You can bring all the organs into view and reveal the densities of each, or you can reduce the distractions by making one or more of them invisible.

An actual colonoscopy is a daunting procedure. It takes two days, the first day for cleansing the bowel (the result of intensive diarrhea) and the second for the procedure, which involves getting sedated and having an intravenous drip inserted. Then the surgeon inserts the long tube for its 30-minute tour of the entire five-foot length of the large colon. (In comparison, a flexible sigmoidoscopy sees only one-third of that length. A barium enema shows the entire colon but reveals only shapes, not details, and it does not require anesthesia.)

Bani-Hashemi says his team needs to improve the virtual colonoscopy so as to "see behind" the sometimes deep folds in the colon. But if a virtual tour of one's colon could possibly substitute for the real thing, many a patient would welcome that. Colon cancer is the second leading cause of cancer death in the United States. With a virtual colonoscopy as a possible test, more patients might submit themselves for early screenings, and more lives would be saved.

Top Of Page
Sarnoff: Views of Breast Tumors

For Sarnoff, using computers for medical purposes is like turning swords into plowshares. Sarnoff is taking its "pyramid vision" technology used only by the military until now, and is turning it into a medical tool that can detect breast cancer.

"It has been an exciting transition from where we were helping the defense establishment, saving lives in that arena, to helping in the health care arena, saving lives of women," says Victor Korsun, vice president of vision technology at Sarnoff. He has an electrical engineering degree from the University of Pennsylvania, Class of 1966, a master's from Penn, and an MBA from Drexel, and he joined the firm 30 years ago, when it was RCA.

As RCA Laboratories, Sarnoff was and is known for its television and telecommunications work. Now, with 850 employees on Fisher Place, it is a technical think tank, spun off as a not-for-profit research center by its parent, SRI. Fortune Magazine ranks it with the great labs of the nation: IBM's Watson Lab, Xerox PARC, and Bell Labs/Lucent. Its business plan is to put its valuable talent to work to develop products and to spin off at least two high-tech companies annually: Sarnoff Real Time, Delsys, Orchid Biocomputers, and Sarif are among the successful spinoffs so far.

Detecting change is the groundwork for Sarnoff's special "pyramid vision" technology, and this "automatic target recognition" involves multiple pictures of the same area taken over time. The changes could be the introduction of more planes to an airfield or the appearance of a suspicious lesion in the breast.

Using artificial intelligence Sarnoff's neural nets train algorithms to look for suspicious areas in the context of the whole picture. SCUD missiles, for instance, must travel on a hard surface and can't be hidden in swamps or in desert sand. First it trains the computer to align the images correctly. Then the pyramid vision quickly zeros in on roads, then human analysts study those areas more carefully.

"We applied the neural net approach to mammograms," says Korsun. "You want to find the suspicious micro-calcifications when they are too small to see, so you look for context -- what are the arteries that begin to feed that lesion in an early growth stage? Then you do further analysis to refine the search."

Two heads are better than one, when it comes to reading mammograms, but even though two readings improve accuracy by 15 to 20 percent, today's HMOs don't like to pay for two. Sarnoff's software could be an economical substitute for the first doctor's reading.

Sarnoff did real-time tests of its neural networks at the University of Chicago. Then it went to the Hospital of the University of Pennsylvania's "outcomes archive" of 900 breast images to put the algorithm through its paces. The next step is clinical testing, and Sarnoff hopes the Food and Drug Administration will count the archive testing as sufficient for this phase.

A second way to use Sarnoff's imaging technology is to substitute MRIs for mammograms or for some biopsies. A $1,000 MRI is 10 times as costly as a $100 mammogram but it can forestall a biopsy. By using data from an MRI and supplying special glasses to the radiologist, Sarnoff can create a three-dimensional picture of the lesion that more accurately shows the shape.

"If it is smooth, it is likely benign, but if gestipulated or jagged, it is likely cancerous," says Korsun. Of 500,000 tumors biopsied every year, says Korsun, three-fourths of them are benign. If MRI images and Sarnoff's vision technology were used as the initial screening there would be fewer false positives, and this method would certainly save money as the preliminary step to a $3,000 biopsy, he believes.

Sarnoff has not spun off its cancer detection software into a young company. It is not even in the seed company stage. It is just a variant of the technology known as pyramid processing. But within a two year window, if the FDA accepts the archival nature of Penn's clinical tests, Korsun believes this technology could go to commercialization and clinical use.

Apart from the software, a new hardware product has come out of this research: high end video terminals. When Sarnoff was trying to convert hard copy photos to digital "soft copy" photos on contract for the National Information Display Laboratory, it needed to supply super quality 5 million pixel (five-megapixel) display terminals to 1,000 image analyzers around the world. Conventional medical terminals, configured with 1,000 by 1,000 pixels, were inadequate. Even HDTV monitors are only 2,000 by 1,000 pixels (2 megapixels). Few manufacturers would set up an assembly line for just 1,000 terminals.

By transferring the technology to breast cancer detection, the market for these high-end terminals expanded. Now six firms are making the five-megapixel terminal for radiologists. Korsun takes pride in having given United States-based firms a head start on the development of these terminals. At the biggest radiology convention, he relates, four companies that Sarnoff had worked with came to the show with five-megapixel monitors. Two companies from Europe, Siemens and Barco, did not have five megapixel displays.

"Twelve months later, all six of them came with five-megapixel monitors. What that showed is that the companies that worked with us had a 12-month lead on the rest of the world in the niche of the market," says Korsun.

Korsun also takes pleasure in the notion that the whole thing might have started with a directive that could have come from the First Lady. "Three or four years ago a government sponsor sent a question to us that had been sent to many different agencies: `Are there any programs you are doing that could be applicable in the fight against breast cancer?'" He looked at 36 different projects and suggested 30 that might be applicable.

"A month later we got a call from Health and Human Services. Susan Blumenthal, director in charge of the office on women's health, came to visit with a dozen other medical doctors, half in the government, half from leading medical centers around the country. They were astounded at what they saw. Immediately they started having us work with them on three specific projects, all directed toward breast cancer."

"We are not sure where the question originated," says Korsun, "but we believe it originated from the White House, from Hillary."

Top Of Page
NEC's Pure Research: Study Flies Now, Use It Later

The live fly's head, projected on a big screen, looks like a scene from a Hitchcock movie. With its proboscis it sips sugar water. A very tiny needle connected to a single cell in its brain beeps electrical responses in a randomly laconic pattern. But when a researcher waves his hand over the fly, the audible pulses intensify so they sound like hail on a roof.

NEC scientist Rob de Ruyter van Steveninck is studying how the blowfly or housefly (Calliphora erythrocephala) manages to evade our fly swatters. It is, after all, an amazingly skilled air acrobat. When a male fly chases a female he responds to her turns within 30 milliseconds. Such skill is partly due to the fly's large compound eye, a 5,000 pixel array of photoreceptor cells (count 'em, 5,000) that converts light changes into electrical responses. And partly due to the fly's brain, which separates out which light changes are caused by movement and sends choreographic signals to the wing muscles.

"For me, the excitement is trying to understand the language of the brain. I don't really care that it is a simple brain, as long as it is a brain that does a good job," says de Ruyter. "Tapping into the signals and listening, and getting a grasp of what those signals mean, is a bit like deciphering the Rosetta Stone."

Unlike his cohorts at Siemens Corporate Research and Sarnoff, Rob de Ruyter is not working towards a saleable product. Someday down the road his study of neural information processing might be useful in improving robot vision for territories where humans dare not go, such as undersea, in space, or in a nuclear reactor. But in the meantime, he is doing basic research, research for its own sake, a luxury usually restricted to universities.

NEC Research Institute ( is one of the very few corporate laboratories devoted to basic research. "The senior people at NEC understand that in the future they will be dependent on new ideas to grow new business," says C. William Gear, president of the Institute, "that you can support business with current technology for just so long. They also understand that to get new creative ideas you have to provide a lot of individual freedom to the scientists, so they can follow their noses." Even the corporate labs -- Xerox Park in California, IBM's Yorktown Heights in New York, and Bell Labs -- that formerly did just basic research have had to be more focused, says Gear.

At 4 Independence Way the Japanese firm set up three laboratories: one for applied computers and communications research, one for software development, and the third, the institute for basic research. Nine years later the institute has 80 scientists and 30 support staff. "We publish everything that we do," says Gear, a British native who graduated from Cambridge in 1956, worked for IBM, and taught for 28 years at the University of Illinois, where he headed the computer science department. "We compete for the very best scientists with universities and try to provide an atmosphere that is at least as good as the university one."

Rob de Ruyter is a native of the Netherlands, where his father was an aeronautical engineer. He majored in physics at the University of Groningen, Class of 1981, and earned his PhD there, joining NEC's bioinformatics group in 1992. He consults with William Bialek, a theorist, and works with an associate, Geoff Lewen, plus assorted postdocs, students, and visiting collaborators. "All the people in the bioinformatics group here are essentially physicists," says de Ruyter. Others work on cockroaches, locusts, and dragonflies; like the flies they are standard research subjects.

But working with flies or any insect requires a certain gumption. In the summer, you have to catch them and store them, and in the winter, you have to breed their eggs on raw liver and let it sit for two weeks. "You can imagine what the smell is," says de Ruyter ruefully. "We have it on the ventilator, otherwise it is unbearable."

-- Barbara Fox

Next Story

Corrections or additions?

This page is published by -- the web site for U.S. 1 Newspaper in Princeton, New Jersey.

Facebook Comments