Methods to search tens of thousands of documents could become more effective with a system developed by a team led by Cathy Jiao of ORNL's Computational Sciences and Engineering Division. The method, called dynamic dimensionality reduction, helps search engines perform their jobs much more efficiently by reducing the original amount of information, making what remains manageable. "Conventional methods to perform dimensionality reduction typically make tradeoffs that ultimately sacrifice the effectiveness when dealing with changing data streams and the high dimensionality of data," Jiao said. Each unique word in a document is called a dimension. Unlike conventional approaches to dimensionality reduction, ORNL's proprietary method is designed to handle dynamically changing data. The system stores a small fixed amount of information and reduces the dimensionality of data in real time. This avoids the problem of too much stored information, which over time can crash systems. This project, designed to identify threats to the U.S., is funded by the Department of Homeland Security.
Menu