Skip to main content
SHARE
Publication

Machine Learning in the Big Data Era: Are We There Yet?...

by Sreenivas R Sukumar
Publication Type
Conference Paper
Publication Date
Conference Name
ACM SIGKDD Conference on Knowledge Discovery and Data Mining: Workshop on Data Science for Social Good
Conference Location
New York City, New York, United States of America
Conference Date
-

In this paper, we discuss the machine learning challenges of the Big Data era. We observe that recent innovations in being able to collect, access, organize, integrate, and query massive amounts of data from a wide variety of data sources have brought statistical machine learning under more scrutiny and evaluation for gleaning insights from the data than ever before. In that context, we pose and debate the question - Are machine learning algorithms scaling with the ability to store and compute? If yes, how? If not, why not?
We survey recent developments in the state-of-the-art to discuss emerging and outstanding challenges in the design and implementation of machine learning algorithms at scale. We leverage experience from real-world Big Data knowledge discovery projects across domains of national security and healthcare to suggest our efforts be focused along the following axes: (i) the ‘data science’ challenge - designing scalable and flexible computational architectures for machine learning (beyond just data-retrieval); (ii) the ‘science of data’ challenge – the ability to understand characteristics of data before applying machine learning algorithms and tools; and (iii) the ‘scalable predictive functions’ challenge – the ability to construct, learn and infer with increasing sample size, dimensionality, and categories of labels. We conclude with a discussion of opportunities and directions for future research.