Krishnaram Kenthapadi's Homepage

Krishnaram Kenthapadi

Dr. Krishnaram Kenthapadi has over a 20-year record of advancing new research areas, identifying early trends in AI/ML & influencing thought leadership in industry, shaping the technical vision, development & launch of new AI products, and steering company-wide initiatives in new domains. As the Chief Scientist, Clinical AI at Oracle Health, he leads the AI initiatives for Clinical Digital Assistant and other Oracle Health products. Previously, as the Chief AI Officer & Chief Scientist of Fiddler AI, he led initiatives on generative AI (e.g., Fiddler Auditor, an open-source library for evaluating & red-teaming LLMs before deployment; AI safety, observability & feedback mechanisms for LLMs in production), and on AI safety, alignment, observability, and trustworthiness, as well as the technical strategy, customer-driven innovation, and thought leadership for Fiddler. Prior to that, he was a Principal Scientist at Amazon AWS AI, where he led the fairness, explainability, privacy, and model understanding initiatives in Amazon AWS AI platform, and shaped new initiatives such as Amazon SageMaker Clarify from inception to launch. He was previously part of the LinkedIn AI team, where he led one of the early large-scale responsible AI deployments in industry (fairness-aware LinkedIn Talent Search), incubated responsible AI initiatives across different LinkedIn applications, and served as LinkedIn's representative in Microsoft's AI and Ethics in Engineering and Research (AETHER) Advisory Board. He shaped the technical roadmap and led the privacy/modeling efforts for LinkedIn Salary product, and prior to that, served as the relevance lead for the LinkedIn Careers and Talent Solutions Relevance team, which powers search/recommendation products at the intersection of members, recruiters, and career opportunities. Previously, he was a Researcher at Microsoft Research Silicon Valley, where his work resulted in product impact (and Gold Star / Technology Transfer awards), and several publications/patents. He received his Ph.D. in Computer Science from Stanford University in 2006, advised by Professor Rajeev Motwani. Before joining Stanford, he received his Bachelors degree in Computer Science and Engineering from Indian Institute of Technology-Madras.

Krishnaram has advanced the state-of-the-art in areas such as fairness, explainability, privacy, and robustness in ML, and helped start new research areas (e.g., co-authoring the second paper on Differential Privacy). Leveraging his expertise in research and industry, he has steered company-wide initiatives on responsible AI and AI observability, led the technical roadmap/design/launch of new AI products, and improved business metrics for existing products via technology transfers. He serves regularly on the senior program committees of conferences such as KDD, FAccT, and WWW, and co-chaired the 2014 ACM Symposium on Computing for Development. His work has been recognized through awards at NAACL, WWW, SODA, CIKM, ICML AutoML workshop, and Microsoft's AI/ML conference (MLADS). He has published 60+ papers, with 7500+ citations and filed 150+ patents (71 granted). He has given several invited industry talks, presented tutorials on trustworthy generative AI, privacy, fairness, explainable AI, ML model monitoring, and responsible AI at forums such as ICML, KDD, WSDM, WWW, FAccT, and AAAI, and instructed a course on responsible AI at Stanford, thereby influencing industry thinking, thought leadership, and practice in his areas of work.

Research

Trustworthy Generative AI, Trustworthy Health AI, AI Observability, AI Safety, Fairness/Explainability/Privacy/Robustness in AI/ML Systems, Algorithms for Large Datasets and Graphs, Data Mining, Web Search, Information Retrieval, Search and Recommendation Systems, Social Network Analysis, Computational Education.

Publications [Google Scholar] [Patents (71 Granted)] [Selected Projects]

Contact

Email:

kkenthapadi [at] gmail.com [LinkedIn] [Twitter]

Selected Initiatives:

Technical Strategy and Scientific Roadmap at Fiddler AI:
- Led the AI org with responsibility for customer-driven scientific & product innovation, customer success, and thought leadership at Fiddler AI.
- Drove technical strategy and initiatives on trustworthy generative AI, AI safety, ML model monitoring, explainability, fairness, model understanding, MLOps, and AI/ML governance at Fiddler AI.
- Led scientific investigations on topics such as robustness, security, and safety mechanisms for large language models (LLMs), ML model monitoring, drift detection for NLP models (using LLM-based embeddings), drift detection for CV models, and explanation based membership inference attacks.
- Delivered invited talks at the Knowledge-First World Symposium 2023 (slides, video, Tweet thread), the AI Conference (video), CMU Privacy & AI Governance Seminar, MLconf Online 2023, ODSC West 2023, O'Reilly Enterprise LLMs Conference 2023, UC Berkeley Center for Long-Term Cybersecurity & Algorithmic Fairness and Opacity Group Seminar, the Edge AI & Vision Alliance Embedded Vision Summit, and Qualcomm Ventures AI Startup Event.
- Presented as part of podcasts (such as EqualAI's In AI We Trust podcast, MLOps Community Podcast, and Your Bytes, Your Rights: Biased Machines & Prejudiced Programming), webinars (such as Insight Rise & IGNITE webinar on responsible AI, MLOps Community Coffee Session on Responsible AI and Model Monitoring in Practice, and AI Explained: Model Monitoring Best Practices), and panels (such as ACM SIGKDD Generative AI in India panel, FinRegLab's Virtual Conference on Machine Learning Explained: Evidence on the Explainability and Fairness of ML Credit Models and Charter for Change panel discussion on Social, Economic, and Ethical Implications of AI).
- Instructed a course on responsible AI in practice at Stanford (slides).
- Invited to serve as an honorary research fellow at the Centre for Responsible AI (CeRAI), Indian Institute of Technology (IIT) Madras.
- Presented tutorials on trustworthy generative AI (slides) and ML model monitoring in practice (slides) at FAccT '22 '23, KDD '22 '23, WWW '23, and ICML '23, combining experiences across Amazon, Fiddler, Hugging Face & Microsoft.
Fairness, Explainability, Privacy, and Model Understanding Initiatives in Amazon AWS AI Platform: While at Amazon, Krishnaram led the fairness, explainability, privacy, and model understanding initiatives in Amazon AWS AI platform, and led the science roadmap for Cloud, Edge, and Responsible AI in AWS ML Services.
- Shaped the technical roadmap for Amazon SageMaker Clarify, a new capability of Amazon SageMaker that helps customers detect bias in machine learning (ML) models, and increase transparency by helping explain model behavior to stakeholders and customers (launched in December 2020; AWS Blog; Amazon Science Article [Amazon AI Fairness and Explainability Whitepaper] [Amazon SageMaker Clarify KDD'21 Paper] [Fairness Measures for Machine Learning Paper] [Overview Slides]) and guided the roadmap for products such as Amazon SageMaker Clarify and Amazon SageMaker Debugger.
- Led scientific investigations on topics such as fairness and explainability system design, robustness of explanations, generating adversarial examples, debugging models using adversarial examples, ML debugger system design, profiling deep learning workloads, ML model monitoring system design, fairness in human-AI settings, human-AI collaboration, model tuning under fairness constraints (ICML'20 AutoML Workshop Best Paper Award), minimax group fairness, and differentially private synthetic data generation.
- Co-organized Amazon-wide workshops/off-sites on fairness, on explainability, and on privacy in ML; Generated awareness on these topics through keynotes/talks at various internal forums.
- Provided technical guidance for the Amazon Research Award program.
- Presented keynotes at LinkedIn-MSR-IISc Workshop on Fairness in ML (slides) and EMNLP'20 PrivateNLP workshop (slides; video) and several invited talks.
- Presented tutorials on explainable AI and on responsible AI in industry at AAAI '20 '21, FAccT '20 '21, WWW '20 '21 & ICML '21, combining experiences across Amazon, Google, LinkedIn & Microsoft.
- Co-organized ICLR'21 Workshop on Synthetic Data Generation.
Fairness, Transparency, Explainability, and Privacy Initiatives across LinkedIn Products: While at LinkedIn, Krishnaram led LinkedIn's efforts in the areas of fairness, transparency, explainability, and privacy in AI/machine learning, and worked with senior leadership across different function areas (AI/ML, Legal, Product, etc.) towards developing and deploying scalable solutions across LinkedIn products.
- Served as LinkedIn's representative in Microsoft's AI and Ethics in Engineering and Research (AETHER) Advisory Board.
- Drafted vision/proposal on fairness/privacy/explainability/trust in AI, and successfully championed for the formation of a new org.
- Tech lead for a virtual team of researchers & engineers working on related areas at LinkedIn.
- Received Distinguished Contribution Award in Microsoft's internal AI/ML conference (MLADS Spring 2018) for our work on detecting and addressing bias in AI systems and designing fairness-aware machine learning/ranking approaches. Video of LinkedIn CEO, Jeff Weiner referring to our work & this award.
- Developed and deployed fairness-aware talent-search system to all users of the LinkedIn Recruiter product worldwide (ACM KDD 2019 paper; LinkedIn Engineering Blog describing the technical architecture).
- Developed and deployed scalable mechanisms for computing privacy-preserving analytics at LinkedIn (ACM CIKM 2018 paper; LinkedIn Engineering Blog Post).
- Collaborated with Microsoft Research [Papers on fairness in ACM FAccT'19 and NAACL-HLT'19 (Best Thematic Paper Award)].
- Invited to speak at RecSys'19 & several privacy/fairness workshops; Gave invited talks titled "Fairness and Privacy in AI/ML Systems" at QConSF'18 (slides; video), @Scale'19, and Pinterest (slides; video).
- Presented tutorials on Privacy-preserving Data Mining in Industry at KDD 2018 [slides], WSDM 2019 [slides] [proposal], and WWW 2019 [slides] [proposal].
- Presented tutorials on Fairness-aware Machine Learning in Practice at WSDM 2019 [slides] [proposal], WWW 2019 [slides] [proposal], and KDD 2019 [slides].
- Presented tutorial on Explainable AI in Industry at KDD 2019 [slides].
- Filed over 15 patents.
LinkedIn Salary: As the Privacy and Modeling Architect & Lead, Krishnaram shaped the technical roadmap for the LinkedIn Salary product (launched to the public in 2016) and provided technical guidance/leadership since its inception in 2015. The key contributions include:
- Designed the engineering and modeling architecture, taking into account privacy, de-identification, security, accuracy, and scalability requirements, and led a ~15 people team for the privacy/modeling system implementation. This involved balancing the needs of privacy and modeling, and bridging the large gap between the security and engineering teams.
- Designed a privacy-preserving data-processing pipeline to ensure that individual salaries are not discoverable. This involved encrypting/de-identifying different aspects of the data at different stages so that there is no single point of failure, and the design is robust to any potential storage/service breaches.
- Designed and implemented an extensible and robust salary modeling rest.li API, considering the initial product needs (incentive for data collection) as well as subsequent scenarios (basis for LinkedIn Salary product and use as a platform for other LinkedIn verticals). Designed schemas for multiple Voldemort distributed key-value stores to support the Rest.Li interface and the offline aggregation flow. These designs helped bring more stability to the overall software design, which tended to evolve with the UI design.
- Developed and deployed AI/statistical models to measure/improve data quality. The modeling efforts included outlier detection, regression models for estimating compensation, Bayesian hierarchical smoothing, low rank matrix factorization/completion, propensity modeling for bias correction, and addressing numerous conceptual issues such as coverage vs quality trade-off and mapping external datasets to LinkedIn taxonomy.
- Co-authored LinkedIn Engineering Blog Posts titled "The Statistical Modeling System Powering LinkedIn Salary" and "Using Economic Graph Data to Power the LinkedIn Salary Product".
- Published papers on the statistical modeling system (ACM KDD 2018 & ACM CIKM 2017 (Best Case Studies Paper Award)) and the privacy architecture (IEEE PAC 2017).
- Filed 42 patents.
LinkedIn Careers and Talent Solutions AI Initiatives: Krishnaram led the relevance efforts for the Careers and Talent Solutions Relevance team, which powers large scale recommendation/search products at the intersection of members, recruiters, and career opportunities. Key milestones for the 10+ people team:
- Built the recommendation models for the newly launched LinkedIn Referrals product, incorporating models for capturing employee-candidate relationship, job fit, connection fit, etc.
- Built new models for providing flavored job recommendations and associated explanations (jobs in my network, high growth/trending companies, educational edge, etc), thereby personalizing and redefining LinkedIn job seeking experience. Resulted in xx% lift in key metrics such as job application rate.
- Launched latent personalization (seniority and relocation) and activity based job recommendation models (+xx% sitewide job applications / job application rate).
- Launched new models for sponsored job recommendations (doubled CPC revenue) and similar jobs (+xx% job application rate).
- Migrated 10+ production services from legacy recommendation platform to newer offline model training / online recommendation platform.
- Internationalized title standardizer based on machine translated titles, and launched country specific recommendation models in many markets (India, China, ...), with xx% lifts in key metrics.
- Represented the team across organizational boundaries (e.g., product reviews with CEO/executive team, planning with application teams, coordination with other stakeholder teams).
- Cultivated a culture of cross-team & intra-team collaborations, by breaking silos and ensuring collective ownership of relevance efforts.
- Published/presented our work in forums such as KDD'16, KDD'17, SIGIR'17, RecSys'17, CIKM'17, SIGIR'18, KDD'18, and CIKM'18.
- Filed 50+ patent applications.
Document Understanding and Augmentation with Multimedia Content: The rapid proliferation of low-cost tablet devices that are readily connected to the cloud opens up several new opportunities to transform reading experience. With printed books being digitized, as is, we address the following question: How do we enrich the experience of reading from electronic versions of printed books/documents, on cloud-connected tablets? Towards this goal, as a team, we have developed approaches for enriching documents by first understanding the semantics of the content, and using that representation to algorithmically augment a non-fiction book with links to selective web content such as text, images and videos, at any desired granularity. We have incorporated these technologies into a Windows 8 app as well as a web app. These apps are serviced off a Microsoft Azure cloud service that automatically identifies relevant content to augment the book. The app is in the last stage of being released to the Windows App Store. We also transferred the book augmentation technology in the form of the code, necessary data, and documentation to the Office Reader team. Our work enabled the product team to endow the Office Reader with a differentiating capability, and helped enhance the e-reader demos shown to Bill Gates and other top Microsoft executives. (Project page)
Video demonstration of the app
Computational Education: Taking into account the vast amount of existing textbooks designed for traditional printed medium and the potential for enabling new kinds of functionalities through the medium of electronic textbooks, our research focuses on algorithmically diagnosing and enhancing the quality of textbooks. Specifically, we have developed a diagnostic tool for authors and educators to identify weaknesses in a textbook. We have also developed algorithmic and data mining techniques for augmenting different sections of a textbook with links to selective web content (web articles, images & videos) and providing easy access to concepts explained elsewhere in the book that are necessary for understanding the present section. We have built a demonstration of the ideas running on Windows 8 devices as well as on Aakash, the low cost tablet being developed by the Indian government for distribution to millions of students. We showcased our technology and Aakash app (by invitation) to India's Ministry of Human Resource and Development, towards the goal of deploying our app on millions of Aakash tablets. (Project page)
Video demonstration of the app
Algorithmic Identification and Ranking of Twitter Groups: Discovered a new kind of group conversation on Twitter that we call a group chat. These chats are periodic, synchronized group conversations focused on specific topics and they exist at a massive scale. Developed techniques for algorithmically identifying and ranking these groups, and performed large scale evaluation over one year of tweets.
Lincoln: Next Generation Product Search Engine: Built a working end-to-end experimental commerce search engine showcasing state-of-the-art breakthrough technologies in structured search query processing, hierarchical query classification, indexing strategies, catalog creation and decision aids for users to make informed decisions. Received Microsoft Research Tech Transfer Award and Gold Star Award from the product team for transferring and integrating hierarchical query classifier and automatic training data generation technologies as part of Bing Commerce Search product. (Project page)
Privacy in Search Logs: Provided the first solution to a long standing open problem: is there a way to publish internet search logs in a manner that both preserves usefulness and provably protects privacy of the users posing queries. My solution is simple to state and thus practical, yet non-trivial to analyze. Showed how search queries, clicks and their perturbed counts can be released in a privacy-preserving manner and further that the released data can be used for important practical applications such as helping online advertising campaigns and social science research to understand human nature. [WWW 2009 Best Paper Award Finalist]. Subsequently, invented a method for privacy preserving distance estimation that involves projecting each user's representation into a random, lower-dimensional space via a sparse Johnson-Lindenstrauss transform and then adding Gaussian noise to each entry of the lower-dimensional representation. Provided consultation to a product team in Microsoft, highlighting privacy leaks in the data that was planned to be released, thereby saving Microsoft from a huge PR fiasco.
Web Search Ranking: Worked on challenging problems in the areas of relevance and ranking for web search, and engaged with Bing Search product team to transfer the results of the investigations. Examples of investigations included designing a dynamic ranker with a pruned set of most important features, improving the quality of results for specific classes of queries by designing special rankers, and automatic generation of large amounts of training data from search logs. Showed that temporally relevant and user intent aware labeled training data can be generated automatically by aggregating click activity of hundreds of millions of users, by first forming a preference graph using a probabilistic interpretation of the click log and then formulating as a novel graph partitioning problem. This investigation enabled Bing to deploy country/locale specific rankers in over 25 international markets. Received multiple "Ship It" awards for my contributions to web search ranking and relevance improvements in Bing.
Trust and Rating Propagation in Social Networks: Developed random walk based algorithms for trust computation and rating propagation in social networks, implemented over MSN messenger network (consisting of 300M+ nodes), and engaged with product teams across Microsoft to demonstrate the results and to advocate using MSN messenger network and the notion of trust towards providing a differentiating web search experience.

Personal: Photos

Krishnaram Kenthapadi