IEEE SMC-IT/SCC 2024: Trustworthiness of Foundation Models and What They Generate

Speakers and Abstracts:

Manil Maskey

Sr. Research Scientist
Data Science and Innovation Lead
NASA HQ/NASA MSFC

Dr. Manil Maskey holds two main roles: one at the Marshall Space Flight Center (MSFC) and another at NASA Headquarters. In his role as Senior Research Scientist and Project Manager at NASA MSFC, Dr. Maskey leads research and development efforts in data science tailored to the unique demands of the scientific community. In his capacity at NASA Headquarters as the Data Science and Innovation Lead for the Office of the Chief Science Data Officer, Dr. Maskey oversees the development and execution of NASA Science Mission Directorate's artificial intelligence strategy. With a career that extends over two decades including academia, industry, and government, his expertise encompasses data systems, cloud computing, artificial intelligence, and visualization. Dr. Maskey is an affiliated faculty member in the Atmospheric Science department at the University of Alabama in Huntsville. He is a senior member of the Institute of Electrical and Electronics Engineers (IEEE), serves as the chair of the IEEE Geoscience and Remote Sensing Society (GRSS) Earth Science Informatics Technical Committee, and is a member of the Association for the Advancement of Artificial Intelligence (AAAI).

AI Foundation Models for NASA Science: a Culture of Openness

Abstract:
Developing Foundation Models (FMs) demands significant resources, including data, computing power, and specialized foundational artificial intelligence expertise. At NASA Science, we have cultivated a culture of openness to enhance collaborative efforts in FM development across all science domains. The goal is to leverage these collaborative efforts to lessen the burden on individual groups and maximize the benefits for everyone. This talk will explore NASA Science's 5+1 foundation model strategy, implementation approaches, challenges, and envisioned benefits.

Geoffrey Fox

Professor of Computer Science, School of Engineering and Applied Science, UVA, Biocomplexity Institute

Fox received a Ph.D. in Theoretical Physics from Cambridge University, where he was Senior Wrangler. He is now a Professor at the Biocomplexity Institute & Initiative and Computer Science Department at the University of Virginia. He previously held positions at Caltech, Syracuse University, Florida State University, and Indiana University. after being a postdoc at the Institute for Advanced Study at Princeton, Lawrence Berkeley Laboratory, and Peterhouse College Cambridge. He has supervised the Ph.D. of 78 students. He has an hindex of 89 with over 44,000 citations. He received the High-Performance Parallel and Distributed Computing (HPDC) Achievement Award and the ACM - IEEE CS Ken Kennedy Award for Foundational contributions to parallel computing in 2019. He is a Fellow of APS (Physics) and ACM (Computing) and works on the interdisciplinary interface between computing and applications. He is currently active in the Industry consortium MLCommons/MLPerf.

Foundation Models and Patterns for Science Time Series

Abstract:
In the last three years, hundreds of papers and over fifty new models have been described for Time Series Foundation models. These mainly focus on the key aspect of time dependence, whereas in some science time series, there are multiple sources with individual features (often static) that describe the different streams. These add heterogeneity to the system that could be challenging for the Foundation model. We analyze earthquake and hydrology forecasting. In the former case, spatial correlations caused by faults and relations between shocks, consequent from physics, can be usefully exploited. In the hydrology case, there are over two hundred static variables that distinguish the thousands of catchments. Measurement errors should be reflected in loss functions. One can compare the use of mathematical expansions of time dependence (e.g. Legendre Polynomials or Fourier series) with time dependence learned by the neural network. We also remark on Foundation models for image-based science observables.

Brian Green

Brian Patrick Green is the director of technology ethics at the Markkula Center for Applied Ethics at Santa Clara University and teaches AI ethics and space ethics in Santa Clara University’s Graduate School of Engineering. He is author of the book Space Ethics, co-author of Ethics in the Age of Disruptive Technologies: An Operational Roadmap (The ITEC Handbook), co-author of the Ethics in Technology Practice corporate technology ethics resources, and co-author/editor of three additional volumes on technology, ethics, religion, and society. Green has worked with the World Economic Forum, the Vatican’s Dicastery for Culture and Education, and technology companies ranging from startups to the largest.

Ethics and Trustworthy Foundation Models

Abstract:
What does it mean for something to be "trustworthy"? At the very least, it must be both technically trustworthy - it does what it is supposed to do - and ethically trustworthy - it does not violate ethical ideals necessary for trust (such as violating privacy, deceiving, harming, or exploiting users, etc.). This talk will explore linkages between AI and trust and present some ethical tools for thinking about and building trustworthy technology.

Bjorn Andersson

Bjorn Andersson earned his PhD from Chalmers University in Sweden in 2003. His PhD thesis was "Static-priority scheduling on multiprocessors" and it solved a problem left open by C. Liu in the Apollo program in 1969; see C. Liu, "Scheduling algorithms for multiprocessors in a hard real-time environment," in JPL Space Programs Summary, vol. 37-60. JPL, Pasadena, CA, 28–31, 1969. Today, Bjorn Andersson is Principal Researcher at the Software Engineering Institute at Carnegie Mellon University, and he is the author of more than 100 peer reviewed publications.

Leveraging AI for Assurance of Critical Software Systems

Abstract:
The reliance on software for executing safety-critical functions is expanding, seen in applications such as flight control systems and military counter-measures. The challenge in ensuring the safety of these systems stems from their vast input space and the critical timing of output delivery. My presentation will explore two application of AI in this context: (i) the use of large language models for hazard analysis, and (ii) machine learning for analyzing worst-case execution time.

Jack Lightholder

Jack Lightholder is a data scientist at NASA’s Jet Propulsion Laboratory in Pasadena, California. As a member of the Machine Learning and Instrument Autonomy Group, Jack supports a variety of missions in concept, development and operational phases. Currently Jack acts as a payload uplink lead on the engineering camera subsystem of the Mars Science Laboratory. Jack is also supporting instrument flight software development for the Near Earth Asteroid CubeSat mission.

Intelligent Parsing of Academic Literature Using Large Language Models

Abstract:
In all areas of science and engineering there is a challenge in creating and maintaining exhaustive and accurate Analysis-Ready Data (ARD) compilations. How information is described and reported may vary substantially between publications, even when they are reporting on the same phenomena. Curation tasks can commonly require tracking hundreds of pieces of information across tens of thousands of objects. Such a scale makes manual curation impractical, and the wide variety of literary expressions makes automation difficult. The recent rapid advances in the ability of Large Language Models (LLMs) to distill, summarize and utilize large text corpora make them an excellent candidate technology for solving such challenges. We explored the use of LLMs to parse, organize and curate knowledge from published academic literature on strong gravitational lenses. To date around 10,000 of these astrophysical phenomena have been published on, but that number is expected to increase rapidly over the coming decade due to new missions more capable of detecting them. We utilized multiple LLM models, including GPT-4, Llama 3 and Claude 3, and characterized their ability to identify gravitational lenses referenced in provided literature and extract a list of metadata for each published lens. Models were evaluated on their ability to comprehensively identify all lenses in a provided publication as well as their ability to parse metadata without missing provided information or hallucinating information not provided. LLMs are rapidly evolving, making the performance of a specific model less important than the lessons which can be provided to others to bootstrap their first attempts to utilize this technology to solve real world problems. We outline our approach to addressing our use case, and present lessons learned along the way and a roadmap for others attempting to utilize LLMs for literature parsing in the science and engineering domains.

Workshop Organizers:

Daniel Crichton

Dan Crichton is a program manager, principal investigator, principal computer scientist, and the Leader of the Center for Data Science and Technology at JPL, working on data science, software, and computing projects, particularly to support physical and biological sciences. He led re-development of the NASA Planetary Data System and led a project to support the transfer of space data and computing methodologies to cancer research.

Richard Doyle

Richard J. Doyle recently retired from the JPL. He had been the Program Manager for Information and Data Science, and he continues to provide technical consulting in this broad area. His interests span data science, autonomous systems, computing, software engineering, mission protection and similar topics applying computer science to space missions. He has authored articles on machine learning, model-based reasoning, space-based computing, autonomous systems, and data science. He is an Associate Fellow of the American Institute of Aeronautics and Astronautics (AIAA). He is past Executive Council member of the Association for the Advancement of Artificial Intelligence (AAAI).

Ashish Mahabal

Ashish Mahabal is an astronomer and the deputy director at the Center for Data Driven Discovery at Caltech. He has worked on many large sky surveys, and leads the machine learning group for the Zwicky Transient Facility (ZTF) with a special interest in classification. He also applies data science techniques to early detection of cancer and other medical data in collaboration with JPL.