A domain-agnostic approach for characterization of lifelong learning systems.

Journal: Neural networks : the official journal of the International Neural Network Society
Published Date: Jan 20, 2023

Abstract

Despite the advancement of machine learning techniques in recent years, state-of-the-art systems lack robustness to "real world" events, where the input distributions and tasks encountered by the deployed systems will not be limited to the original training context, and systems will instead need to adapt to novel distributions and tasks while deployed. This critical gap may be addressed through the development of "Lifelong Learning" systems that are capable of (1) Continuous Learning, (2) Transfer and Adaptation, and (3) Scalability. Unfortunately, efforts to improve these capabilities are typically treated as distinct areas of research that are assessed independently, without regard to the impact of each separate capability on other aspects of the system. We instead propose a holistic approach, using a suite of metrics and an evaluation framework to assess Lifelong Learning in a principled way that is agnostic to specific domains or system techniques. Through five case studies, we show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems. We highlight how the proposed suite of metrics quantifies performance trade-offs present during Lifelong Learning system development - both the widely discussed Stability-Plasticity dilemma and the newly proposed relationship between Sample Efficient and Robust Learning. Further, we make recommendations for the formulation and use of metrics to guide the continuing development of Lifelong Learning systems and assess their progress in the future.

Authors

  • Megan M Baker
    Johns Hopkins University Applied Physics Laboratory, 11100 Johns Hopkins Rd., Laurel, 20723, MD, USA. Electronic address: megan.baker@jhuapl.edu.
  • Alexander New
  • Mario Aguilar-Simon
    Intelligent Systems Laboratory, Teledyne Scientific, Research Triangle Park, NC, 27709, USA. Electronic address: mario.aguilar@teledyne.com.
  • Ziad Al-Halah
    Department of Computer Science, University of Texas at Austin, Austin, TX, USA.
  • Sébastien M R Arnold
    Department of Computer Science, University of Southern California, Los Angeles, CA, USA.
  • Ese Ben-Iwhiwhu
    Department of Computer Science, Loughborough University, Loughborough, England, UK.
  • Andrew P Brna
    Intelligent Systems Laboratory, Teledyne Scientific, Research Triangle Park, NC, 27709, USA.
  • Ethan Brooks
    Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA.
  • Ryan C Brown
    Intelligent Systems Laboratory, Teledyne Scientific, Research Triangle Park, NC, 27709, USA.
  • Zachary Daniels
    SRI International, 201 Washington Rd, Princeton, NJ, USA.
  • Anurag Daram
    University of Texas at San Antonio, San Antonio, TX, USA.
  • Fabien Delattre
    Department of Computer Science, University of Massachusetts Amherst, Amherst, MA, USA.
  • Ryan Dellana
    Sandia National Laboratories, Albuquerque, NM, USA.
  • Eric Eaton
    Department of Computing and Information Science, University of Pennsylvania, Philadelphia, PA 19104.
  • Haotian Fu
    Department of Computer Science, Brown University, Providence, RI, USA.
  • Kristen Grauman
    Department of Computer Science, University of Texas at Austin, Austin, TX, USA.
  • Jesse Hostetler
    SRI International, 201 Washington Rd, Princeton, NJ, USA.
  • Shariq Iqbal
    Department of Computer Science, University of Southern California, Los Angeles, CA, USA.
  • Cassandra Kent
    Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA.
  • Nicholas Ketz
  • Soheil Kolouri
    Department of Computer Science, Vanderbilt University, Nashville, TN 37212, USA.
  • George Konidaris
    Department of Computer Science, Brown University, Providence, RI, USA.
  • Dhireesha Kudithipudi
  • Erik Learned-Miller
    College of Information and Computer Sciences, University of Massachusetts, Amherst, USA.
  • Seungwon Lee
    Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA.
  • Michael L Littman
    Department of Computer Science, Brown University, Providence, Rhode Island 02912, USA.
  • Sandeep Madireddy
    Argonne National Laboratory, 9700 S Cass Ave, Lemont, IL, USA.
  • Jorge A Mendez
    Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA.
  • Eric Q Nguyen
    Johns Hopkins University Applied Physics Laboratory, 11100 Johns Hopkins Rd., Laurel, 20723, MD, USA.
  • Christine Piatko
    Johns Hopkins University Applied Physics Laboratory, 11100 Johns Hopkins Rd., Laurel, 20723, MD, USA.
  • Praveen K Pilly
    Information and Systems Sciences Laboratory, HRL Laboratories LLC, Malibu, CA 90265, USA. Electronic address: pkpilly@hrl.com.
  • Aswin Raghavan
    SRI International, 201 Washington Rd, Princeton, NJ, USA.
  • Abrar Rahman
    SRI International, 201 Washington Rd, Princeton, NJ, USA.
  • Santhosh Kumar Ramakrishnan
    Department of Computer Science, University of Texas at Austin, Austin, TX, USA.
  • Neale Ratzlaff
    Information and Systems Sciences Laboratory, HRL Laboratories, 3011 Malibu Canyon Road, Malibu, 90265, CA, USA.
  • Andrea Soltoggio
    Department of Computer Science, Loughborough University, LE11 3TU, Loughborough, UK. Electronic address: a.soltoggio@lboro.ac.uk.
  • Peter Stone
  • Indranil Sur
    SRI International, 201 Washington Rd, Princeton, NJ, USA.
  • Zhipeng Tang
    Department of Computer Science, University of Massachusetts Amherst, Amherst, MA, USA.
  • Saket Tiwari
    Department of Computer Science, Brown University, Providence, RI, USA.
  • Kyle Vedder
    Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA.
  • Felix Wang
    Sandia National Laboratories, Albuquerque, NM, USA.
  • Zifan Xu
    Department of Computer Science, University of Texas at Austin, Austin, TX, USA.
  • Angel Yanguas-Gil
    Argonne National Laboratory, 9700 S Cass Ave, Lemont, IL, USA.
  • Harel Yedidsion
    Department of Computer Science, University of Texas at Austin, Austin, TX, USA.
  • Shangqun Yu
    Department of Computer Science, Brown University, Providence, RI, USA.
  • Gautam K Vallabha
    Johns Hopkins University Applied Physics Laboratory, 11100 Johns Hopkins Rd., Laurel, 20723, MD, USA.

Keywords

External Resources

Popular Topics
Recent Journals