LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal Sensor Traces

Introduction

The provided text discusses the use of sensor data from mobile and Internet of Things (IoT) devices, and the integration of this data with machine learning techniques to create an environment with ambient intelligence. It highlights that most current studies focus on low-level perception tasks, such as activity recognition, object detection, and speech recognition, which process raw sensor data over a short time window to make predictions about the current state.

However, the text emphasizes that many practical applications require understanding concepts and making inferences based on long-term sensor data, referred to as high-level reasoning tasks. Examples of such tasks include detecting cognitive impairment, occupancy tracking, human routine modeling, environmental monitoring, and smart energy management. These tasks require sophisticated reasoning abilities to interpret complex sensor traces and integrate domain knowledge to make predictions or decisions.

The existing approaches for high-level reasoning tasks either train machine learning models with long-term sensor traces as input or apply first-principle rules to aggregate low-level perception results. However, the text notes that machine learning-based approaches do not generalize well on data collected from different environments or populations due to limited training samples and the high dimensionality of sensor traces. Designing first-principle models for aggregating short-time perception results requires careful integration of human knowledge.

The paper proposes LLMSense, a system that leverages large language models (LLMs) for high-level reasoning on long-term sensor data traces. The authors argue that LLMs possess reasoning capabilities and vast knowledge, making them suitable for analyzing complex patterns in sensor data for tasks like activity recognition, medical diagnosis, and occupancy tracking.

However, there are challenges in applying LLMs to sensor data, such as converting sensor traces into natural language and handling long sequences of data. The proposed LLMSense system addresses these challenges through an effective prompting framework that incorporates instructions, context information, textualized sensor traces, and output format constraints.

The framework also includes two approaches to enhance performance with long sensor traces: summarization before reasoning and selective inclusion of historical traces. LLMSense can be implemented in an edge-cloud architecture, with small LLMs on the edge extracting summarizations and larger LLMs on the cloud performing high-level reasoning, preserving data privacy.

The authors evaluated LLMSense on two tasks, dementia diagnosis with behavior traces and occupancy tracking with environmental sensor traces. The system achieved around 80% accuracy on these complex high-level reasoning tasks. The authors provide insights and guidelines for leveraging LLMs for high-level reasoning on sensor traces and outline directions for future work.

Related Work

The paper discusses the challenges of high-level reasoning on sensor traces and the potential of using large language models (LLMs) to address this problem. Many practical applications require sophisticated reasoning abilities to interpret complex sensor data and integrate domain knowledge for tasks like occupancy tracking, human routine modeling, environmental monitoring, and energy management.

Existing approaches, such as training machine learning models on sensor traces or applying rules to aggregate low-level perception results, have limitations in terms of generalizability across different environments or populations.

Recent studies have explored the use of LLMs for time-series data analysis, demonstrating their ability to learn and extract complex patterns from time-series data. However, fewer works have focused on effectively executing high-level reasoning on sensor traces, which requires not only understanding the patterns but also employing substantial domain knowledge relevant to the application.

LLMs, being trained on large amounts of language data, possess a vast repository of world knowledge and can be adapted to various domains through zero-shot learning or fine-tuning. They have been leveraged for complex tasks like activity recognition, root cause analysis, and medical diagnosis. However, how to interpret long-term sensor traces with world knowledge in LLMs remains an open question.

Motivation

The paper proposes leveraging pre-trained large language models (LLMs) for high-level reasoning over spatial-temporal sensor traces. Two primary formats of data input are suggested: directly processing streaming sensor data or utilizing low-level perception results.

The proposed framework, LLMSense, involves effective prompting for high-level reasoning over sensor traces. To enhance performance with long sensor traces, two approaches are proposed: summarization of the traces before reasoning and selective inclusion of historical traces. LLMSense can be implemented in an edge-cloud framework, with small LLMs running on the edge to extract summarization of traces and high-level reasoning performed on the cloud to preserve data privacy.

Two application scenarios are discussed:

1. Directly processing streaming sensor data: When the sensor data has a low data rate and it is not intuitive to define an intermediate low-level perception task, the sensor data can be directly textualized and structured as input to LLMs. For example, in occupancy tracking with environmental sensor data.

2. Utilizing low-level perception results: When the sensor data has a high data rate, like depth videos, it is inefficient to input the raw sensor data to LLMs. Instead, a classical machine learning model can be used for low-level perception tasks, and the perception results can be structured as input to LLMs. For example, in cognitive impairment detection with longitudinal sensor data, activity recognition results can be input to LLMs for analysis.

The advantages of leveraging LLMs for high-level reasoning on sensor traces are outlined as:

• Making use of the world knowledge possessed by LLMs for complex tasks like activity recognition and medical diagnosis.
• The ability of LLMs to understand long-series texts, which is useful for sensor traces consisting of continuous measurements or observations.
• The generalizable ability of LLMs to extrapolate from previous knowledge and adapt to diverse settings, even with little training data.

V Design of LLMSense

The paper proposes an effective prompting framework called LLMSense for high-level reasoning over sensor traces using large language models (LLMs). The key components of the framework include:

1. Careful prompt design with four parts - objective, context, data, and format - to enable LLMs to perform reasoning tasks on sensor data.

2. Two approaches to handle long sensor traces: a. Summarizing the traces using the LLM's summarization ability before performing the main reasoning task. This makes the information more concise. b. Selectively including relevant historical traces in addition to the latest traces, condensing the information while preserving essential context.

The prompt design aims to inject domain knowledge and provide background information to enhance the LLM's reasoning capability on sensor data. The summarization and selective history inclusion approaches tackle the challenge of context limits when dealing with extensive sensor traces. Overall, the framework leverages the capabilities of LLMs for high-level reasoning on sensor data while addressing the limitations of long input sequences.

Evaluation and Results

The paper evaluates the performance of different language models (LLama2-13B, LLama2-70B, and GPT3.5) on two high-level reasoning tasks: dementia diagnosis with behavior traces and occupancy tracking with sensor traces. The experiments are conducted in a zero-shot setting, where the models are prompted with data samples without any task-specific training.

For the dementia diagnosis task, the dataset contains multimodal sensor data from 16 subjects, including those with Alzheimer's Disease, mild cognitive impairment, and cognitively normal individuals. The goal is to diagnose whether the subjects have cognitive impairment or not based on the sensor data and timestamps.

For the occupancy tracking task, the dataset contains sensor data (ambient light, sound, temperature, air quality, humidity, and CO2) collected over 80 working days in an office room. The objective is to detect whether there is occupancy in the room or not based on the sensor data and timestamps.

The paper evaluates the accuracy, consistency, and uncertainty of the models' predictions across multiple trials. The results show that GPT3.5 achieves comparable accuracy to previous studies that trained machine learning models with extensive data, demonstrating the reasoning ability and knowledge of large language models.

The paper also investigates the effectiveness of summarizing sensor traces before providing them to the language models. Summarization improves the accuracy, consistency, and reduces uncertainty compared to using raw sensor traces. Additionally, the performance improves when considering longer historical sensor data, but selectively adding relevant historical information yields better results.

Finally, the paper compares the latency of running language models on edge devices (local machine) or in the cloud. Running larger models like GPT3.5 on the cloud provides higher accuracy but requires sharing raw sensor data, while running smaller models like LLama2-13B on edge devices preserves privacy but sacrifices accuracy. A hybrid approach, where summarization is done on the edge and reasoning on the cloud, offers a trade-off between accuracy, latency, and privacy.

Discussion

The paper discusses several future research directions:

1. Process longer or infinite sensor traces: While the proposed approaches enhance LLM performance on long sensor traces, interpreting lengthy or infinite traces remains challenging due to the contextual limits of LLMs. Future work will incorporate stateful LLMs for high-level reasoning or adaptive online selection of input traces.

2. Improve LLM performance based on verifications: The prediction results of LLMs can still be inconsistent and uncertain, especially for complex tasks requiring advanced reasoning abilities. Future research will focus on quantifying the uncertainty or errors of LLM outputs and iteratively improving LLM performance through rule-based or human-feedback-based verifications.

3. Joint optimization of low-level perception and high-level reasoning tasks: Conventional neural networks for low-level perception tasks require extensive labeled training data, which may not be available in practical applications. On the other hand, labels for high-level reasoning tasks tend to be sparse. Future work will explore leveraging the reasoning ability of LLMs to associate low-level perception and high-level reasoning tasks and enhance the training of low-level perception models.

Conclusion

The paper proposes utilizing Large Language Models (LLMs) to analyze observations derived from long-term sensor data for high-level reasoning tasks. An effective prompting framework is designed, enabling LLMs to handle both raw sensor data and low-level perception results. To enhance performance with long sensor traces, two strategies are introduced: summarizing data before reasoning and selectively including historical traces. The paper provides insights into leveraging LLMs for high-level reasoning on sensor data and highlights potential future research directions.

Acknowledgment

The research described in this paper received funding from several sources. The Air Force Office of Scientific Research provided support through a cooperative agreement. The DARPA ANSR Program contributed funding through a contract. The DEVCOM ARL also provided funding via a cooperative agreement. Additionally, the NIH mDOT Center contributed funding through an award. It is important to note that the views and conclusions expressed in the paper belong solely to the authors and do not necessarily represent the official policies of the funding agencies.