Index for Various Fields

• See More

  LLM-based time series analysis. LINK

• Backdoor attacks. LINK()

• AI4Bio.(single cell, etc.) LINK

• Efficient AI. LINK

• Pruning. LINK

• Interesting papers. LINK

This Page collects the papers and codes of Large Language Models (LLMs) and Foundation Models (FMs) for Time Series (TS).

After the success of BERT, GPT, and other LLMs in NLP, some researchers have proposed to apply LLMs to Time Series (TS) tasks. They fintune the LLMs on TS datasets and achieve SOTA results.

🦙 LLMs for Time Series

• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. [Paper] [Note]

• TEST: Text Prototype Aligned Embedding to Activate LLM’s Ability for Time Series. [Paper] [Note]

• PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecasting Hao, in arXiv 2022. [Paper] [Note]

• One Fits All: Power General Time Series Analysis by Pretrained LM, in arXiv 2023. [Paper] [Note]

• Temporal Data Meets LLM – Explainable Financial Time Series Forecasting, in arXiv 2023. [Paper]

• LLM4TS: Two-Stage Fine-Tuning for Time-Series Forecasting with Pre-Trained LLMs. [Paper] [Note]

• The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models. [Paper]

• Large Language Models Are Zero-Shot Time Series Forecasters. [Paper] [Note]

• TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting. [Paper] [Note]

• S2IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting. [Paper]

📍 Survey

• Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook. [Survey]

• Position Paper: What Can Large Language Models Tell Us about Time Series Analysis. [Survey]

• Foundation Models for Time Series Analysis: A Tutorial and Survey [Survey]

🔗 Related Fields

Here, some related fields are listed. These fields are not the main focus of this project, but they are also important for understanding how LLMs are applied to other fields rather than NLP and FMs in specific fields are developed.

📍 LLM for Recommendation Systems

• Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5), in arXiv 2022. [Paper]
• LLM4Rec. [GitHub]

This Page collects the papers and codes of Backdoor attacks on LLM or TS. Additional, I read paper and take notes.

📍 Backdoor Attacks on LLM or Time Series

• Backdoor Learning: A Survey [Paper] [Note]

• A survey on Large Language Model (LLM) security and privacy: The Good, The Bad, and The Ugly [Paper] [Note]

• A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networks [Paper] [Note]

• Backdoor Attacks on Time Series: A Generative Approach, in arXiv 2022. [Paper]

• Paper List: Awesome Data Poisoning and Backdoor Attacks [GitHub]

PS: There are more paper and notes in my FEISHU doc, click link to veiw(I will gradually transfer them from FEISHU doc to this page.)

This Page collects the papers and codes of AI4Bio. Additional, I read paper and take notes.

🦙 Single-cell Foundation Model

• scGPT: toward building a foundation model for single-cell multi-omics using generative AI [Paper] [Note]

This Page collects the papers and codes of Efficient AI, Efficient Large Language Models (LLMs). Additional, I read paper and take notes.

🦙 Efficient LLM

Survey

• Efficient Large Language Models: A Survey, Arxiv, Github repo

Books & Courses

• TinyML and Efficient Deep Learning @MIT by Prof. Song Han (I may update some my learning notes later on my homepage)

This Page collects the papers and codes of which attracted my interests. Additional, I read paper and take notes.

effective LLM, multimodal, cross-discipline, Leverage Learning

🦙 Interesting Paper

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces, in arXiv 2023. [Paper]

• Token-Efficient Leverage Learning in Large Language Models, in arXiv 2024. [Paper]

• Heterogeneous Graph Neural Network, in ACM 2019 [Paper] [Note]

• S4 model, [Paper] [Note]

TODO

Heterogeneous Graph Neural Network

1. Introduction

This paper addresses the representation learning challenges of content-associated heterogeneous graphs (HetG). Traditional works primarily focused on homogeneous structural information and ignored the rich, diverse content across different node types, such as textual, attribute, and image data. We introduce HetGNN, a model designed to encapsulate both the structural and content heterogeneity within graphs.

2. Methodology

2.1 Challenges Addressed

HetGNN tackles several key challenges in the realm of heterogeneous graphs:

• C1: Sampling relevant heterogeneous neighbors: Unlike most GNNs, which only aggregate first-degree neighbors, HetGNN utilizes a random walk with restart strategy to sample a diverse set of strongly associated heterogeneous neighbors.
• C2: Encoding diverse content information: Nodes in HetG have varied content types. HetGNN employs a content encoder using RNNs to model the intricate interactions of these content types deeply.
• C3: Aggregating heterogeneous neighbor features: To account for the varying influences of different neighbor types, HetGNN integrates a type-based attention mechanism, ensuring that each node type’s contribution is weighted appropriately.

2.2 HetGNN Architecture

HetGNN’s architecture comprises two main modules:

• Content Encoding Module: Uses a Bi-LSTM to capture deep feature interactions of node content, resulting in robust content embeddings.
• Neighbor Aggregation Module: Aggregates embeddings of different neighbor types using another Bi-LSTM, with attention mechanisms to adjust the influence of each type on the final node embedding.

3. Experiments and Results

HetGNN was evaluated against state-of-the-art models across multiple datasets and graph mining tasks, including link prediction, recommendation, node classification, and clustering, both in transductive and inductive settings. The results demonstrate that HetGNN consistently outperforms existing methods, particularly in environments rich in node content information.

4. Contributions and Model Advantages

HetGNN significantly advances the field by:

• Defining the heterogeneous graph representation learning problem that involves both structural and content heterogeneity.
• Developing a robust model that effectively captures both elements, applicable to both transductive and inductive tasks.
• Achieving state-of-the-art performance on multiple graph mining tasks, demonstrating the effectiveness of our dual-module architecture and the importance of considering both node and edge types in heterogeneous graphs.

5. Conclusion

HetGNN represents a comprehensive approach to the challenges of heterogeneous graph analysis. The model’s ability to integrate and learn from both the structural connections and the rich content of nodes leads to superior performance and broad applicability. This work not only sets a new benchmark for heterogeneous graph neural networks but also opens new avenues for future research in this area.

Large Language Models Are Zero-Shot Time Series Forecasters

1. 研究背景/动机

和之前文章差不多，不再赘述

2. 创新点

方法虽然比较平凡，但可能在当时具有一定的新颖性。

3. 主要方法

本研究的思路直观简单：输入由时序数值组成的句子，预测后续数值组成的句子。

Token标记

由于模型中存在的各种标记问题，作者选择在每个数字间加入几个逗号，以强制规定标记方法。具体是否加空格则根据不同的LLM进行调整。

Rescaling

为了避免某些数值过大，覆盖了过多的token，数据需要进行预处理，例如进行缩放。不同的LLM之间具体的缩放方法有所不同。

Sampling / Forecasting

每次预测时，通过多次采样实验获得多组预测值，取这些预测值的中位数或均值作为点预测的结果，以增加结果的鲁棒性。

Continuous Likelihoods

LLM的概率分布是离散的，需要将其转换为连续概率密度，方法是简单地在段内赋予均匀分布。

Language Models as Flexible Distributions

序列预测本质上是对未来值的条件分布进行建模，因此LLM自然也适用于此类任务。

4. 数据集

使用了 Darts、Monash、Informer 等数据集（具体见论文）。

5. 实验结果

作者提供了详细的实验结果，但在此省略相关细节。

6. 实验环境

具体实验条件未详细说明，且未开源

这个Youtuber讲的可以，同时需要自己查一些basic knowledge，我就不赘述了。