用Transformer建立数值时间序列预测模型开源代码汇总
Transformer是一个利用注意力机制来提高模型训练速度的模型。，trasnformer可以说是完全基于自注意力机制的一个深度学习模型，因为它适用于并行化计算，和它本身模型的复杂程度导致它在精度和性能上都要高于之前流行的RNN循环神经网络。
记录一下Transformer做数值时间序列预测的一下开源代码
time_series_forcasting
代码地址/CVxTz/time_series_forecasting
Transformer-Time-Series-Forecasting
代码地址
/nklingen/Transformer-Time-Series-Forecasting
Article: https://natasha-/transformer-implementation-for-time-series-forecasting-a9db2db5c820
szZack的博客
Transformer_Time_Series
代码地址
/mlpotter/Transformer_Time_Series
论文地址：
Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting (NeurIPS )
/pdf/1907.00235.pdf
Non-AR Spatial-Temporal Transformer
Introduction
Implementation of the paper NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting (submitted to ICML ).
We propose a Non-Autoregressive Transformer architecture for time series forecasting, aiming at overcoming the time delay and accumulative error issues in the canonical Transformer. Moreover, we present a novel spatial-temporal attention mechanism, building a bridge by a learned temporal influence map to fill the gaps between the spatial and temporal attention, so that spatial and temporal dependencies can be processed integrally.
论文地址：/pdf/2102.05624.pdf代码地址
/Flawless1202/Non-AR-Spatial-Temporal-Transformer
Multidimensional-time-series-with-transformer
Transformer/self-attention for Multidimensional time series forecasting 使用transformer架构实现多维时间预测
Rerfer to /oliverguhr/transformer-time-series-prediction
代码地址
/RuifMaxx/Multidimensional-time-series-with-transformer
szZack的博客
TCCT
Convolutional Transformer Architectures Complementary to Time Series Forecasting Transformer Models
Paper: TCCT: Tightly-Coupled Convolutional Transformer on Time Series Forecasting /abs/2108.12784
It has already been accepted by Neurocomputing:
Journal ref.: Neurocomputing, Volume 480, 1 April , Pages 131-145
doi: 10.1016/j.neucom..01.039
代码地址
/OrigamiSL/TCCT-Neurocomputing-
Time_Series_Transformers
Introduction
This directory contains a Pytorch/Pytorch Lightning implementation of transformers applied to time series. We focus on Transformer-XL and Compressive Transformers.
Transformer-XL is described in this paper Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov (*: equal contribution) Preprint .
Part of this code is from the authors at /kimiyoung/transformer-xl.
代码地址
/Emmanuel-R8/Time_Series_Transformers
Multi-Transformer: A new neural network-based architecture for forecasting S&P volatility
Transformer layers have already been successfully applied for NLP purposes. This repository adapts Transfomer layers in order to be used within hybrid volatility forecasting models. Following the intuition of bagging, this repository also introduces Multi-Transformer layers. The aim of this novel architecture is to improve the stability and accurateness of Transformer layers by averaging multiple attention mechanism.
The article collecting theoretical background and empirical results of the proposed model can be downloaded here. The stock volatility models based on Transformer and Multi-Transformer (T-GARCH, TL-GARCH, MT-GARCH and MTL-GARCH) overcome the performance of traditional autoregressive algorithms and other hybrid models based on feed forward layers or LSTM units. The following table collects the validation error (RMSE) by year and model.
代码地址
/EduardoRamosP/MultiTransformer
szZack的博客
一个很好的完整的例子
代码
/OrigamiSL/TCCT-Neurocomputing-
/zhouhaoyi/Informer
parser = argparse.ArgumentParser(description='[Informer] Long Sequences Forecasting')parser.add_argument('--model', type=str, required=True, default='informer',help='model of experiment, options: [informer, informerstack, informerlight(TBD)]')parser.add_argument('--data', type=str, required=True, default='ETTh1', help='data')parser.add_argument('--root_path', type=str, default='./data/ETT/', help='root path of the data file')parser.add_argument('--data_path', type=str, default='ETTh1.csv', help='data file') parser.add_argument('--features', type=str, default='M', help='forecasting task, options:[M, S, MS]; M:multivariate predict multivariate, S:univariate predict univariate, MS:multivariate predict univariate')parser.add_argument('--target', type=str, default='OT', help='target feature in S or MS task')parser.add_argument('--freq', type=str, default='h', help='freq for time features encoding, options:[s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly], you can also use more detailed freq like 15min or 3h')parser.add_argument('--checkpoints', type=str, default='./checkpoints/', help='location of model checkpoints')parser.add_argument('--seq_len', type=int, default=96, help='input sequence length of Informer encoder')parser.add_argument('--label_len', type=int, default=48, help='start token length of Informer decoder')parser.add_argument('--pred_len', type=int, default=24, help='prediction sequence length')# Informer decoder input: concat[start token series(label_len), zero padding series(pred_len)]parser.add_argument('--enc_in', type=int, default=7, help='encoder input size')parser.add_argument('--dec_in', type=int, default=7, help='decoder input size')parser.add_argument('--c_out', type=int, default=7, help='output size')parser.add_argument('--d_model', type=int, default=512, help='dimension of model')parser.add_argument('--n_heads', type=int, default=8, help='num of heads')parser.add_argument('--e_layers', type=int, default=2, help='num of encoder layers')parser.add_argument('--d_layers', type=int, default=1, help='num of decoder layers')parser.add_argument('--s_layers', type=str, default='3,2,1', help='num of stack encoder layers')parser.add_argument('--d_ff', type=int, default=2048, help='dimension of fcn')parser.add_argument('--factor', type=int, default=5, help='probsparse attn factor')parser.add_argument('--distil', action='store_false', help='whether to use distilling in encoder, using this argument means not using distilling', default=True)parser.add_argument('--CSP', action='store_true', help='whether to use CSPAttention, default=False', default=False)parser.add_argument('--dilated', action='store_true', help='whether to use dilated causal convolution in encoder, default=False', default=False)parser.add_argument('--passthrough', action='store_true', help='whether to use passthrough mechanism in encoder, default=False', default=False)parser.add_argument('--dropout', type=float, default=0.05, help='dropout')parser.add_argument('--attn', type=str, default='prob', help='attention used in encoder, options:[prob, full, log]')parser.add_argument('--embed', type=str, default='timeF', help='time features encoding, options:[timeF, fixed, learned]')parser.add_argument('--activation', type=str, default='gelu',help='activation')parser.add_argument('--output_attention', action='store_true', help='whether to output attention in encoder')parser.add_argument('--do_predict', action='store_true', help='whether to predict unseen future data')parser.add_argument('--num_workers', type=int, default=0, help='data loader num workers')parser.add_argument('--itr', type=int, default=2, help='experiments times')parser.add_argument('--train_epochs', type=int, default=6, help='train epochs')parser.add_argument('--batch_size', type=int, default=16, help='batch size of train input data')parser.add_argument('--patience', type=int, default=3, help='early stopping patience')parser.add_argument('--learning_rate', type=float, default=0.0001, help='optimizer learning rate')parser.add_argument('--des', type=str, default='test',help='exp description')parser.add_argument('--loss', type=str, default='mse',help='loss function')parser.add_argument('--lradj', type=str, default='type1',help='adjust learning rate')parser.add_argument('--use_amp', action='store_true', help='use automatic mixed precision training', default=False)parser.add_argument('--inverse', action='store_true', help='inverse output data', default=False)parser.add_argument('--use_gpu', type=bool, default=True, help='use gpu')parser.add_argument('--gpu', type=int, default=0, help='gpu')parser.add_argument('--use_multi_gpu', action='store_true', help='use multiple gpus', default=False)parser.add_argument('--devices', type=str, default='0,1,2,3',help='device ids of multile gpus')
szZack的博客
数据集
/zhouhaoyi/ETDataset
