The iNeuOS_Yolo2CausalTCN project implements a complete conversion pipeline from YOLOv8 inference results to causal TCN time series data sets, supports sliding window construction of time series samples, feature normalization and chronological data set division, and has built-in training, evaluation and inference code for causal TCN, opening up the full link of “video target detection – time series data set generation – TCN time series prediction”, which can be used for target number prediction, trajectory trend prediction, video abnormal time series warning and other scenarios.
Open source: Yolo2CausalTCN, an open source tool that converts YOLOv8 datasets into causal TCN timing models
Open source project technical description |
In video analysis scenarios, YOLOv8 can output spatial information such as target detection frames, categories, and confidence levels frame by frame. However, the original detection output is essentially frame-level discrete spatial data and cannot be directly used as input to a time series prediction model. Causal Temporal Convolutional Network (Causal-TCN) relies on causal convolution and atrous convolution, and has long sequence modeling capabilities and parallel computing advantages without leaking future information. It is more suitable for such tasks than recurrent neural networks. However, there is currently a lack of an open source tool that can convert YOLO detection output into TCN standard training samples with one click. Researchers often need to repeatedly write code for frame alignment, window segmentation, normalization and data set division.
The iNeuOS_Yolo2CausalTCN project implements a complete conversion pipeline from YOLOv8 inference results to causal TCN time series data sets, supports sliding window construction of time series samples, feature normalization and chronological data set division, and has built-in training, evaluation and inference code for causal TCN, opening up the full link of “video target detection – time series data set generation – TCN time series prediction”, which can be used for target number prediction, trajectory trend prediction, video abnormal time series warning and other scenarios.
Keywords:YOLOv8; causal TCN; time series data set; spatiotemporal sequence; sliding window; video prediction
In video surveillance and industrial vision scenarios, many tasks are not limited to single-frame target detection, but also require predicting future trends based on historical frame detection results, such as predicting changes in the number of targets in the area, target movement trends, and the probability of occurrence of abnormal events. The detection results output by YOLOv8 frame by frame are a list composed of coordinates, width and height, categories and confidence levels. They are spatial detection results and do not have a time series sample structure themselves and cannot be directly sent to the time series model. Causal TCN requires the input to be fixed-length time series samples with the shape of “number of samples × window length × feature dimension”. The observation statistics of consecutive multiple frames must be organized into historical sequences as input and the values of future moments as labels before they can be used for training.
Most existing open source projects implement target detection and timing model training separately, lacking automated conversion tools. Researchers need to complete frame number alignment, sliding window segmentation, missing frame completion, feature normalization, and data set division by themselves. This results in repetitive work and is prone to errors in timing segmentation. iNeuOS_Yolo2CausalTCN is an end-to-end tool chain provided to eliminate this link. The three specific problems it solves are as follows.
（1）Frame-level detection results cannot be modeled directly.The single-frame detection results are only spatial information and lack the organization method of the time dimension. They need to be rearranged according to the video frame serial number and aggregated into a fixed-length sequence that can be input into the network.
（2）There is a lack of standard procedures for time series sample construction.Historical window length, prediction step size, label alignment and normalization parameters will all affect the training effect, and unified construction rules are required and can be reproduced.
（3）Data division can easily cause time leakage.Time series tasks must be divided into training sets, verification sets, and test sets in chronological order. Random disruption will cause the model to see future information during the verification phase, resulting in falsely high indicators.
The project consists of three modules: YOLOv8 detection result export module, data set conversion module, and causal TCN training and evaluation module. The converter is located in the middle of the entire link, accepting frame-by-frame detection results from any source upwards, and outputting downwards a time series data set that can be directly loaded by PyTorch’s DataLoader. The overall process has a four-segment structure as shown in the figure.
Video file → YOLOv8 inference → each frame detection results json/csv
↓
Yolo2CausalTCN converter (frame alignment, feature extraction, sliding window sampling, normalization, partitioning)
↓
TCN standard time series data set (npy format, X/Y separation)
↓
Causal-TCN training → verification → prediction → indicator output
Figure 1 The overall data processing flow of Yolo2CausalTCN
2.1 Input format of test results
The converter accepts two types of input: one is the JSON result derived from YOLOv8 inference, in which each frame records the coordinates, category numbers and confidence of all target frames in the frame; the other is the frame-by-frame detection summary results in CSV format, with each row corresponding to one frame or one target. The two types of inputs are uniformly mapped to the internal structure of “frame number-target list” after parsing, so the subsequent conversion logic has nothing to do with the detector model. When connecting to other detectors such as YOLOv9, YOLOv11, etc., you only need to ensure that the exported fields are consistent, and there is no need to modify the conversion code.
2.2 Core logic of data set conversion
The conversion module is the most critical part of the entire project. It maps YOLO detection results to supervised learning samples available for causal TCN, and specifically contains five links.
（1）Timing alignment.Sort by video frame number to handle frame skipping and missed detection: missing frames can be filled by zero padding or linear interpolation to ensure that each time step has a certain feature vector and avoid uneven lengths when windows are crossed.
（2）Sliding window sampling.Set the historical window length seq_len and prediction step size pred_len, and slide on the timeline with the step size. The input sample X takes the statistical characteristics of seq_len frames from t-seq_len to t-1, and the label Y takes the value to be predicted from t to t+pred_len-1.
（3）Feature construction and normalization.Features that can be extracted include the number of targets in each frame, the average coordinates of the detection frame, the average width and height, the average confidence level, and the count of each category, etc. Users can combine them as needed. Normalization supports Min-Max and Z-Score methods. Statistics are calculated and saved on the training set for verification, testing and online reasoning reuse to avoid inconsistency between training and inference calibers.
（4）Data set partitioning.Split into training set, verification set and test set in chronological order, and random shuffling is strictly prohibited; at the same time, it is ensured that adjacent windows do not cross the segmentation boundary, preventing the same continuous frame from appearing in the training set and test set at the same time.
（5）Sample placement.The output is a NumPy array file. X and Y are saved separately according to the data set. The shape is[X数, seq_len, feature_dim]and[X数, pred_len, output_dim]can be directly wrapped by torch.utils.data.TensorDataset.
The causal constraint needs to be emphasized here: the output of causal TCN at time t can only depend on the input at t and before, so the label must be strictly located after the input window. The converter verifies that the window does not overlap with the time interval of the label when sampling, fundamentally eliminating time leakage. This is essentially different from ordinary sequence models that allow bidirectional attention.
Table 1 Data set files and uses output by the converter
|
document |
shape |
Content and purpose |
|
X_train.npy |
[N, seq_len, F] |
Training set input sequence, historical detection features intercepted by sliding window |
|
Y_train.npy |
[N, pred_len, O] |
Training set label, the values to be predicted for several frames after the window |
|
X_val.npy / Y_val.npy |
[N, seq_len, F] / [N, pred_len, O] |
Validation set for early stopping and hyperparameter selection |
|
X_test.npy / Y_test.npy |
[N, seq_len, F] / [N, pred_len, O] |
Test set, chronologically located at the last period |
|
norm_params.json |
— |
Normalized statistics are reused in the inference stage to ensure consistent caliber. |
N in Table 1 is the number of samples, F is the feature dimension, and O is the output dimension. For single-variable target number prediction, O takes 1; for multi-step multi-variable prediction, O expands with the number of output variables, and the converter does not impose hard constraints on this.
2.3 Built-in causal TCN model
The project has a built-in standard causal TCN network, which is composed of multi-layer atrous causal convolution blocks, residual connections and Dropout stacks. Causal convolution ensures that the output at time t only depends on t and the previous time steps, structurally preventing future information leakage; dilated convolution expands the receptive field exponentially with the number of layers, and can cover long-term dependencies with fewer layers; residual connections alleviate the gradient disappearance of deep networks, making training more stable.
The model supports two output methods: single-step and multi-step prediction. The loss function can be optionally the mean square error or the mean absolute error. The evaluation uses the RMSE and MAE indicators. Training and inference share the same set of data loading and normalization parameters to avoid common mistakes such as “normalized during training but not normalized during inference”.
The following process assumes that code cloning has been completed in an environment with Python and PyTorch. The overall process is divided into four steps: exporting detection results, converting data sets, training models, evaluation and prediction.
3.1 Environmental dependence
pip install ultralytics torch numpy pandas scikit-learn git clone .git cd iNeuOS_Yolo2CausalTCN
3.2 Export YOLOv8 frame-by-frame detection results
Use YOLOv8 to perform inference on the video and export the results as JSON. The exported files are organized by frame number and serve as input to the converter.
from ultralytics import YOLO
model = YOLO("yolov8s.pt")
results = model.predict(source="demo.mp4", save_json=True)
3.3 Convert to TCN time series data set
Instantiate the converter and specify the historical window length, prediction step size and normalization method to generate the npy data set required for training. The values of the window and step size need to be determined based on the video frame rate and the time span of the prediction task. For example, seq_len=48 for a 25fps video corresponds to approximately 2 seconds of history.
from yolo2causaltcn import Yolo2CausalTCN
converter = Yolo2CausalTCN(
seq_len=48, # TCN 输入的历史窗口长度（帧）
pred_len=6, # 预测未来帧数
stride=1, # 滑动步长
norm_type="z-score"
)
converter.convert(
yolo_json_path="predictions.json",
out_dir="./tcn_dataset/"
)
3.4 Training and evaluation
After the data set is generated, the built-in trainer is directly called, and the training is completed by specifying the number of rounds, batch size, and learning rate, and then evaluation and prediction are performed.
from train_tcn import Trainer trainer = Trainer(dataset_dir="./tcn_dataset") trainer.train(epochs=100, batch_size=16, lr=1e-3) trainer.evaluate() # 输出 RMSE / MAE trainer.predict() # 对未来窗口做推理
4.1 Typical application scenarios
In intelligent monitoring scenarios, the changing trends of people and vehicle flows in the area can be predicted based on the number of targets detected in historical frames, providing a basis for scheduling and alarm threshold adjustment; in industrial vision scenarios, trend modeling can be done on the detection timing of equipment videos, and early warnings can be given before abnormal indicators; in general video behavior analysis, target statistical characteristics can also be used as lightweight timing representations to complete short-term trend forecasts.
4.2 Expansion capabilities
The input of the converter is a unified frame-level detection field, so changing the detector model does not affect the conversion and training process. When connecting to other YOLO series models, you only need to ensure that the export format is consistent. At the feature level, derivative features such as target movement speed, displacement difference, and tracking ID statistics can be added to describe richer dynamic information. At the output level, the generated npy data set is compatible with Temporal Convolution, LSTM and Transformer type timing models, which facilitates horizontal comparison of the effects of different network structures under the same data caliber.
iNeuOS_Yolo2CausalTCN solves the data gap problem between visual detection and temporal convolutional networks. It automatically organizes YOLO’s frame-by-frame detection output into time series samples that comply with causal constraints, eliminating repeated work such as frame alignment, sliding window segmentation, normalization, and data set division, allowing researchers to focus on model structure and task design. The project has built-in time leakage checking in the conversion process, which is particularly critical to the reliability of the conclusions of timing tasks.
Follow-up scalable directions include: accessing tracking algorithms such as ByteTrack, directly converting target trajectory sequences into TCN data sets, and upgrading from “quantity statistics” to “individual trajectory prediction”; adding a visualization module to draw and compare the original sequence and prediction results to quickly judge whether the model has learned effective trends; and supplementing the benchmark experimental configuration of multi-variable and multi-step prediction to facilitate reproduction and horizontal comparison.
Project address: , you are welcome to submit Issues and Pull Requests.
(1) Hardware gateway:
(2) Internet of Things (IOT):
(3)Visual analysis (Vision):
(4) Large model think tank (AiMind):
(5) Large model intelligent questioning (AiInsight):
(6) Xiaoi Assistant:
(7)Visual model:
Internet of Things & Big Data Technology QQ Group: 54256083
Internet of Things & Big Data Project QQ Group: 727664080
QQ：504547114
