Autoencoder (Plc912)

Autoencoder deep learning time series anomaly detection tool

Author: Pang Licheng

Email: 3522236586@qq.com

A time series anomaly detection tool based on PyTorch Autoencoder, supporting CSV and TXT format data, providing FastAPI HTTP interface, SSE real-time push, and MCP tool encapsulation.

Features

• ✅ Anomaly detection algorithm based on autoencoder
• ✅ Support automatic recognition of CSV and TXT format data
• ✅ FastAPI HTTP Interface Service
• ✅ Server Send Events (SSE) real-time push detection results
• ✅ MCP tool class encapsulation for easy integration
• ✅ Models and thresholds can be saved and loaded
• ✅ Automatic data standardization processing

Install dependencies

pip install -r requirements.txt

Instructions for use

1. Train the model

Train an autoencoder model using normal data:

# 使用默认版本号（时间戳）
python train.py --data normal_data.csv --epochs 100

# 指定版本名称
python train.py --data normal_data.csv --version v1 --epochs 100

# 手动指定所有文件路径（不推荐，会覆盖）
python train.py --data normal_data.csv --model models/model.pth --threshold models/threshold.json --scaler models/scaler.pkl --epochs 100

Parameter description:

• --dataTraining data file path (CSV or TXT)
• --model: Model save path (default: models/model. psh)
• --thresholdThreshold save path (default: models/threshold. json)
• --scalerStandardization tool save path (default: models/scaler. pkl)
• --config: Path for saving model configuration (optional, automatically generated by default: models/modelconfig. json)
• --epochs: Training rounds (default: 100)
• --batch_sizeBatch size (default: 32)
• --learning_rateLearning rate (default: 0.001)
• --encoding_dimEncoding layer dimension (optional, automatically calculated by default)
• --percentileThreshold calculation quantile (default: 95)

After completing the training, you will models/ Generate the following files in the folder:

• models/model.pth: Model weight file
• models/model_config.jsonModel configuration file (including network structure information)
• models/threshold.jsonAbnormal detection threshold
• models/scaler.pkl: Data Standardizer

2. Command line detection

Using a trained model for anomaly detection:

python detect.py --data test_data.csv --model models/model.pth --threshold models/threshold.json --scaler models/scaler.pkl --output results.csv

Parameter description:

• --data: Path of data file to be tested
• --model: Model file path (default: models/model. pth)
• --thresholdThreshold file path (default: models/threshold. json)
• --scalerStandardization file path (default: models/scaler. pkl)
• --outputResult output file path (optional)

3. FastMCP server

Start MCP server (using SSE transmission):

python server.py

The server will be in http://127.0.0.1:3001 Start, automatically provide /sse The endpoint is used for MCP client connections.

MCP tool

The server provides the following MCP tools:

1. run_detection - Perform Exception Detection

{
    "file_path": "test_data.csv",
    "model_path": "models/model.pth",  # 可选，默认 "models/model.pth"
    "threshold_path": "models/threshold.json",  # 可选，默认 "models/threshold.json"
    "scaler_path": "models/scaler.pkl",  # 可选，默认 "models/scaler.pkl"
    "config_path": null,  # 可选
    "return_details": true  # 可选，默认 true
}

The returned result includes:

• status: Status ("success" or "error")
• total_rows: Total number of branches
• anomaly_count: Number of abnormal rows
• anomaly_ratioAbnormal proportion
• thresholdThreshold used
• results: Detailed result list (each line contains index, reconstruction-error, is_anomaly)

2. get_madel_info - Get model information

{
    "model_path": "models/model.pth",  # 可选，默认 "models/model.pth"
    "threshold_path": "models/threshold.json",  # 可选，默认 "models/threshold.json"
    "scaler_path": "models/scaler.pkl",  # 可选，默认 "models/scaler.pkl"
    "config_path": null  # 可选
}

3. train_madel - Train the model

{
    "data_file": "training_data.csv",
    "version": "v1",  # 可选，模型版本名称（默认使用时间戳）
    "model_path": null,  # 可选，默认使用版本号自动生成
    "threshold_path": null,  # 可选，默认使用版本号自动生成
    "scaler_path": null,  # 可选，默认使用版本号自动生成
    "config_path": null,  # 可选
    "epochs": 100,  # 可选，默认 100
    "batch_size": 32,  # 可选，默认 32
    "learning_rate": 0.001,  # 可选，默认 0.001
    "encoding_dim": null,  # 可选
    "percentile": 95  # 可选，默认 95
}

Version management instructions:

• If not specified versionThe system will automatically use timestamps (such as:20241205_143025）
• If the file path is not specified, the system will automatically generate a path based on the version number
• Each training session generates a new version file that does not overwrite the previous model

MCP client connection

FastMCP automatically provides standard MCP SSE endpoints, and clients can connect through the following methods:

http://127.0.0.1:3001/sse

MCP clients (such as Claude Desktop, cursor, etc.) will automatically discover and call available tools.

4. MCP Tools

Use MCP tools for detection:

from mcp_tool import MCPAnomalyDetectionTool, run_detection

# 方式1：使用工具类
tool = MCPAnomalyDetectionTool(
    model_path="models/model.pth",
    threshold_path="models/threshold.json",
    scaler_path="models/scaler.pkl"
)

result = tool.run_detection("test_data.csv")
print(result)

# 方式2：使用便捷函数
result = run_detection("test_data.csv")
print(result)

Return result format:

{
    "status": "success",
    "total_rows": 1000,
    "anomaly_count": 25,
    "anomaly_ratio": 0.025,
    "threshold": 0.123456,
    "results": [
        {
            "index": 0,
            "reconstruction_error": 0.05,
            "is_anomaly": False
        },
        # ...
    ]
}

Data format requirements

• CSV format: Standard comma separated value file
• TXT format: Supports tab, comma, or space separation
• Data requirementsContains at least one column of numerical data, non numerical columns (such as timestamps, IDs) will be automatically ignored

Example data:

timestamp,value1,value2,value3
2024-01-01,1.2,3.4,5.6
2024-01-02,1.3,3.5,5.7
...

Algorithm Principles

1. training phase：

- Train autoencoder with normal data - Autoencoder learns feature representation of normal data - Calculate the reconstruction error distribution of training data - Use the quantile of reconstruction error (default 95%) as the anomaly detection threshold

1. detection phase：

- Input the data to be tested into a trained autoencoder - Calculate reconstruction error (mean square error between original data and reconstructed data) - If the reconstruction error exceeds the threshold, it is judged as abnormal

Precautions

1. The training data should only include normal samples and not abnormal samples
2. The data will be automatically standardized using the standardized parameters used during training
3. The model file, threshold file, and normalizer file need to exist simultaneously for detection
4. If the data dimension is inconsistent with the training, the detection will fail
5. Model configuration file（*_config.json）It will be automatically generated during training to reconstruct the network structure when loading the model
6. Attention: Do not manually specify the -- model, -- threshold, -- scaler parameters to allow the system to automatically generate paths with version numbers. Now all files are in the models/folder and have version numbers, so there will be no more overwrite issues.