🔍 OpenSearch集中式日志服务器

🚀 具有人工智能分析功能的生产就绪集中式测井平台

*通过模型上下文协议（MCP）集成，利用实时安全监控、高级事件关联和智能自动化从Linux和Windows系统收集、处理和分析日志。*

🚀 快速开始 • ✨ 特性 • 🔧 配置 • 📊 仪表盘 • 🛡️ 安全 • 🤝 贡献

______________________________________________________________________

✨ 特性

🔍 多平台日志收集

🐧 Linux系统：Syslog、身份验证日志、应用程序日志、systemd日志
🪟 Windows事件：事件日志、安全事件、通过Winlogbeat的应用程序日志
🌐 网络设备：防火墙、路由器、交换机和安全设备
☁️ 云服务：AWS CloudTrail、Azure日志、谷歌云日志

📊 高级分析引擎

🔍 实时搜索：数十亿日志的亚秒级查询性能
📈 时间序列分析：趋势检测和异常识别
🔗 事件关联：跨系统事件关系映射
🎯 自定义仪表板：基于角色的可视化和报告

🤖 人工智能驱动的智能

🧠 MCP服务器：用于人工智能辅助集成的HTTP API
📱 流媒体分析：高效处理海量数据集
🔮 预测分析：基于机器学习的威胁预测
💬 自然语言查询：通过REST API进行类似SQL的查询

🛡️ 企业安全

⚡ 实时威胁检测：MITRE ATT&CK框架集成
🚨 自动警报：Slack/团队/电子邮件通知
🔐 访问控制：基于角色的身份验证和授权
📋 合规报告：SOC 2、PCI-DSS、HIPAA就绪

______________________________________________________________________

🏗️ 建筑

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Linux Logs    │    │  Windows Events │    │  Network Logs   │
│   (Syslog/TCP)  │    │  (Winlogbeat)   │    │   (UDP/TCP)     │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          └──────────────────────┼──────────────────────┘
                                 │
                    ┌─────────────▼───────────────┐
                    │        Logstash             │
                    │   (Processing Pipeline)     │
                    └─────────────┬───────────────┘
                                  │
                    ┌─────────────▼───────────────┐
                    │      OpenSearch Cluster     │
                    │    (Node1 + Node2 + HA)    │
                    └─────────────┬───────────────┘
                                  │
          ┌───────────────────────┼───────────────────────┐
          │                       │                       │
┌─────────▼──────────┐  ┌─────────▼──────────┐  ┌─────────▼──────────┐
│   MCP HTTP Server  │  │ OpenSearch Dashboard│ │   Direct API       │
│  (AI Integration)  │  │   (Visualizations)  │ │    (REST/SQL)      │
└──────────|─────────┘  └────────────────────┘  └────────────────────┘
           | 
           |
┌─────────▼──────────┐
│   Claud Desktop    │  
│  (AI Integration)  │  
└────────────────────┘

______________________________________________________________________

🚀 快速开始

先决条件

码头工人 和 Docker Compose 安装
4GB+内存 推荐以获得最佳性能
可用端口: 5601, 8000, 9200, 9600, 514, 5044

1.克隆和配置

git clone https://github.com/cybernazmul/OpenLogAI.git
cd OpenLogAi
cp .env_example .env
# Edit .env with your settings

2.启动堆栈

# Start all services
docker-compose up -d

# Check service health
docker-compose ps
docker-compose logs -f

3.克劳德桌面集成

{
  "mcpServers": {
    "opensearch-logs": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://YOUR_SERVER_IP:8000/mcp"
      ]
    }
  }
}

4.访问服务

服务	URL	目的
🔍 OpenSearch API	https://localhost:9200	直接搜索和管理
📊 仪表板	http://localhost:5601	数据可视化
🤖 MCP服务器	http://localhost:8000	AI集成端点
📈 Logstash API	http://localhost:9600	管道监测

5.测试日志收集

# Test syslog ingestion
logger "Test message from $(hostname)"

# Verify in OpenSearch
curl -k -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD} \
  "https://localhost:9200/logstash-logs-*/_search?pretty&size=1"

______________________________________________________________________

🔧 配置

环境变量

所有配置都通过环境变量进行管理。复制 .env_example 向 .env 并自定义：

# OpenSearch Cluster
OPENSEARCH_INITIAL_ADMIN_PASSWORD=your_secure_password
OS_URL=https://opensearch-node1:9200
OS_USERNAME=admin
OS_PASSWORD=your_secure_password

# MCP Server Settings
MCP_HOST=0.0.0.0
MCP_PORT=8000
MCP_PATH=/mcp

# Optional: API Key authentication
OS_API_KEY=your_api_key_here

日志源配置

📧 Linux系统

# Add to /etc/rsyslog.conf or /etc/rsyslog.d/50-logstash.conf
*.* @@YOUR_SERVER_IP:514

# For specific facilities
auth.* @@YOUR_SERVER_IP:514
kern.* @@YOUR_SERVER_IP:514
mail.* @@YOUR_SERVER_IP:514

# Restart rsyslog
sudo systemctl restart rsyslog

🪟 Windows系统

# Install and configure Winlogbeat
# Download from: https://www.elastic.co/downloads/beats/winlogbeat

# winlogbeat.yml configuration:
winlogbeat.event_logs:
  - name: Application
    level: error, warning
  - name: Security
    event_id: 4624, 4625, 4648, 4656, 4719, 4720
  - name: System
  - name: Microsoft-Windows-Sysmon/Operational

output.logstash:
  hosts: ["YOUR_SERVER_IP:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

🐳 Docker容器

# Add to your docker-compose.yml services
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
    labels: "service,environment"

# Or use Filebeat for Docker logs

🌐 网络设备

# Cisco/Juniper/Fortigate syslog forwarding
logging YOUR_SERVER_IP:514

# pfSense/OPNsense
Configure System > Advanced > Logging
Remote Logging Options: YOUR_SERVER_IP:514

______________________________________________________________________

📊 仪表板和可视化

预构建仪表板类别

🔒 安全运营中心（SOC）：登录失败、权限升级、可疑活动
💻 系统健康：CPU、内存、磁盘使用率、服务状态
🌐 网络分析：流量模式、连接监控、防火墙事件
📱 应用程序日志：错误率、性能指标、用户活动
📋 合规：审计跟踪、访问日志、监管报告

示例查询

OpenSearch REST API

# Search recent security events
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/logstash-logs-*/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "bool": {
        "must": [
          {"range": {"@timestamp": {"gte": "now-1h"}}},
          {"match": {"syslog_program": "sshd"}}
        ]
      }
    },
    "size": 100
  }'

通过MCP进行SQL查询

-- Find failed SSH attempts
SELECT timestamp, source_ip, syslog_message 
FROM "logstash-logs-*" 
WHERE syslog_message LIKE '%Failed password%' 
AND timestamp > DATE_SUB(NOW(), INTERVAL 1 HOUR)
ORDER BY timestamp DESC;

-- Windows login analysis
SELECT user_name, logon_type, source_ip, COUNT(*) as login_count
FROM "windows-events-*"
WHERE event_id = 4624
  AND timestamp > DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY user_name, logon_type, source_ip
ORDER BY login_count DESC;

-- Top error producers
SELECT syslog_server, syslog_program, COUNT(*) as error_count
FROM "logstash-logs-*" 
WHERE syslog_severity  DATE_SUB(NOW(), INTERVAL 1 HOUR)
GROUP BY syslog_server, syslog_program
ORDER BY error_count DESC
LIMIT 20;

______________________________________________________________________

🛡️ 安全功能

🎯 自动威胁检测

暴力检测：>5分钟内登录失败5次
横向移动：不寻常的跨系统访问模式
权限提升: sudo, su，管理员组更改
数据渗漏：大额出站转账或异常访问模式
恶意软件指示器：流程执行异常、文件修改

📊 安全监控规则

关键警报

# Failed Authentication Attempts
- Query: syslog_message:"Failed password" OR event_id:4625
- Threshold: >5 occurrences in 5 minutes
- Action: Immediate alert + IP blocking

# Administrative Account Usage
- Query: user_name:"admin" OR user_name:"root" 
- Time: Outside business hours (6PM-8AM)
- Action: Security team notification

# New Process Execution
- Query: event_id:4688 AND process_name:NOT_IN_WHITELIST
- Scope: Critical servers only
- Action: Investigation workflow trigger

警告警报

# Unusual Login Locations
- Geographic IP analysis for user accounts
- First-time login locations
- VPN vs direct connections

# File System Changes
- Sensitive directory access (/etc, /var/log, C:\Windows\System32)
- Configuration file modifications
- Log file tampering attempts

🔐 访问控制

# Create role-based users
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/_plugins/_security/api/internalusers/analyst" \
  -H 'Content-Type: application/json' \
  -d '{
    "password": "analyst_password",
    "roles": ["logs_reader", "dashboard_user"]
  }'

______________________________________________________________________

📈 性能和扩展

资源需求

部署	RAM	CPU	存储	每日日志量
发展	4GB	2核	20GB	\5 failures in 5 minutes per source IP

Response: Auto-block + SOC alert

Privilege Escalation

Pattern: sudo/su usage, admin group additions
Scope: Non-service accounts
Response: Immediate investigation trigger

Suspicious Process Execution

Pattern: PowerShell/cmd.exe with encoded commands
Pattern: Unusual parent-child process relationships
Response: Process analysis + containment

Data Access Anomalies

Pattern: Large file transfers outside business hours
Pattern: Access to sensitive directories
Response: Data loss prevention workflow


#### Windows特定检测

Event ID Monitoring

4624: Successful logon (track unusual locations/times) 4625: Failed logon (brute force detection) 4648: Explicit credential use (lateral movement) 4656: File/object access (data access monitoring) 4719: System audit policy changes (tampering detection) 4720: User account created (account management)

PowerShell Security

4103: Module logging (malicious command detection) 4104: Script block logging (obfuscated script analysis) 4105: Script start (execution monitoring) 4106: Script stop (completion tracking)


### 🔐 **身份验证和访问控制**

Create security roles

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_plugins/_security/api/roles/security_analyst" \ -H 'Content-Type: application/json' \ -d '{ "cluster_permissions": ["cluster_composite_ops_ro"], "index_permissions": [{ "index_patterns": ["logstash-logs-*", "security-events-*"], "allowed_actions": ["read", "search"] }] }'

Create users with specific roles

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_plugins/_security/api/internalusers/analyst1" \ -H 'Content-Type: application/json' \ -d '{ "password": "secure_password", "roles": ["security_analyst"], "attributes": { "department": "Security", "clearance_level": "L2" } }'


______________________________________________________________________

## 🔧 高级配置

### 高可用性设置

Add load balancer

nginx: image: nginx:alpine ports: - "80:80" - "443:443" volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf - ./nginx/ssl:/etc/nginx/ssl depends_on: - opensearch-node1 - opensearch-node2

Cross-cluster replication

opensearch-remote: image: opensearchproject/opensearch:latest environment: - cluster.name=opensearch-backup - replication.enabled=true


### 自定义日志处理管道

Enhanced logstash pipeline for Windows events

filter { if [agent][type] == "winlogbeat" { # Windows Event ID classification if [winlog][event_id] in [4624, 4634] { mutate { add_tag => ["authentication"] } } if [winlog][event_id] in [4625, 4648] { mutate { add_tag => ["failed_auth", "security_alert"] } } if [winlog][event_id] in [4656, 4658, 4660, 4663] { mutate { add_tag => ["file_access", "audit"] } }

# Extract user and computer information grok { match => { "[winlog][event_data][SubjectUserName]" => "(?[^$]+)" } tag_on_failure => ["_grok_user_failure"] }

# GeoIP enrichment for external IPs if [winlog][event_data][IpAddress] and [winlog][event_data][IpAddress] !~ /^(10\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\.)/ { geoip { source => "[winlog][event_data][IpAddress]" target => "geoip" database => "/usr/share/GeoIP/GeoLite2-City.mmdb" } } } }

Output to specialized indices

output { if "security_alert" in [tags] { opensearch { index => "security-events-%{+YYYY.MM.dd}" # ... connection details } } else if "authentication" in [tags] { opensearch { index => "auth-logs-%{+YYYY.MM.dd}" # ... connection details } } else { opensearch { index => "logstash-logs-%{+YYYY.MM.dd}" # ... connection details } } }


______________________________________________________________________

## 📊 监控和警报

### 健康监测

Service health checks

docker-compose exec opensearch-node1 curl -k \ "https://localhost:9200/_cluster/health?pretty"

Logstash pipeline monitoring

curl "http://localhost:9600/_node/stats/pipelines?pretty"

MCP server status

curl "http://localhost:8000/health"


### 性能指标

Index statistics

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_cat/indices/logstash-logs-*?v&s=store.size:desc"

Search performance

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_nodes/stats/indices/search?pretty"

Memory usage

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu"


______________________________________________________________________

## 🔧 故障排除

### 常见问题

#### OpenSearch无法启动

Check logs

docker-compose logs opensearch-node1

Common fixes

sysctl -w vm.max_map_count=262144 echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

Verify disk space

df -h


#### Logstash处理问题

Check pipeline status

curl "http://localhost:9600/_node/stats/pipelines?pretty"

Monitor processing rates

docker-compose logs logstash | grep -E "(ERROR|WARN)"

Test configuration

docker-compose exec logstash logstash --config.test_and_exit


#### MCP服务器连接问题

Test MCP connectivity

curl http://localhost:8000/health

Check OpenSearch connectivity from MCP

docker-compose exec mcp-server python -c " from mcp_tools import OpenSearchClient client = OpenSearchClient('https://opensearch-node1:9200', 'admin', '${PASSWORD}') print(client.health_check()) "


### 日志分析命令

Find parsing failures

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/logstash-logs-*/_search?q=tags:_grokparsefailure"

Monitor ingestion rates

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_cat/indices/logstash-logs-*?v&s=docs.count:desc"

Check for duplicate events

curl -k -u admin:${PASSWORD} -X POST \ "https://localhost:9200/logstash-logs-*/_search" \ -d '{"aggs":{"duplicates":{"terms":{"field":"message.keyword","size":10}}}}'


______________________________________________________________________

## 🚀 生产部署

### 安全强化

SSL/TLS Configuration

opensearch-node1: environment: - plugins.security.ssl.http.enabled=true - plugins.security.ssl.http.pemcert_filepath=certs/node1.pem - plugins.security.ssl.http.pemkey_filepath=certs/node1-key.pem volumes: - ./certs:/usr/share/opensearch/config/certs:ro

Firewall rules (UFW example)

sudo ufw allow from TRUSTED_NETWORK to any port 9200 sudo ufw allow from TRUSTED_NETWORK to any port 5601 sudo ufw allow 514/tcp # Syslog sudo ufw allow 5044/tcp # Beats


### 备份和恢复

Snapshot configuration

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_snapshot/backup_repository" \ -H 'Content-Type: application/json' \ -d '{ "type": "fs", "settings": { "location": "/usr/share/opensearch/backups", "compress": true } }'

Automated daily snapshots

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_plugins/_ism/policies/snapshot_policy" \ -H 'Content-Type: application/json' \ -d '{ "policy": { "description": "Daily snapshots", "default_state": "snapshot", "states": [{ "name": "snapshot", "actions": [{"snapshot": {"snapshot": "daily-{now/d}"}}], "transitions": [{"state_name": "snapshot", "conditions": {"cron": {"cron": {"expression": "0 2 * * *"}}}}] }] } }'


______________________________________________________________________

## 📚 文档和资源

### 📖 其他文件

- [🔧 高级Logstash配置](./docs/logstash-advanced.md)
- [🪟 Windows事件收集设置](./docs/windows-setup.md)
- [🐧 Linux日志源配置](./docs/linux-setup.md)
- [🛡️ 安全规则和警报](./docs/security-rules.md)
- [🚀 生产部署指南](./docs/production.md)
- [🎯 api参考](./docs/api-reference.md)

### 🔗 外部资源

- [OpenSearch文档](https://opensearch.org/docs/)
- [Logstash配置参考](https://www.elastic.co/guide/en/logstash/current/configuration.html)
- [FastMCP文档](https://github.com/jlowin/fastmcp)
- [MITRE ATT&CK框架](https://attack.mitre.org/)
- [SIGMA规则](https://github.com/SigmaHQ/sigma)

### 🎓 学习资源

- [OpenSearch查询DSL](https://opensearch.org/docs/latest/query-dsl/)
- [Grok模式测试](https://grokdebug.herokuapp.com/)
- [正则表达式测试](https://regex101.com/)

______________________________________________________________________

## 🤝 贡献

我们欢迎捐款！请阅读我们的 [贡献指南](CONTRIBUTING.md) 在提交PR之前。

### 开发设置

Clone repository

git clone https://github.com/your-username/opensearch-centralized-logs.git cd opensearch-centralized-logs

Setup development environment

cp .env_example .env docker-compose -f docker-compose.dev.yml up -d

Run tests

make test-all

Lint code

make lint

Generate documentation

make docs


### 🐛 问题模板

- **错误报告**：用于功能问题
- **功能请求**：建议新功能
- **安保问题**：私下报告安全漏洞
- **文档**：改进文档和示例

______________________________________________________________________

## 📄 许可证

此项目根据MIT许可证获得许可-请参阅 [许可证](LICENSE) 文件以获取详细信息。

______________________________________________________________________

## 🌟 支持与社区

- 📧 **问题**: 
- 💬 **讨论**: 
- 📖 **维基**: [维基工程](https://github.com/your-username/opensearch-centralized-logs/wiki)
- 🆘 **支持**: [社区论坛](https://github.com/your-username/opensearch-centralized-logs/discussions/categories/q-a)

### 🏆 贡献者

感谢我们出色的贡献者！看 [贡献者.md](CONTRIBUTORS.md) 查看完整列表。

______________________________________________________________________

## 🎯 路线图

### 📅 **2025年第一季度**

- \[\]增强的Windows事件解析
- \[\]基于机器学习的异常检测
- \[\]移动设备日志集成
- \[\]高级合规报告

### 📅 **2025年第二季度**

- \[\]Kubernetes日志收集
- \[\]云原生部署选项
- \[\]高级关联引擎
- \[\]自定义插件市场

### 📅 **2025年第3季度**

- \[\]多租户架构
- \[\]企业SSO集成
- \[\]高级威胁追踪工具
- \[\]自动事件响应

______________________________________________________________________

**⭐ 如果你觉得这个存储库有用，请为其加星！**

**🔗  | 📖 [文档](./docs/) | 🚀 [开始使用](#-quick-start)**

*内置于❤️ 面向网络安全和DevOps社区*

Open Log AI

快速接入

详细介绍

🔍 OpenSearch集中式日志服务器

✨ 特性

🔍 多平台日志收集

📊 高级分析引擎

🤖 人工智能驱动的智能

🛡️ 企业安全

🏗️ 建筑

🚀 快速开始

先决条件

1.克隆和配置

2.启动堆栈

3.克劳德桌面集成

4.访问服务

5.测试日志收集

🔧 配置

环境变量

日志源配置

📧 Linux系统

🪟 Windows系统

🐳 Docker容器

🌐 网络设备

📊 仪表板和可视化

预构建仪表板类别

示例查询

OpenSearch REST API

通过MCP进行SQL查询

🛡️ 安全功能

🎯 自动威胁检测

📊 安全监控规则

关键警报

警告警报

🔐 访问控制

📈 性能和扩展

资源需求

Privilege Escalation

Suspicious Process Execution

Data Access Anomalies

Event ID Monitoring

PowerShell Security

Create security roles

Create users with specific roles

Add load balancer

Cross-cluster replication

Enhanced logstash pipeline for Windows events

Output to specialized indices

Service health checks

Logstash pipeline monitoring

MCP server status

Index statistics

Search performance

Memory usage

Check logs

Common fixes

Verify disk space

Check pipeline status

Monitor processing rates

Test configuration

Test MCP connectivity

Check OpenSearch connectivity from MCP

Find parsing failures

Monitor ingestion rates

Check for duplicate events

SSL/TLS Configuration

Firewall rules (UFW example)

Snapshot configuration

Automated daily snapshots

Clone repository

Setup development environment

Run tests

Lint code

Generate documentation

目录标签

接入字段

权限和风险

来源信息

继续浏览同类 MCP