准备docker镜像

elk: sebp/elk latest

docker pull sebp/elk

sebp/elk 文档说明  https://elk-docker.readthedocs.io/

filebeat: filebeat-:7.9.3

docker pull elastic/filebeat:7.9.3

使用以下命令从映像运行容器：

$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

配置logstash

进入容器 sudo docker exec -it elk /bin/bash

修改02-beats-input.conf

vim /etc/logstash/conf.d/02-beats-input.conf

删除 ssl => true

删除 ssl_certificate => "/pki/tls/certs/logstash.crt"

删除 ssl_key => "/pki/tls/private/logstash.key"

注意 1：将以下三行删除掉，这三行的意思是是否使用证书，本例是不使用证书的。如果你需要使用证书，将logstash.crt拷贝到客户端，然后在filebeat.yml里面添加路径即可。

注意 2：sebp/elk docker是自建立了一个证书logstash.crt，默认使用*通配配符，如果你使用证书，filebeat.yml使用的服务器地址必须使用域名，不能使用IP地址，否则会报错。

运行Filebeat容器

docker run --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2  --name filebeat --user=root -d  -v /Users/chentong/Desktop/logs:/logs/ elastic/filebeat:7.9.3

修改Filebeat配置收集多日志并拆分

vim /usr/share/filebeat/filebeat.yml

[2020-11-19 11:49:26 +0800] [9284] [ERROR] Connection in use: ('0.0.0.0', 8002)
[2020-11-06 19:19:27 +0800] [1941] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 203, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/python3/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/home/wl/django3/chentong/my_aleks/my_aleks/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
  File "/usr/local/python3/lib/python3.7/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/usr/local/python3/lib/python3.7/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/python3/lib/python3.7/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/usr/local/python3/lib/python3.7/site-packages/channels/apps.py", line 11, in ready
    monkeypatch_django()
  File "/usr/local/python3/lib/python3.7/site-packages/channels/hacks.py", line 20, in monkeypatch_django
    BaseHandler.old_handle_uncaught_exception = BaseHandler.handle_uncaught_exception
AttributeError: type object 'BaseHandler' has no attribute 'handle_uncaught_exception'
[2020-11-19 11:49:26 +0800] [9284] [ERROR] Connection in use: ('0.0.0.0', 8002)

multiline 为配置多行合并 不以 “ ^\[\d{4}-\d{2}-\d{2} ” 的行都合并到上一行的末  这里是按[2020-11-19 为一个错误拆分

• pattern：正则表达式
• negate：true 或 false；默认是false，匹配pattern的行合并到上一行；true，不匹配pattern的行合并到上一行
• match：after 或 before，合并到上一行的末尾或开头
• max_lines：合并最大行，默认500
• timeout：一次合并事件的超时时间，默认5s，防止合并消耗太多时间甚至卡死 

#=========================== filebeat =============================

filebeat.inputs:
- type: log
  enabled: true
  multiline.pattern: '^\[\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["error"]

  paths:
  - /logs/err.log


- type: log
  enabled: true
  multiline.pattern: '^\[\d{4}-\d{2}-\d{2}'
  multiline.negate: true
  multiline.match: after
  tags: ["info"]
  paths:
    - /logs/info.log


output.logstash:
  hosts: ['172.17.0.2:5044']

logstash配置filebeat日志

#=========================== logstash =============================

input {
  beats {
    port => 5044
  }
}


output {
  #istdout { codec => rubydebug }
  #日志异常信息通过钉钉告警
  http {
    url => "https://oapi.dingtalk.com/robot/send?access_token=488298742547xxxxxxxxxxxxx"
    http_method => "post" 
    #这个mapping很重要，决定了http请求体的数据，要和钉钉上报格式一致
    mapping => {
      "msgtype" => "text" 
      "text" => "{'content':'clogHttpAgent--%{host}--%{message}'}"
    }

  if [tags][0] == "info" {
    elasticsearch {
      hosts => ["localhost"]
      manage_template => true
      index => "logstash-server-access-%{+YYYY-MM-dd}"
    }
  } else if [tags][0] == "error" {
    elasticsearch {
      hosts => ["localhost"]
      manage_template => true
      index => "logstash-server-error-%{+YYYY-MM-dd}"
    }
  }
}

kibana配置