网站开发服务器架设心馨人生网站建设设计

网站开发服务器架设,心馨人生网站建设设计,郑州狼牙网页设计公司,wordpress 含演示数据库文章目录ELK企业日志分析系统1.概述1.1前言1.2.ELK工作原理展示图#xff1a;1.3.知识点概述**核心概念一览**Elasticsearch#xff1a;分布式搜索和分析引擎详细作用#xff1a;Logstash#xff1a;服务器端数据处理管道详细作用#xff1a;Kibana#xff1a;数据可视化…文章目录ELK企业日志分析系统1.概述1.1前言1.2.ELK工作原理展示图1.3.知识点概述**核心概念一览**Elasticsearch分布式搜索和分析引擎详细作用Logstash服务器端数据处理管道详细作用Kibana数据可视化与管理平台详细作用2.案例环境:配置ELK日志分析系统3.配置elasticsearch环境准备主机名、域名、java环境部署elasticsearch软件4.安装logstash安装Apahce服务httpd安装Java环境安装logstashlogstashapache节点与elasticsearchnode节点做对接测试登录192.168.108.43 在Apache服务器上输入采用标准输入 输出采用标准输出使用rubydebug显示详细输出codec为一种编解码器笔记本192.168.108.1 登陆5.安装KibanaELK企业日志分析系统1.概述1.1前言日志分析是运维工程师解决系统故障发现问题的主要手段。日志主要包括系统日志、应用程序日志和安全日志。系统运维和开发人员可以通过日志了解服务器软硬件信息、检查配置过程中的错误及错误发生的原因。经常分析日志可以了解服务器的负荷性能安全性从而及时采取措施纠正错误。通常日志被分散的储存在不同的设备上。如果你管理数十上百台服务器你还在使用依次登录每台机器的传统方法查阅日志即繁琐又效率低下。为此我们可以使用集中化的日志管理例如开源的syslog将所有服务器上的日志收集汇总。集中化管理日志后日志的统计和检索又成为一件比较麻烦的事情一般我们使用grep、awk和wc等Linux命令能实现检索和统计但是对于更高要求的查询、排序和统计等再加上庞大的机器数量使用这样的方法依然难免有点力不从心。开源实时日志分析ELK平台能够完美的解决我们上述的问题ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成。官方网站https://www.elastic.co/productsElasticsearch是个开源分布式搜索引擎它的特点有分布式零配置自动发现索引自动分片索引副本机制restful风格接口多数据源自动搜索负载等。Logstash是一个完全开源的工具它可以对你的日志进行收集、过滤并将其存储供以后使用如搜索。Kibana 也是一个开源和免费的工具Kibana可以为 Logstash 和 ElasticSearch 提供友好的日志分析Web 界面可以帮助您汇总、分析和搜索重要数据日志。1.2.ELK工作原理展示图【APPServer集群】→→【logstash Agent 采集器】→→【ElasticSearch Cluster】→→【Kibana Server】→→【Browser】Logstash收集AppServer产生的Log并存放到ElasticSearch集群中而Kibana则从ES集群中查询数据生成图表再返回给Browser。简单来说进行日志处理分析一般需要经过以下几个步骤将日志进行集中化管理beats将日志格式化logstash对格式化后的数据进行索引和存储elasticsearch前端数据的展示kibana1.3.知识点概述核心概念一览首先一个简单的类比来帮助你理解它们之间的关系想象一下银行系统Logstash就像是各地的柜台职员和运钞车。它负责从各个来源ATM、柜台、网上银行收集货币数据进行清点、辨别真伪、分类整理解析和转换然后运送到总行金库。Elasticsearch就是银行的巨型、高科技、超高效的金库和归档系统。它不仅安全地存储所有货币数据还建立了一套极其精细的索引。当需要查询时它能在一瞬间从海量库存中精准地找到任何一张钞票或任何一笔交易记录。Kibana则是银行的数据分析和监控中心的大屏幕。它连接到金库Elasticsearch让管理人员和分析师可以通过各种图表、仪表盘、地图来可视化金融趋势、交易流水、风险状况而不是去面对一堆枯燥的数字表格。这个系统的核心数据流通常是日志/数据源→Logstash收集、处理→Elasticsearch存储、索引、搜索→Kibana可视化、分析Elasticsearch分布式搜索和分析引擎核心角色大脑和心脏-负责海量数据的存储、检索和分析。详细作用分布式存储与高可用性Elasticsearch 是一个分布式数据库。数据会被分散存储 across 多个服务器节点上。这意味着它可以处理PB****级别的海量数据并且通过副本replicas机制即使某个硬件发生故障数据也不会丢失服务也不会中断实现了高可用性。近乎实时的搜索这是它最核心的能力。数据从被写入到可以被搜索只有毫秒级的延迟约1秒内。这对于需要实时监控和告警的场景至关重要。强大的全文搜索引擎基于 Apache Lucene 构建提供了无比强大和灵活的全文搜索功能。它不仅仅是简单的关键字匹配还支持模糊搜索即使拼写错误也能找到结果。同义词处理搜索“手机”也能找到包含“电话”的结果。相关性评分根据多种因素对搜索结果进行排序将最相关的结果排在前面。先进的索引技术数据在存储时会被倒排索引Inverted Index。这是一种类似书籍最后“索引”页的技术例如提到“数据库”这个词的页面是 10, 23, 45。这使得它的查询速度极快远远优于传统的逐行扫描数据库。丰富的RESTful API几乎所有操作如数据的写入、查询、集群管理都可以通过简单的 HTTP REST API 进行这使得它非常容易与其他系统集成和开发。强大的聚合分析能力除了搜索它还能执行复杂的聚合Aggregation操作相当于 SQL 中的 GROUP BY 再加上很多数学运算。例如“统计过去5分钟内错误日志的数量并按应用分组”“计算网站访问量的95百分位延迟”“绘制用户地理位置的分布图”简单总结Elasticsearch 是一个专门为快速查找而设计的海量数据存储系统。它不擅长频繁更新事务像MySQL那样但极其擅长快速过滤、查询和分析巨量数据。Logstash服务器端数据处理管道核心角色搬运工和预处理车间-负责收集、解析、转换和输送数据。详细作用Logstash 的工作流程被划分为三个阶段Input→Filter→Output。Input输入数据收集从各种来源实时地收集数据和日志。它拥有大量的插件支持极其丰富的输入源例如文件如 *.log 系统日志Syslog消息队列如 Kafka, Redis, RabbitMQ数据库如 MySQL, PostgreSQL监控数据如 Beats、JMX网络协议如 TCP/UDP, HTTPFilter过滤数据解析与 enrichment这是 Logstash 最强大的部分。原始日志比如一行文本通常是非结构化的Filter 插件将其解析成结构化的、有意义的字段以便于在 Elasticsearch 中高效查询。常见操作包括Grok使用正则模式匹配来解析复杂的文本行。例如从日志 “127.0.0.1 - -[10/Oct/2023:14:16:39 0000] “GET / HTTP/1.1” 200 3574” 中解析出client_ip , timestamp , http_method , response_code 等字段。Mutate对字段进行修改如重命名、删除、替换、转换数据类型。Date解析时间戳并设置为事件的正式 timestamp 字段。GeoIP根据 IP 地址查询出对应的国家、城市和经纬度信息极大地丰富了数据维度。KV解析 keyvalue 这样的数据。Output输出数据发送将处理好的数据发送到指定的目的地。最常用的输出就是Elasticsearch。但它也支持输出到其他数据库如 MongoDB消息队列如 Kafka文件系统监控系统如 Nagios电子邮件等简单总结Logstash 是一个数据流引擎负责将杂乱无章的原始数据加工成干净、结构化的数据然后喂给 Elasticsearch。它是一个重量级工具功能全面但资源消耗相对较高。小提示在需要轻量级数据收集的场景下Elastic 提供了Beats如 Filebeat, Metricbeat来替代 Logstash 的 Input 功能它们更轻量、更专一通常将数据直接发送给 Logstash 做进一步处理或直接发送给 Elasticsearch。Kibana数据可视化与管理平台核心角色窗口和仪表盘-用于数据的探索、可视化和交互。详细作用数据探索与发现Discover提供一个交互式界面让你能够直接查询和浏览存储在 Elasticsearch 中的原始数据。你可以使用搜索框支持 Lucene 查询语法或 KQL添加过滤器来动态地缩小数据范围并查看匹配文档的详细内容。可视化Visualize提供丰富的可视化组件让你可以将数据转化为各种直观的图表包括柱状图、折线图、饼图指标看板显示单个数字如总错误数数据表用于聚合数据坐标地图与 GeoIP 结合在地图上显示数据点热力图等仪表盘Dashboard可以将多个可视化组件拖放组合成一个统一的、综合的仪表盘。这对于构建实时监控大屏非常有用运维和开发团队可以在一个屏幕上全局掌握系统的健康状态、性能指标和业务趋势。管理和维护Kibana 还提供了Stack Management界面用于管理和配置整个 Elastic Stack例如索引生命周期管理ILM用户和角色权限控制导入/导出保存的可视化和仪表盘配置 Kibana 本身简单总结Kibana 是 ELK Stack 的“面子”它将 Elasticsearch 中深奥的数据以人类易于理解的图形方式呈现出来使得任何人都能轻松地进行数据分析和决策而无需编写复杂的查询代码。这就是 ELK/Elastic Stack 的强大之处——它将数据的收集、存储和可视化三个环节无缝衔接提供了一个端到端的解决方案广泛应用于日志分析、应用性能监控APM、安全信息与事件管理SIEM、业务指标分析等多个领域。2.案例环境:配置ELK日志分析系统实验拓扑图配置和安装ELK日志分析系统安装集群方式2个elasticsearch节点并监控apache服务器日志主机操作系统主机名IP地址主要软件服务器Centos7.9node1192.168.108.41/24Elasticsearch Kibana服务器Centos7.9node2192.168.108.42/24Elasticsearch服务器Centos7.9apache192.168.108.43/24Logstash Apache3.配置elasticsearch环境准备主机名、域名、java环境登录192.168.108.41node1 更改主机名 配置域名解析 查看Java环境[rootnode1 ~]# systemctl stop firewalld.service[rootnode1 ~]# vim /etc/hosts[rootnode1 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.108.41 node1192.168.108.42 node2[rootnode1 ~]# java -versionopenjdk version1.8.0_262OpenJDK Runtime Environment(build1.8.0_262-b10)OpenJDK64-Bit Server VM(build25.262-b10, mixed mode)登录192.168.108.42node2 更改主机名 配置域名解析 查看Java环境[rootnode2 ~]# systemctl stop firewalld.service[rootnode2 ~]# vim /etc/hosts[rootnode2 ~]# cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.108.41 node1192.168.108.42 node2[rootnode2 ~]# java -versionopenjdk version1.8.0_262OpenJDK Runtime Environment(build1.8.0_262-b10)OpenJDK64-Bit Server VM(build25.262-b10, mixed mode)部署elasticsearch软件1.登录192.168.108.41node1安装elasticsearch—rpm包上传elasticsearch-5.5.0.rpm到/opt目录下[rootnode1 ~]# cd /opt[rootnode1 opt]# lsrh[rootnode1 opt]# lselasticsearch-5.5.0.rpm rh[rootnode1 opt]# rpm -ivh elasticsearch-5.5.0.rpmwarning: elasticsearch-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing...################################# [100%]Creating elasticsearch group... OK Creating elasticsearch user... OK Updating / installing...1:elasticsearch-0:5.5.0-1################################# [100%]### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemdsudosystemctl daemon-reloadsudosystemctlenableelasticsearch.service### You can start elasticsearch service by executingsudosystemctl start elasticsearch.service加载系统服务[rootnode1 opt]# systemctl daemon-reload[rootnode1 opt]# systemctl enable elasticsearch.serviceCreated symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.更改elasticsearch主配置文件[rootnode1 opt]# cd[rootnode1 ~]# cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak[rootnode1 ~]# vim /etc/elasticsearch/elasticsearch.yml17cluster.name: my-elk-cluster#集群名字23node.name: node1#节点名字33path.data: /data/elk_data#数据存放路径37path.logs: /var/log/elasticsearch/#日志存放路径43bootstrap.memory_lock:false#不在启动的时候锁定内存锁定物理内存地址防止es内存被交换出去也就是避免es使用swap交换分区频繁的交换会导致IOPS变高。55network.host:0.0.0.0#提供服务绑定的IP地址0.0.0.0代表所有地址59http.port:9200#侦听端口为920068discovery.zen.ping.unicast.hosts:[node1,node2]#集群发现通过单播实现创建数据存放路径并授权[rootnode1 ~]# mkdir -p /data/elk_data[rootnode1 ~]# chown elasticsearch:elasticsearch /data/elk_data/启动elasticsearch是否成功开启[rootnode1 ~]# systemctl start elasticsearch.service[rootnode1 ~]# netstat -antp |grep 9200tcp600:::9200 :::* LISTEN51240/java查看节点信息 用真机192.168.108.1 的浏览器打开 http://192.168.108.41:9200 有文件打开 下面是节点的信息{name:node1,cluster_name:my-elk-cluster,cluster_uuid:g78kS99pTu2xYqCKRih86A,version:{number:5.5.0,build_hash:260387d,build_date:2017-06-30T23:16:05.735Z,build_snapshot:false,lucene_version:6.6.0},tagline:You Know, for Search2.登录192.168.108.42安装elasticsearch—rpm包上传elasticsearch-5.5.0.rpm到/opt目录下[rootnode2 opt]# cd /opt[rootnode2 opt]# rpm -ivh elasticsearch-5.5.0.rpmwarning: elasticsearch-5.5.0.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing...################################# [100%]Creating elasticsearch group... OK Creating elasticsearch user... OK Updating / installing...1:elasticsearch-0:5.5.0-1################################# [100%]### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemdsudosystemctl daemon-reloadsudosystemctlenableelasticsearch.service### You can start elasticsearch service by executingsudosystemctl start elasticsearch.service加载系统服务[rootnode2 opt]# systemctl daemon-reload[rootnode2 opt]# systemctl enable elasticsearch.serviceCreated symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.更改elasticsearch主配置文件[rootnode2 opt]# cd[rootnode2 ~]# cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak[rootnode2 ~]# vim /etc/elasticsearch/elasticsearch.yml17cluster.name: my-elk-cluster####集群名字23node.name: node2####节点名字33path.data: /data/elk_data####数据存放路径37path.logs: /var/log/elasticsearch/####日志存放路径43bootstrap.memory_lock:false####不在启动的时候锁定内存55network.host:0.0.0.0####提供服务绑定的IP地址0.0.0.0代表所有地址59http.port:9200####侦听端口为920068discovery.zen.ping.unicast.hosts:[node1,node2]####集群发现通过单播实现创建数据存放路径并授权[rootnode2 ~]# mkdir /data/elk_data -p[rootnode2 ~]# chown elasticsearch:elasticsearch /data/elk_data/启动elasticsearch是否成功开启[rootnode2 ~]# systemctl start elasticsearch.service [rootnode2 ~]# netstat -antp |grep 9200tcp600:::9200 :::* LISTEN51674/java查看节点信息 用真机192.168.108.41 的浏览器打开 http://192.168.108.42:9200 有文件打开 下面是节点的信息{name:node2,cluster_name:my-elk-cluster,cluster_uuid:g78kS99pTu2xYqCKRih86A,version:{number:5.5.0,build_hash:260387d,build_date:2017-06-30T23:16:05.735Z,build_snapshot:false,lucene_version:6.6.0},tagline:You Know, for Search3.集群检查健康和状态在真机浏览器192.168.108.41 打开http://192.168.108.41:9200/_cluster/health?pretty检查群集健康情况{cluster_name:my-elk-cluster,status:green,timed_out:false,number_of_nodes:2,number_of_data_nodes:2,active_primary_shards:0,active_shards:0,relocating_shards:0,initializing_shards:0,unassigned_shards:0,delayed_unassigned_shards:0,number_of_pending_tasks:0,number_of_in_flight_fetch:0,task_max_waiting_in_queue_millis:0,active_shards_percent_as_number:100.0在真机浏览器192.168.108.1 打开http://192.168.108.41:9200/_cluster/state?pretty检查群集状态信息{cluster_name:my-elk-cluster,version:3,state_uuid:NDvZBaCrSqqhSW8Th-8d9A,master_node:DdKpI-_CQZWCgcU1Xs9TBw,blocks:{},nodes:{DdKpI-_CQZWCgcU1Xs9TBw:{name:node2,ephemeral_id:9CUNsIw9Q8qrqMDJAubtGw,transport_address:192.168.108.42:9300,attributes:{}},kIHczsycTI-liz7TXBRMqg:{name:node1,ephemeral_id:7CM6JJujQoCVeG3LX8l5uQ,transport_address:192.168.108.41:9300,attributes:{}}},metadata:{cluster_uuid:g78kS99pTu2xYqCKRih86A,templates:{},indices:{},index-graveyard:{tombstones:[]}},routing_table:{indices:{}},routing_nodes:{unassigned:[],nodes:{DdKpI-_CQZWCgcU1Xs9TBw:[],kIHczsycTI-liz7TXBRMqg:[]}}}4.安装elasticsearch-head插件查看上述集群的方式及其不方便我们可以通过安装elasticsearch-head插件后来管理集群登录192.168.108.41 node1主机上传node-v8.2.1.tar.gz到/opt[rootnode1 ~]# yum install -y gcc gcc-c make -y编译安装node组件依赖包耗时比较长 30分钟[rootnode1 ~]# cd /opt[rootnode1 opt]# tar xzzvf node-v8.2.1.tar.gz[rootnode1 opt]# cd node-v8.2.1/[rootnode1 node-v8.2.1]# ./configure[rootnode1 node-v8.2.1]# make -j4[rootnode1 node-v8.2.1]# make install5.安装phantomjs端框架上传软件包到/usr/local/src/[rootnode1 node-v8.2.1]# cd /usr/local/src/[rootnode1 src]# tar xjvf phantomjs-2.1.1-linux-x86_64.tar.bz2[rootnode1 src]# cd phantomjs-2.1.1-linux-x86_64/bin[rootnode1 bin]# cp phantomjs /usr/local/bin6.安装elasticsearch-head数据可视化工具[rootnode1 bin]# cd /usr/local/src/[rootnode1 src]# tar xzvf elasticsearch-head.tar.gz[rootnode1 src]# cd elasticsearch-head/[rootnode1 elasticsearch-head]# npm installnpmWARN deprecated fsevents1.2.13: Upgrade to fsevents v2 to mitigate potential security issuesnpmWARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents^1.0.0(node_modules/karma/node_modules/chokidar/node_modules/fsevents):npmWARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platformforfsevents1.2.13: wanted{os:darwin,arch:any}(current:{os:linux,arch:x64})npmWARN elasticsearch-head0.0.0 license should be a valid SPDX license expression up todatein6.53s修改主配置文件[rootnode1 elasticsearch-head]# cd[rootnode1 ~]# vim /etc/elasticsearch/elasticsearch.yml[rootnode1 ~]# systemctl restart elasticsearch.service#下面配置文件插末尾##http.cors.enabled:true##开启跨域访问支持默认为falsehttp.cors.allow-origin:*## 跨域访问允许的域名地址[rootnode2 elasticsearch-head]# systemctl restart elasticsearch.service启动elasticsearch-head 启动服务器[rootnode1 ~]# cd /usr/local/src/elasticsearch-head/####切换到后台运行[rootnode1 elasticsearch-head]# npm run start [1]98832[rootnode1 elasticsearch-head]#elasticsearch-head0.0.0 start /usr/local/src/elasticsearch-headgrunt server Runningconnect:server(connect)task Waiting forever... Started connect web server on http://localhost:9100[rootnode1 elasticsearch-head]# netstat -lnupt |grep 9100tcp000.0.0.0:91000.0.0.0:* LISTEN98842/grunt[rootnode1 elasticsearch-head]# netstat -lnupt |grep 9200tcp600:::9200 :::* LISTEN98742/java[rootnode1 elasticsearch-head]#登录192.168.108.42 node2主机上传node-v8.2.1.tar.gz到/opt编译安装node组件依赖包耗时比较长20分钟[rootnode2 ~]# yum install -y gcc gcc-c make[rootnode2 ~]# cd /opt[rootnode2 opt]# lselasticsearch-5.5.0.rpm node-v8.2.1.tar.gz rh[rootnode2 opt]# tar xzvf node-v8.2.1.tar.gz[rootnode2 opt]# cd node-v8.2.1/[rootnode2 node-v8.2.1]# ./configure[rootnode2 node-v8.2.1]# make -j4[rootnode2 node-v8.2.1]# make install5.安装phantomjs上传软件包到/usr/local/src/[rootnode2 node-v8.2.1]# cd /usr/local/src/[rootnode2 src]# lsphantomjs-2.1.1-linux-x86_64.tar.bz2[rootnode2 src]# tar xjvf phantomjs-2.1.1-linux-x86_64.tar.bz2[rootnode2 src]# cd phantomjs-2.1.1-linux-x86_64/bin[rootnode2 bin]# cp phantomjs /usr/local/bin6.安装elasticsearch-head[rootnode2 bin]# cd /usr/local/src/[rootnode2 src]# tar xzvf elasticsearch-head.tar.gz[rootnode2 src]# cd elasticsearch-head/[rootnode2 elasticsearch-head]# npm installnpmWARN deprecated fsevents1.2.13: Upgrade to fsevents v2 to mitigate potential security issuesnpmWARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents^1.0.0(node_modules/karma/node_modules/chokidar/node_modules/fsevents):npmWARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platformforfsevents1.2.13: wanted{os:darwin,arch:any}(current:{os:linux,arch:x64})npmWARN elasticsearch-head0.0.0 license should be a valid SPDX license expression up todatein5.132s修改主配置文件[rootnode2 elasticsearch-head]# cd[rootnode2 ~]# vim /etc/elasticsearch/elasticsearch.yml####下面配置文件插末尾##http.cors.enabled:true##开启跨域访问支持默认为falsehttp.cors.allow-origin:*## 跨域访问允许的域名地址[rootnode2 ~]# systemctl restart elasticsearch.service启动elasticsearch-head 启动服务器[rootnode2 ~]# cd /usr/local/src/elasticsearch-head/[rootnode2 elasticsearch-head]# npm run start [1]98766[rootnode2 elasticsearch-head]#elasticsearch-head0.0.0 start /usr/local/src/elasticsearch-headgrunt server Runningconnect:server(connect)task Waiting forever... Started connect web server on http://localhost:9100[rootnode2 elasticsearch-head]# netstat -lnupt |grep 9100tcp000.0.0.0:91000.0.0.0:* LISTEN98776/grunt[rootnode2 elasticsearch-head]# netstat -lnupt |grep 9200tcp600:::9200 :::* LISTEN98805/java7.验证结果node1现象真机上打开浏览器输入http://192.168.108.41:9100/ 可以看见群集很健康是绿色在Elasticsearch 后面的栏目中输入http://192.168.108.41:9200然后点连接 会发现集群健康值: green (0 of 0)●node1信息动作★node2信息动作node2现象笔记本上打开浏览器输入http://192.168.108.42:9100/ 可以看见群集很健康是绿色在Elasticsearch 后面的栏目中输入http://192.168.108.42:9200然后点连接 会发现集群健康值: green (0 of 0)●node1信息动作★node2信息动作登录192.168.108.41 node1主机 索引为index-demo,类型为test,可以看到成功创建[rootnode1 elasticsearch-head]# curl -XPUT localhost:9200/index-demo/test/1?prettypretty -H content-Type: application/json -d {user:zhangsan,mesg:hello world}{_index:index-demo,_type:test,_id:1,_version:1,result:created,_shards:{total:2,successful:2,failed:0},created:true}在192.168.108.1 打开浏览器输入http://192.168100.41:9100/ 查看索引信息上面图可以看见索引默认被分片5个并且有一个副本node1信息动作 01234node2信息动作 01234node1和node2是平等关系不是主从关系点击数据浏览可以看到存储的数据点击数据浏览–会发现在node1上创建的索引为index-demo,类型为test, 相关的信息4.安装logstash用作日志搜集输出到elasticsearch中登录主机192.168.108.43关闭防火墙关闭核心防护安装Apahce服务httpd[rootapache ~]# systemctl stop firewalld.service[rootapache ~]# vim /etc/hosts[rootapache ~]# yum install -y httpd[rootapache ~]# systemctl start httpd#访问下日志[rootapache ~]# cd /var/log/httpd/[rootapache httpd]# lsaccess_log error_log# access_log 访问日志# error_log 错误日志安装Java环境[rootapache ~]# java -versionopenjdk version1.8.0_262OpenJDK Runtime Environment(build1.8.0_262-b10)OpenJDK64-Bit Server VM(build25.262-b10, mixed mode)安装logstash上传logstash-5.5.1.rpm到/opt目录下[rootapache ~]# cd /opt#安装logstash[rootapache opt]# rpm -ivh logstash-5.5.1.rpmwarning: logstash-5.5.1.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY Preparing...################################# [100%]Updating / installing...1:logstash-1:5.5.1-1################################# [100%]Using provided startup.options file: /etc/logstash/startup.options Successfully created system startup scriptforLogstash#启动logstash[rootapache opt]# systemctl enable logstash.service --nowCreated symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.#建立logstash软连接[rootapache opt]# ln -s /usr/share/logstash/bin/logstash /usr/local/bin/logstashapache节点与elasticsearchnode节点做对接测试Logstash这个命令测试字段描述解释● -f 通过这个选项可以指定logstash的配置文件根据配置文件配置logstash● -e 后面跟着字符串 该字符串可以被当做logstash的配置如果是” ”,则默认使用stdin做为输入、stdout作为输出● -t 测试配置文件是否正确然后退出登录192.168.108.43 在Apache服务器上输入采用标准输入 输出采用标准输出[rootapache opt]# logstash -e input { stdin{} } output { stdout{} }#正常 JAVA的东西ERROR StatusLogger No log4j2 configurationfilefound. Using default configuration: logging only errors to the console. WARNING: Could notfindlogstash.ymlwhichis typically locatedin$LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could notfindlog4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default configwhichlogs to console15:25:55.980[main]INFO logstash.setting.writabledirectory - Creating directory{:settingpath.queue, :path/usr/share/logstash/data/queue}15:25:55.987[main]INFO logstash.setting.writabledirectory - Creating directory{:settingpath.dead_letter_queue, :path/usr/share/logstash/data/dead_letter_queue}15:25:56.033[LogStash::Runner]INFO logstash.agent - No persistent UUIDfilefound. Generating new UUID{:uuid5b578ab0-5aea-4e59-bd6a-65687f1d7ab4, :path/usr/share/logstash/data/uuid}15:25:56.239[[main]-pipeline-manager]INFO logstash.pipeline - Starting pipeline{idmain,pipeline.workers4,pipeline.batch.size125,pipeline.batch.delay5,pipeline.max_inflight500}15:25:56.303[[main]-pipeline-manager]INFO logstash.pipeline - Pipeline main started The stdin plugin is now waitingforinput:15:25:56.399[Api Webserver]INFO logstash.agent - Successfully started Logstash API endpoint{:port9600}#需要输入www.baidu.comwww.baidu.com2025-12-12T07:26:07.015Z apache www.baidu.com#需要输入www.sina.comwww.sina.com.cn2025-12-12T07:26:16.322Z apache www.sina.com.cn ^C#ctrl_c退出15:26:18.710[SIGINT handler]WARN logstash.runner - SIGINT received. Shutting down the agent.15:26:18.723[LogStash::Runner]WARN logstash.agent - stopping pipeline{:idmain}使用rubydebug显示详细输出codec为一种编解码器[rootapache opt]# logstash -e input { stdin{} } output { stdout{ codecrubydebug} }ERROR StatusLogger No log4j2 configurationfilefound. Using default configuration: logging only errors to the console. WARNING: Could notfindlogstash.ymlwhichis typically locatedin$LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could notfindlog4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default configwhichlogs to console15:27:41.867[[main]-pipeline-manager]INFO logstash.pipeline - Starting pipeline{idmain,pipeline.workers4,pipeline.batch.size125,pipeline.batch.delay5,pipeline.max_inflight500}15:27:41.971[[main]-pipeline-manager]INFO logstash.pipeline - Pipeline main started The stdin plugin is now waitingforinput:15:27:42.066[Api Webserver]INFO logstash.agent - Successfully started Logstash API endpoint{:port9600}#需要输入www.baidu.comwww.baidu.com{timestamp2025-12-12T07:27:55.422Z,version1,hostapache,messagewww.baidu.com}^C#ctrl_c退出15:27:59.220[SIGINT handler]WARN logstash.runner - SIGINT received. Shutting down the agent.15:27:59.269[LogStash::Runner]WARN logstash.agent - stopping pipeline{:idmain}使用logstash将信息写入elasticsearch中 输入 输出 对接[rootapache opt]# logstash -e input { stdin{} } output { elasticsearch { hosts[192.168.108.41:9200] } }ERROR StatusLogger No log4j2 configurationfilefound. Using default configuration: logging only errors to the console. WARNING: Could notfindlogstash.ymlwhichis typically locatedin$LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could notfindlog4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default configwhichlogs to console15:28:40.014[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated{:changes{:removed[], :added[http://192.168.108.41:9200/]}}15:28:40.018[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Running health check to seeifan Elasticsearch connection is working{:healthcheck_urlhttp://192.168.108.41:9200/, :path/}15:28:40.148[[main]-pipeline-manager]WARN logstash.outputs.elasticsearch - Restored connection to ES instance{:url#Java::JavaNet::URI:0x15b92e86}15:28:40.152[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Using mapping template from{:pathnil}15:28:40.384[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Attempting toinstalltemplate{:manage_template{templatelogstash-*,version50001,settings{index.refresh_interval5s},mappings{_default_{_all{enabledtrue,normsfalse},dynamic_templates[{message_field{path_matchmessage,match_mapping_typestring,mapping{typetext,normsfalse}}},{string_fields{match*,match_mapping_typestring,mapping{typetext,normsfalse,fields{keyword{typekeyword,ignore_above256}}}}}],properties{timestamp{typedate,include_in_allfalse},version{typekeyword,include_in_allfalse},geoip{dynamictrue,properties{ip{typeip},location{typegeo_point},latitude{typehalf_float},longitude{typehalf_float}}}}}}}}15:28:40.432[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Installing elasticsearch template to _template/logstash15:28:40.700[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - New Elasticsearch output{:classLogStash::Outputs::ElasticSearch, :hosts[#Java::JavaNet::URI:0x6d280c11]}15:28:40.711[[main]-pipeline-manager]INFO logstash.pipeline - Starting pipeline{idmain,pipeline.workers4,pipeline.batch.size125,pipeline.batch.delay5,pipeline.max_inflight500}15:28:40.865[[main]-pipeline-manager]INFO logstash.pipeline - Pipeline main started The stdin plugin is now waitingforinput:15:28:40.971[Api Webserver]INFO logstash.agent - Successfully started Logstash API endpoint{:port9600}#输入内容www.baidu.com www.360.cn www.qq.com ^C15:29:03.094[SIGINT handler]WARN logstash.runner - SIGINT received. Shutting down the agent.15:29:03.113[LogStash::Runner]WARN logstash.agent - stopping pipeline{:idmain}笔记本192.168.108.1 登陆打开浏览器 输入http://192.168.108.41:9100/多出 logstash-2025.12.12(没命名自动用时间后缀查看索引信息点击数浏览查看响应的内容登录192.168.108.43 (Apache主机) 做对接配置Logstash配置文件Logstash配置文件主要由三部分组成input、output、filter根据需要[rootapache ~]# chmod or /var/log/messages[rootapache ~]# ll /var/log/messages-rw----r--.1root root856951Dec1215:51 /var/log/messages[rootapache ~]# vim /etc/logstash/conf.d/system.conf[rootapache ~]# cat /etc/logstash/conf.d/system.confinput{file{path/var/log/messagestypesystemstart_positionbeginning}}output{elasticsearch{hosts[192.168.108.41:9200]indexsystem-%{YYYY.MM.dd}}}[rootapache ~]# systemctl restart logstash.service笔记本打开浏览器 输入http://192.168.108.41:9100/ 查看索引信息多出 system-2025.12.12点击数据浏览看内容内容不直观装个kibana数据可视化5.安装Kibana登录192.168.108.41 node1主机在node1主机安装kibana上传kibana-5.5.1-x86_64.rpm 到/usr/local/src目录[rootnode1 ~]# cd /usr/local/src/[rootnode1 src]# rpm -ivh kibana-5.5.1-x86_64.rpm[rootnode1 src]# cd /etc/kibana/[rootnode1 kibana]# cp kibana.yml kibana.yml.bak[rootnode1 kibana]# vim kibana.yml2server.port:5601# kibana打开的端口7server.host:0.0.0.0#kibana侦听的地址21elasticsearch.url:http://192.168.108.41:9200#和elasticsearch建立联系30kibana.index:.kibana#在elasticsearch中添加.kibana索引[rootnode1 kibana]# systemctl start kibana.service #启动kibana服务[rootnode1 kibana]# systemctl enable kibana.service #开机启动kibana服务笔记本浏览器输入192.168.108.41:5601对接Apache主机的Apache 日志文件访问的、错误的apache主机操作[rootapache ~]# cd /etc/logstash/conf.d/[rootapache conf.d]# touch apache_log.conf[rootapache conf.d]# cat apache_log.confinput{file{path/etc/httpd/logs/access_logtypeaccessstart_positionbeginning}file{path/etc/httpd/logs/error_logtypeerrorstart_positionbeginning}}output{if[type]access{elasticsearch{hosts[192.168.108.41:9200]indexapache_access-%{YYYY.MM.dd}}}if[type]error{elasticsearch{hosts[192.168.108.41:9200]indexapache_error-%{YYYY.MM.dd}}}}[rootapache conf.d]# /usr/share/logstash/bin/logstash -f apache_log.confERROR StatusLogger No log4j2 configurationfilefound. Using default configuration: logging only errors to the console. WARNING: Could notfindlogstash.ymlwhichis typically locatedin$LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could notfindlog4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default configwhichlogs to console16:58:23.975[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated{:changes{:removed[], :added[http://192.168.108.41:9200/]}}16:58:23.979[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Running health check to seeifan Elasticsearch connection is working{:healthcheck_urlhttp://192.168.108.41:9200/, :path/}16:58:24.072[[main]-pipeline-manager]WARN logstash.outputs.elasticsearch - Restored connection to ES instance{:url#Java::JavaNet::URI:0x7ff54888}16:58:24.075[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Using mapping template from{:pathnil}16:58:24.266[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Attempting toinstalltemplate{:manage_template{templatelogstash-*,version50001,settings{index.refresh_interval5s},mappings{_default_{_all{enabledtrue,normsfalse},dynamic_templates[{message_field{path_matchmessage,match_mapping_typestring,mapping{typetext,normsfalse}}},{string_fields{match*,match_mapping_typestring,mapping{typetext,normsfalse,fields{keyword{typekeyword,ignore_above256}}}}}],properties{timestamp{typedate,include_in_allfalse},version{typekeyword,include_in_allfalse},geoip{dynamictrue,properties{ip{typeip},location{typegeo_point},latitude{typehalf_float},longitude{typehalf_float}}}}}}}}16:58:24.347[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - New Elasticsearch output{:classLogStash::Outputs::ElasticSearch, :hosts[#Java::JavaNet::URI:0x1dbed19a]}16:58:24.352[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Elasticsearch pool URLs updated{:changes{:removed[], :added[http://192.168.108.41:9200/]}}16:58:24.353[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Running health check to seeifan Elasticsearch connection is working{:healthcheck_urlhttp://192.168.108.41:9200/, :path/}16:58:24.357[[main]-pipeline-manager]WARN logstash.outputs.elasticsearch - Restored connection to ES instance{:url#Java::JavaNet::URI:0xc32ec29}16:58:24.359[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Using mapping template from{:pathnil}16:58:24.367[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - Attempting toinstalltemplate{:manage_template{templatelogstash-*,version50001,settings{index.refresh_interval5s},mappings{_default_{_all{enabledtrue,normsfalse},dynamic_templates[{message_field{path_matchmessage,match_mapping_typestring,mapping{typetext,normsfalse}}},{string_fields{match*,match_mapping_typestring,mapping{typetext,normsfalse,fields{keyword{typekeyword,ignore_above256}}}}}],properties{timestamp{typedate,include_in_allfalse},version{typekeyword,include_in_allfalse},geoip{dynamictrue,properties{ip{typeip},location{typegeo_point},latitude{typehalf_float},longitude{typehalf_float}}}}}}}}16:58:24.373[[main]-pipeline-manager]INFO logstash.outputs.elasticsearch - New Elasticsearch output{:classLogStash::Outputs::ElasticSearch, :hosts[#Java::JavaNet::URI:0x1b5f73cc]}16:58:24.375[[main]-pipeline-manager]INFO logstash.pipeline - Starting pipeline{idmain,pipeline.workers4,pipeline.batch.size125,pipeline.batch.delay5,pipeline.max_inflight500}16:58:24.553[[main]-pipeline-manager]INFO logstash.pipeline - Pipeline main started16:58:24.611[Api Webserver]INFO logstash.agent - Successfully started Logstash API endpoint{:port9601}^C16:59:03.516[SIGINT handler]WARN logstash.runner - SIGINT received. Shutting down the agent.16:59:03.521[LogStash::Runner]WARN logstash.agent - stopping pipeline{:idmain}登录192.168.108.1 node1主机打开输入http://192.168.108.43打开浏览器 输入http://192.168.108.41:9100/ 查看索引信息能发现apache_error-2025.12.12 apache_access-2025.12.12打开浏览器 输入http://192.168.108.41:5601。可以多刷新几次观察kibana,日志很直观显示