Splunk（2）安装与配置

Posted on Edited on In Word count in article: 3.3k Reading time ≈ 12 mins.

安装

系统使用的Centos7.9。

Splunk Server

先去官网下载安装压缩包文件splunk-9.0.1-82c987350fde-Linux-x86_64.tgz。

1
2
$ ls
splunk-9.0.1-82c987350fde-Linux-x86_64.tgz

解压缩得到splunk运行文件夹。

1
2
3
$ tar xvzf splunk-9.0.1-82c987350fde-Linux-x86_64.tgz
$ ls
splunk  splunk-9.0.1-82c987350fde-Linux-x86_64.tgz

启动splunk，第一次启动需要同意服务条款，并且设置admin的用户名和密码，这里随便设置了一个admin/adminadmin。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
$ cd splunk
$ bin/splunk start
...用户条款此处省略...
"Statement of Work" means the statements of work and/or any and all applicable
Orders, that describe the specific services to be performed by Splunk,
including any materials and deliverables to be delivered by Splunk.
Do you agree with this license? [y/n]: y

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: admin
Password must contain at least:
   * 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Copying '/home/shuyi/apps/splunk/splunk/etc/openldap/ldap.conf.default' to '/home/shuyi/apps/splunk/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
....................+++++
..........+++++
e is 65537 (0x10001)
writing RSA key

Generating RSA private key, 2048 bit long modulus
.........................................+++++
......................................................................................................+++++
e is 65537 (0x10001)
writing RSA key

Moving '/home/shuyi/apps/splunk/splunk/share/splunk/search_mrsparkle/modules.new' to '/home/shuyi/apps/splunk/splunk/share/splunk/search_mrsparkle/modules'.

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration... Done.
                Creating: /home/shuyi/apps/splunk/splunk/var/lib/splunk
                Creating: /home/shuyi/apps/splunk/splunk/var/run/splunk
                Creating: /home/shuyi/apps/splunk/splunk/var/run/splunk/appserver/i18n
                Creating: /home/shuyi/apps/splunk/splunk/var/run/splunk/appserver/modules/static/css
                Creating: /home/shuyi/apps/splunk/splunk/var/run/splunk/upload
                Creating: /home/shuyi/apps/splunk/splunk/var/run/splunk/search_telemetry
                Creating: /home/shuyi/apps/splunk/splunk/var/spool/splunk
                Creating: /home/shuyi/apps/splunk/splunk/var/spool/dirmoncache
                Creating: /home/shuyi/apps/splunk/splunk/var/lib/splunk/authDb
                Creating: /home/shuyi/apps/splunk/splunk/var/lib/splunk/hashDb
New certs have been generated in '/home/shuyi/apps/splunk/splunk/etc/auth'.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _configtracker _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/home/shuyi/apps/splunk/splunk/splunk-9.0.1-82c987350fde-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Generating a RSA private key
.............+++++
..........................................+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=bogon/O=SplunkUser
Getting CA Private Key
writing RSA key
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
                                                           [  OK  ]

Waiting for web server at http://127.0.0.1:8000 to be available........ Done


If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://bogon:8000

然后我们再放开服务的8000端口。

1
2
$ sudo firewall-cmd --permanent --add-port=8000/tcp
$ sudo firewall-cmd --reload

就可以在浏览器里面访问到。

p1

然后为了能让系统自动启动（使用本地用户shuyi启动），还需要配置一下。

1
2
3
4
5
6
7
8
9
10
11
12
$ bin/splunk enable boot-start -user shuyi
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
$ reboot
...rebooting...
$ ps -aux | grep splunk
shuyi     1604 10.3  5.1 608744 201444 ?       Sl   14:30   0:01 splunkd -p 8089 start
shuyi     1609  0.2  0.3 130944 14848 ?        Ss   14:30   0:00 [splunkd pid=1604] splunkd -p 8089 start [process-runner]
shuyi     1730  5.7  2.7 587104 106796 ?       SLl  14:30   0:00 mongod --dbpath=/home/shuyi/apps/splunk/splunk/var/lib/splunk/kvstore/mongo --storageEngine=wiredTiger --wiredTigerCacheSizeGB=0.450000 --port=8191 --timeStampFormat=iso8601-utc --oplogSize=200 --keyFile=/home/shuyi/apps/splunk/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --setParameter=oplogFetcherSteadyStateMaxFetcherRestarts=0 --replSet=AEF4CC9E-4BB7-4508-B17D-F2CFA4428975 --bind_ip=0.0.0.0 --sslMode=requireSSL --sslAllowInvalidHostnames --sslPEMKeyFile=/home/shuyi/apps/splunk/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --tlsDisabledProtocols=noTLS1_0,noTLS1_1 --sslCipherConfig=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 --nounixsocket --noscripting
shuyi     1872  1.8  2.0 241584 80768 ?        Sl   14:30   0:00 /home/shuyi/apps/splunk/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore
shuyi     1971  5.3  1.6 2645580 64084 ?       Sl   14:30   0:00 /home/shuyi/apps/splunk/splunk/bin/python3.7 -O /home/shuyi/apps/splunk/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8000
shuyi     2172  0.0  0.0 112780   708 pts/0    R+   14:30   0:00 grep --color=auto splunk

如果需要停止splunk服务，可以使用stop命令。

1
2
3
4
5
6
7
$ bin/splunk stop
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
....                                                       [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.

重新启动。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
$ bin/splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
        Checking configuration... Done.
        Checking critical directories...        Done
        Checking indexes...
                Validated: _audit _configtracker _internal _introspection _metrics _metrics_rollup _telemetry _thefishbucket history main summary
        Done
        Checking filesystem compatibility...  Done
        Checking conf files for problems...
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/home/shuyi/apps/splunk/splunk/splunk-9.0.1-82c987350fde-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
PYTHONHTTPSVERIFY is set to 0 in splunk-launch.conf disabling certificate validation for the httplib and urllib libraries shipped with the embedded Python interpreter; must be set to "1" for increased security
Done
                                                           [  OK  ]

Waiting for web server at http://127.0.0.1:8000 to be available...... Done


If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://bogon:8000

Universal Forwarder

从官网下载forwarder的安装文件splunkforwarder-9.0.5-e9494146ae5c.x86_64.rpm。

1
2
$ ls
splunkforwarder-9.0.5-e9494146ae5c.x86_64.rpm

为了安全方面的考虑，最好不要以root用户运行universal forwarder，这里创建一个新用户splunk。

1
2
$ sudo useradd -m splunk
$ sudo groupadd splunk

创建安装目录。

1
2
$ export SPLUNK_HOME="/opt/splunkforwarder"
$ sudo mkdir $SPLUNK_HOME

安装forwarder安装包。

1
2
3
$ sudo rpm -i splunkforwarder-9.0.5-e9494146ae5c.x86_64.rpm
warning: splunkforwarder-9.0.5-e9494146ae5c.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
complete

修改文件夹splunk的文件夹权限。

1
2
3
4
5
6
$ sudo chown -R splunk:splunk $SPLUNK_HOME
$ ls -la /opt/
total 0
drwxr-xr-x.  3 root   root    29 Jul 15 02:55 .
dr-xr-xr-x. 17 root   root   224 Jul  5 07:47 ..
drwxr-xr-x.  9 splunk splunk 256 Jul 15 02:56 splunkforwarder

最后启动forwarder（第一次启动需要同意条款）。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
$ sudo $SPLUNK_HOME/bin/splunk start
...用户条款此处省略...
"Statement of Work" means the statements of work and/or any and all applicable
Orders, that describe the specific services to be performed by Splunk,
including any materials and deliverables to be delivered by Splunk.
Do you agree with this license? [y/n]: y

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: admin
Password must contain at least:
   * 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Creating unit file...
Important: splunk will start under systemd as user: splunk
The unit file has been created.


Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking mgmt port [8089]: open
                Creating: /opt/splunkforwarder/var/lib/splunk
                Creating: /opt/splunkforwarder/var/run/splunk
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
                Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
                Creating: /opt/splunkforwarder/var/run/splunk/upload
                Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
                Creating: /opt/splunkforwarder/var/spool/splunk
                Creating: /opt/splunkforwarder/var/spool/dirmoncache
                Creating: /opt/splunkforwarder/var/lib/splunk/authDb
                Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.5-e9494146ae5c-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
                                                           [  OK  ]

可以看到forwarder的进程， 比server会少很多。

1
2
3
4
$ ps -aux | grep splunk
splunk   11553  0.6  4.2 378816 164528 ?       Ssl  02:59   0:00 splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd
splunk   11588  0.0  0.3 131280 14668 ?        Ss   02:59   0:00 [splunkd pid=11553] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
shuyi    11660  0.0  0.0 112780   708 pts/0    S+   03:01   0:00 grep --color=auto splunk

停止forwarder服务可以使用stop命令。

1
2
3
4
5
6
7
8
9
$ sudo $SPLUNK_HOME/bin/splunk stop
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
                                                           [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.

安装过程中，splunk forwarder已经自动设置成开机启动了，不需要像splunk server一样需要额外配置。

配置

Splunk Server添加本地数据源。

在右上角的选项里面添加数据源.

Settings-> Data Inputs -> Files & Directories -> New Local File & Directory（右上角）

p2

这里我们以nginx默认日志为例。

Browse -> 输入 /var/log/nginx/access.log -> 点击上方的Next

使用默认的source type的选项，然后输入source type的名称。

p3

点击next之后， Index选择main，然后点击review，最后submit。

然后就可以开始愉快的搜索了。

p4

Splunk Universal Forwarder添加数据源

因为数据源可能在另一个服务器上面，server需要收集远程服务器的日志信息。

Universal Forwarder服务会把日志推送到server服务器上面。

首先需要设置允许server接收远程数据，添加监听端口9997。

在server的服务器上面进入splunk目录，使用命令行启动监听9997端口。

1
2
3
4
5
$ bin/splunk enable listen 9997 -auth admin:adminadmin
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Listening for Splunk data on TCP port 9997.
$ sudo netstat -nlpt | grep "9997"
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      4302/splunkd

需要在这个服务器上开通9997端口。

1
2
3
4
$ sudo firewall-cmd --permanent --add-port=9997/tcp
success
$ sudo firewall-cmd --reload
success

然后再配置forwarder推送内容到server。

1
2
3
4
5
$ sudo /opt/splunkforwarder/bin/splunk add forward-server 192.168.0.112:9997
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Added forwarding to: 192.168.0.112:9997.

然后可以查看所有的forward server。

1
2
3
4
5
6
7
8
$ sudo /opt/splunkforwarder/bin/splunk list forward-server
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Active forwards:
        192.168.0.112:9997
Configured but inactive forwards:
        None

然后让forwarder去监听nginx的日志文件。

1
2
3
4
5
$ sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/nginx/access.log
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Added monitor of '/var/log/nginx/access.log'.

然后就可以在搜索页面看到这个113服务器的日志已经过来了。

p5

但是发现虽然112和113两个服务器的nginx日志完全一模一样，但是sourceType却不一样，从113的forwarder过来的sourceType是access-too_small，不是我们先前定义的nginx:log。

这里我们最好为我们的监听创造一个input.conf文件来方便我们管理。

先删除这个监听配置。

1
2
3
4
5
$ sudo /opt/splunkforwarder/bin/splunk remove monitor /var/log/nginx/access.log
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Removed monitor of '/var/log/nginx/access.log'

然后添加配置文件。

1
2
3
4
5
6
$ sudo touch /opt/splunkforwarder/etc/system/local/inputs.conf
$ sudo cat /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/nginx/access.log]
disable = 0
index = main
sourcetype = nginx:log

修改配置文件的权限（其实这一步没有必要，因为在重启forwarder的时候，会自动把/opt/splunkforwarder文件夹下面的所有文件的权限都修改成splunk的）。

1
$ sudo chown splunk:splunk /opt/splunkforwarder/etc/system/local/inputs.conf

此时我们可以先检查一下splunk的配置文件是否正确。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
$ sudo /opt/splunkforwarder/bin/splunk btool check --debug
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Checking: /opt/splunkforwarder/etc/apps/learned/local/props.conf
Checking: /opt/splunkforwarder/etc/apps/search/local/inputs.conf
No spec file for: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/default-mode.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/health.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/server.conf
Checking: /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/web.conf
No spec file for: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/inputs.conf
Checking: /opt/splunkforwarder/etc/apps/introspection_generator_addon/default/server.conf
Checking: /opt/splunkforwarder/etc/apps/journald_input/default/authorize.conf
Checking: /opt/splunkforwarder/etc/apps/journald_input/default/inputs.conf
No spec file for: /opt/splunkforwarder/etc/apps/search/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/restmap.conf
Checking: /opt/splunkforwarder/etc/apps/search/default/transforms.conf
Checking: /opt/splunkforwarder/etc/apps/splunk_httpinput/default/inputs.conf
No spec file for: /opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/app.conf
Checking: /opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/props.conf
Checking: /opt/splunkforwarder/etc/apps/splunk_internal_metrics/default/transforms.conf
No spec file for: /opt/splunkforwarder/etc/manager-apps/_cluster/default/indexes.conf
Checking: /opt/splunkforwarder/etc/system/default/alert_actions.conf
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
No spec file for: /opt/splunkforwarder/etc/system/default/app.conf
Checking: /opt/splunkforwarder/etc/system/default/audit.conf
Checking: /opt/splunkforwarder/etc/system/default/authentication.conf
Checking: /opt/splunkforwarder/etc/system/default/authorize.conf
No spec file for: /opt/splunkforwarder/etc/system/default/conf.conf
Checking: /opt/splunkforwarder/etc/system/default/default-mode.conf
Checking: /opt/splunkforwarder/etc/system/default/federated.conf
Checking: /opt/splunkforwarder/etc/system/default/global-banner.conf
Checking: /opt/splunkforwarder/etc/system/default/health.conf
Checking: /opt/splunkforwarder/etc/system/default/inputs.conf
Checking: /opt/splunkforwarder/etc/system/default/limits.conf
Checking: /opt/splunkforwarder/etc/system/default/livetail.conf
Checking: /opt/splunkforwarder/etc/system/default/messages.conf
Checking: /opt/splunkforwarder/etc/system/default/metric_alerts.conf
Checking: /opt/splunkforwarder/etc/system/default/metric_rollups.conf
Checking: /opt/splunkforwarder/etc/system/default/outputs.conf
Checking: /opt/splunkforwarder/etc/system/default/procmon-filters.conf
Checking: /opt/splunkforwarder/etc/system/default/props.conf
Checking: /opt/splunkforwarder/etc/system/default/restmap.conf
Checking: /opt/splunkforwarder/etc/system/default/server.conf
Checking: /opt/splunkforwarder/etc/system/default/source-classifier.conf
No spec file for: /opt/splunkforwarder/etc/system/default/telemetry.conf
Checking: /opt/splunkforwarder/etc/system/default/transforms.conf
Checking: /opt/splunkforwarder/etc/system/default/visualizations.conf
Checking: /opt/splunkforwarder/etc/system/default/web-features.conf
Checking: /opt/splunkforwarder/etc/system/default/web.conf
Checking: /opt/splunkforwarder/etc/system/default/workload_policy.conf
Checking: /opt/splunkforwarder/etc/system/default/workload_pools.conf
Checking: /opt/splunkforwarder/etc/system/default/workload_rules.conf
Checking: /opt/splunkforwarder/etc/system/local/inputs.conf
                Invalid key in stanza [monitor:///var/log/nginx/access.log] in /opt/splunkforwarder/etc/system/local/inputs.conf, line 2: disable (value: 0).
Checking: /opt/splunkforwarder/etc/system/local/outputs.conf
Checking: /opt/splunkforwarder/etc/system/local/server.conf

可以看到我们刚才修改的inputs.conf文件有问题，需要把disable改成disabled，并且value要改成false。

再次检查就通过了，然后重启forwarder。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ sudo /opt/splunkforwarder/bin/splunk restart
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk /opt/splunkforwarder"
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
                                                           [  OK  ]
Stopping splunk helpers...
                                                           [  OK  ]
Done.
splunkd.pid doesn't exist...

Splunk> CSI: Logfiles.

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
                Invalid key in stanza [webhook] in /opt/splunkforwarder/etc/system/default/alert_actions.conf, line 229: enable_allowlist (value: false).
                Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
        Done
        Checking default conf files for edits...
        Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.0.5-e9494146ae5c-linux-2.6-x86_64-manifest'
        All installed files intact.
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done
 [  OK  ]

其实我们最后可以去掉这个disabled的配置，最后日志也能成功导入进来。

参考

Install on Linux
Enable a receiver for Splunk Enterprise
I can’t find my data!