Nginx（3）限流

Posted on Edited on In Word count in article: 4.5k Reading time ≈ 16 mins.

配置文件默认的文件夹位置是/etc/nginx。

默认的日志文件位置是/var/log/nginx/access.log。

nginx频控是根据漏桶原理来控制请求速率的，也就是说不管请求并发量多大，允许的并发数的最大值是一定的。

请求排队原则根据先入先出（FIFO）原则，先过来的请求会先得到相应。

正常限流

nginx使用limit_req相关的指令来配置，参考配置如下。

1
2
3
4
5
6
7
8
9
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s;
 
server {
    location /login/ {
        limit_req zone=mylimit;
        
        proxy_pass http://my_upstream;
    }
}

limit_req_zone命令用来定义限流命令，其中的配置解释如下：

  • $binary_remote_addr请求发起方的ip地址，nginx根据这个值类进行限流
    • 这里没有使用$remote_addr，相比之下$binary_remote_addr占用空间更小，IPv4只占4个字节，IPv6占16个字节
    • 每一个保存的状态在64位的操作系统里面都占用128字节，所以10M的空间大概能保存80000个请求状态
  • zone=mylimit定义这条规则的名字叫mylimit
  • :10m使用10M的内存空间，如果超出的这个空间，最早的没有被修改过的记录会被删除掉，如果删除掉了空间还不够，就会返回异常
  • rate=10r/s平均请求速率不能超过10个request/秒

下面就做一个简单的测试。

首先使用python3运行一个简单的服务器。

1
2
$ python3 -m http.server 8080
Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) ...

开放8080端口。

1
2
$ sudo firewall-cmd  --add-port=8080/tcp
success

有的机器在nginx转发之后可能出现502的情况，查看日志出现了这个错误：(13: Permission denied) while connecting to upstream。

这是因为SeLinux的安全限制，需要我们修改一下配置。

可以先修改临时配置，但是重启之后会失效。

1
$ sudo setenforce 0                  ##设置SELinux 成为permissive模式

然后修改配置文件/etc/selinux/config让重启之后也生效。

将SELINUX=enforcing改为SELINUX=disabled。

然后配置nginx，在/etc/nginx/conf.d文件夹内添加新的配置文件file-server.conf内容如下。

1
2
3
4
5
6
7
8
9
10
11
$ cat file-server.conf
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
server {
    listen 80;
    server_name centos1;
    location /file {
        limit_req zone=mylimit;

        proxy_pass http://127.0.0.1:8080/;
    }
}

设置的频率控制是1个请求/秒。

然后再在另外一台机器centos2上面测试请求这个接口。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ curl centos1/file
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ascii">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href="file">file</a></li>
</ul>
<hr>
</body>
</html>

接下来再写jmeter的脚本nginx.jmx。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.6.2">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <stringProp name="LoopController.loops">20</stringProp>
          <boolProp name="LoopController.continue_forever">false</boolProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">2</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <boolProp name="ThreadGroup.delayedStart">false</boolProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <ConstantTimer guiclass="ConstantTimerGui" testclass="ConstantTimer" testname="Constant Timer" enabled="true">
          <stringProp name="ConstantTimer.delay">300</stringProp>
        </ConstantTimer>
        <hashTree/>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP Request" enabled="true">
          <boolProp name="HTTPSampler.postBodyRaw">false</boolProp>
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain">centos1</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.path">/file</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.BROWSER_COMPATIBLE_MULTIPART">false</boolProp>
          <boolProp name="HTTPSampler.image_parser">false</boolProp>
          <boolProp name="HTTPSampler.concurrentDwn">false</boolProp>
          <stringProp name="HTTPSampler.concurrentPool">6</stringProp>
          <boolProp name="HTTPSampler.md5">false</boolProp>
          <intProp name="HTTPSampler.ipSourceType">0</intProp>
        </HTTPSamplerProxy>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

这里JMeter的配置是使用2个线程，每个线程每隔300ms发送一次请求，每个线程一共发送20轮，也就是一共40个请求。

现在执行这个脚本。

1
2
3
4
5
6
7
8
$ jmeter -n -t nginx.jmx -l log.jtl
Creating summariser <summary>
Created the tree successfully using nginx.jmx
Starting standalone test @ July 18, 2023 6:10:45 AM EDT (1689675045726)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary =     40 in 00:00:07 =    6.1/s Avg:     2 Min:     1 Max:    18 Err:    34 (85.00%)
Tidying up ...    @ July 18, 2023 6:10:52 AM EDT (1689675052460)
... end of run

可以看到JMeter的执行结果，40个请求一共耗时6秒，成功了6个，失败了34个。

再看nginx日志可以发现请求被准确控制在了1秒一个，其他的请求全部返回了503。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
$ sudo tail -200f /var/log/nginx/access.log   # 全部请求
192.168.0.114 - - [18/Jul/2023:15:41:18 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:18 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:18 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:24 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:24 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:24 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:24 +0800] "GET /file HTTP/1.1" 503 197 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"

$ sudo tail -200f /var/log/nginx/access.log | grep 200         # 只看200的请求
192.168.0.114 - - [18/Jul/2023:15:41:18 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:19 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:20 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:21 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:22 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:41:23 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"

修改错误码

其中503是默认的请求被拦截的返回错误码，如果我们相使用自定义的值，可以使用limit_req_status指令（1.3.15以后的版本才有）。

修改conf文件如下，这是错误码为429。

1
2
3
4
5
6
7
8
9
10
11
12
$ cat file-server.conf
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
limit_req_status 429;
server {
    listen 80;
    server_name centos1;
    location /file {
        limit_req zone=mylimit;

        proxy_pass http://127.0.0.1:8080/;
    }
}

重新进行压测发现错误码变成了429。

1
2
3
4
5
6
$ sudo tail -200f /var/log/nginx/access.log   # 全部请求
...
192.168.0.114 - - [18/Jul/2023:15:48:52 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:48:52 +0800] "GET /file HTTP/1.1" 429 169 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:15:48:53 +0800] "GET /file HTTP/1.1" 429 169 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
...

突发流量

上面的limit_req指令会严格限制请求速率在1r/s，但是有时流量突然增大，突发的流量无法处理，此时可以使用burst指令。

比如我们修改我们的nginx配置，设置burst=2，代表如果请求频率超过了1r/s，就会有2个请求会被放入到队列里面，下次请求的时候会优先从队列里面取。

如果队列数量超过了这个burst的值，则会直接返回429的错误。

1
2
3
4
5
6
7
8
9
10
11
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
limit_req_status 429;
server {
    listen 80;
    server_name centos1;
    location /file {
        limit_req zone=mylimit burst=2;

        proxy_pass http://127.0.0.1:8080/;
    }
}

我们再次执行测试可以发现所有的请求全部成功了，并且成功保持了1r/s的频率，但是请求的耗时基本都上去了，平均都得等1.6秒左右了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
$  sudo tail -200f /var/log/nginx/access.log
192.168.0.113 - - [18/Jul/2023:18:54:07 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:08 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:09 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:10 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:11 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:12 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:13 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:14 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:15 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:16 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:17 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:18 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:19 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:20 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:21 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:22 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:23 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:24 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:25 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:26 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:27 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:28 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:29 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:30 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:31 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:32 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:33 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:34 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:35 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:36 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:37 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:38 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:39 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:40 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:41 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:42 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:43 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:44 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:45 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"
192.168.0.113 - - [18/Jul/2023:18:54:46 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/1.8.0_372)" "-"

JMeter的执行结果。

1
2
3
4
5
6
7
8
9
10
$ jmeter -n -t nginx.jmx -l log.jtl
Creating summariser <summary>
Created the tree successfully using nginx.jmx
Starting standalone test @ July 18, 2023 6:54:08 AM EDT (1689677648007)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary +     23 in 00:00:22 =    1.0/s Avg:  1575 Min:    21 Max:  1708 Err:     0 (0.00%) Active: 2 Started: 2 Finished: 0
summary +     17 in 00:00:17 =    1.0/s Avg:  1639 Min:   684 Max:  1712 Err:     0 (0.00%) Active: 0 Started: 2 Finished: 2
summary =     40 in 00:00:39 =    1.0/s Avg:  1602 Min:    21 Max:  1712 Err:     0 (0.00%)
Tidying up ...    @ July 18, 2023 6:54:47 AM EDT (1689677687545)
... end of run

出现这种情况的原因是如果JMeter只有两个线程，模拟两个用户发送请求，每个用户每隔300ms发送一次请求，前一次请求如果不结束，就不会执行下一个请求。

相当于burst的队列里面最多也只有2个请求在等待。

如果仅仅使用burst参数可以让流量变得平滑，但是增大了相应时间，排列在后面的请求的等待时间会越来越长。

限制延迟

此时需要结合delay参数来用，使用nodelay可以将整体的平均请求频率控制在1r/s，峰值的同时并发数量由burst控制。

比如现在这种情况burst=2,rate=1r/s,nodelay，burst相当于slot空缺位置。

第1秒的时候nginx接受处理一个请求，第一秒过后burst的两个slot都会空出来。

然后接下来的2秒都没有请求过来，然后突然来了3个请求，此时nginx正常转发第一个请求，另外两个请求占用了burst的slot，因为nodelay参数，会立刻转发过去。

相当于四秒的时间内，总体有四个请求过去了，平均值还是1r/s。

现在可以测试一下，limit_req后面增加nodelay参数。

1
2
3
4
5
6
7
8
9
10
11
12
$ cat file-server.conf
limit_req_zone $binary_remote_addr zone=mylimit:10m rate=1r/s;
limit_req_status 429;
server {
    listen 80;
    server_name centos1;
    location /file {
        limit_req zone=mylimit burst=2 nodelay;

        proxy_pass http://127.0.0.1:8080/;
    }
}

运行jmeter查看结果。

1
2
3
4
5
6
7
8
9
10
$ sudo tail -f /var/log/nginx/access.log | grep 200
192.168.0.114 - - [18/Jul/2023:21:01:43 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:43 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:43 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:44 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:45 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:46 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:47 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:48 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"
192.168.0.114 - - [18/Jul/2023:21:01:49 +0800] "GET /file HTTP/1.1" 200 330 "-" "Apache-HttpClient/4.5.14 (Java/11.0.12)" "-"

发现除了最开始的请求在1秒内出现了3次，剩下的全部是严格控制在1r/s。

出现开始的的3r/s的原因就是因为burst=2的前面的两个slot已经是空出来的，此时并发请求进来，nginx会允许这两个burst的请求直接转发，让瞬时的请求速率超过1r/s。

限制连接数

上面使用的limit_req限制nginx转发的请求数量，而limit_conn可以限制连接数量（同一时刻处于连接状态的转发数量）。

连接叫connection，是通常说的tcp连接，是三次握手之后的完整状态，是有状态的连接。

在第一次发起请求之后会建立一个tcp的连接，在这个连接中，可能发生了很多次的http的请求。

现在HTTP/1.1协议都支持这个keepalive的特性，主要是为了加快请求效率，避免三次握手浪费带宽。

nginx提供了一个ngx_http_limit_conn_module模块来控制连接数量。

主要需要使用到limit_conn_status、limit_conn_zone和limit_conn指令。

参考配置如下。

1
2
3
4
5
6
7
8
9
10
11
$ cat file-server.conf
limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m; # 设置10M的共享内存，共享空间名称是conn_limit_per_ip，以远端的二进制IP地址为key
limit_conn_status 430                                           # 连接数如果超过设定值，则返回430错误码
server {
    listen 80;
    server_name centos1;
    location /file {
        limit_conn conn_limit_per_ip 30;                       # 限制连接数为30
        proxy_pass http://127.0.0.1:8080/;
    }
}

白名单

限流一般针对外网，内网相对安全，可以设置白名单，利用nginx的ngx_http_geo_module和ngx_http_map_module两个工具模块。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ cat file-server.conf
geo $limit {
    default 1;
    10.0.0.0/8 0;
    192.168.0.0/24 0;
    172.20.0.35 0;
}
map $limit $limit_key {
    0 "";
    1 $binary_remote_addr;
}
limit_req_zone $limit_key zone=mylimit:10m rate=1r/s;
limit_req_status 429;
server {
    listen 80;
    server_name centos1;
    location /file {
        limit_req zone=mylimit burst=2 nodelay;

        proxy_pass http://127.0.0.1:8080/;
    }
}

设置完成之后重新用JMeter跑一下，限制确实都消失了。

1
2
3
4
5
6
7
8
9
10
$ jmeter -n -t nginx.jmx -l log.jtl
Creating summariser <summary>
Created the tree successfully using nginx.jmx
Starting standalone test @ July 18, 2023 9:58:53 AM EDT (1689688733653)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary +     39 in 00:00:06 =    6.2/s Avg:     2 Min:     1 Max:    19 Err:     0 (0.00%) Active: 1 Started: 2 Finished: 1
summary +      1 in 00:00:00 =    3.3/s Avg:     3 Min:     3 Max:     3 Err:     0 (0.00%) Active: 0 Started: 2 Finished: 2
summary =     40 in 00:00:07 =    6.0/s Avg:     2 Min:     1 Max:    19 Err:     0 (0.00%)
Tidying up ...    @ July 18, 2023 9:59:00 AM EDT (1689688740393)
... end of run

注意：如果设置了geo或者map两个模块之后发现不生效，可以尝试重启nginx。

1
$ sudo systemctl restart nginx

重启之后新的设置才能生效，仅仅sudo nginx -s reload还不够。

参考

Rate Limiting with NGINX and NGINX Plus
Nginx 中的两种限流方式
Module ngx_http_limit_req_module
nginx篇10-限速三剑客之limit_req
Module ngx_http_limit_conn_module