流量复制重放工具goreplay

goreplay简介

https://github.com/buger/goreplay

https://goreplay.org

GoReplay是一个开源工具,用于捕获实时HTTP流量并将其重放到测试环境中,以便使用真实数据持续测试系统。
GoReplay不是代理,而是监听网络接口上的流量,不需要更改生产基础架构,而是在与服务相同的计算机上运行GoReplay守护程序。

goreplay工作原理

goreplay常见用法

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1. 简单的 HTTP 流量复制:
gor –input-raw :80 –output-http “http://staging.com”

2.HTTP 流量复制频率控制:
gor –input-tcp :28020 –output-http “http://staging.com|10″

3.HTTP 流量复制缩小:
gor –input-raw :80 –output-tcp “replay.local:28020|10%”

4.HTTP 流量记录到本地文件:
gor –input-raw :80 –output-file requests.gor

5.HTTP 流量回放和压测:
gor –input-file “requests.gor|200%” –output-http “staging.com”

6.HTTP 流量过滤复制:
gor –input-raw :8080 –output-http staging.com –output-http-url-regexp ^www.

7.HTTP指定接口流量复制:
gor --input-raw :80 --http-allow-url '/api/v1' --output-stdout //--output-stdout表示直接在控制台输出

gor参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
[root@~]# gor --help
Gor is a simple http traffic replication tool written in Go. Its main goal is to replay traffic from production servers to staging and dev environments.
Project page: https://github.com/buger/gor
Author: <Leonid Bugaev> leonsbox@gmail.com
Current Version: 1.0.0

-copy-buffer-size int
Set the buffer size for an individual request (default 5M) (default 5242880)
-cpuprofile string
write cpu profile to file
-debug verbose //打开debug模式,显示所有接口的流量
Turn on debug output, shows all intercepted traffic. Works only when with verbose flag
-exit-after duration
exit after specified duration
-http-allow-header value //用一个正则表达式来匹配http头部,如果请求的头部没有匹配上,则被拒绝
A regexp to match a specific header against. Requests with non-matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^v1
-http-allow-method value // 类似于一个白名单机制来允许通过的http请求方法,除此之外的方法都被拒绝.
Whitelist of HTTP methods to replay. Anything else will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-method GET --http-allow-method OPTIONS
-http-allow-url value //一个正则表达式用来匹配url, 用来过滤完全匹配的的url,在此之外的都被过滤掉
A regexp to match requests against. Filter get matched against full url with domain. Anything else will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-url ^www.
-http-basic-auth-filter value //匹配认证头重放
A regexp to match the decoded basic auth string against. Requests with non-matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-basic-auth-filter "^customer[0-9].*"
-http-disallow-header value //用一个正则表达式来匹配http头部,匹配到的请求会被拒绝掉
A regexp to match a specific header against. Requests with matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor"
-http-disallow-url value //用一个正则表达式来匹配url,如果请求匹配上了,则会被拒绝
A regexp to match requests against. Filter get matched against full url with domain. Anything else will be forwarded:
gor --input-raw :8080 --output-http staging.com --http-disallow-url ^www.
-http-header-limiter value //读取请求,基于FNV32-1A散列来拒绝一定比例的特殊请求
Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific header:
gor --input-raw :8080 --output-http staging.com --http-header-limiter user-id:25%
-http-original-host //在--output-http的输出中,通常gor会使用取代请求的http头,所以应该禁用该选项,保留原始的主机头
Normally gor replaces the Host http header with the host supplied with --output-http. This option disables that behavior, preserving the original Host header.
-http-param-limiter value
Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific GET param:
gor --input-raw :8080 --output-http staging.com --http-param-limiter user_id:25%
-http-pprof :8181
Enable profiling. Starts http server on specified port, exposing special /debug/pprof endpoint. Example: :8181
-http-rewrite-header value
Rewrite the request header based on a mapping:
gor --input-raw :8080 --output-http staging.com --http-rewrite-header Host: (.*).example.com,$1.beta.example.com
-http-rewrite-url value
Rewrite the request url based on a mapping:
gor --input-raw :8080 --output-http staging.com --http-rewrite-url /v1/user/([^\/]+)/ping:/v2/user/$1/ping
-http-set-header value
Inject additional headers to http reqest:
gor --input-raw :8080 --output-http staging.com --http-set-header 'User-Agent: Gor'
-http-set-param value
Set request url param, if param already exists it will be overwritten:
gor --input-raw :8080 --output-http staging.com --http-set-param api_key=1
-input-dummy value
Used for testing outputs. Emits 'Get /' request every 1s
-input-file value //从一个文件中读取请求
Read requests from file:
gor --input-file ./requests.gor --output-http staging.com
-input-file-loop
Loop input files, useful for performance testing.
-input-kafka-host string
Send request and response stats to Kafka:
gor --output-stdout --input-kafka-host '192.168.0.1:9092,192.168.0.2:9092'
-input-kafka-json-format
If turned on, it will assume that messages coming in JSON format rather than GoReplay text format.
-input-kafka-topic string
Send request and response stats to Kafka:
gor --output-stdout --input-kafka-topic 'kafka-log'
-input-raw value
Capture traffic from given port (use RAW sockets and require *sudo* access):
# Capture traffic from 8080 port
gor --input-raw :8080 --output-http staging.com
-input-raw-bpf-filter string
BPF filter to write custom expressions. Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80'
-input-raw-buffer-size int
Controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value.
-input-raw-engine libpcap
Intercept traffic using libpcap (default), and `raw_socket` (default "libpcap")
-input-raw-expire duration
How much it should wait for the last TCP packet, till consider that TCP message complete. (default 2s)
-input-raw-immediate-mode
Set pcap interface to immediate mode.
-input-raw-override-snaplen
Override the capture snaplen to be 64k. Required for some Virtualized environments
-input-raw-realip-header string
If not blank, injects header with given name and real IP value to the request payload. Usually this header should be named: X-Real-IP
-input-raw-timestamp-type string
Possible values: PCAP_TSTAMP_HOST, PCAP_TSTAMP_HOST_LOWPREC, PCAP_TSTAMP_HOST_HIPREC, PCAP_TSTAMP_ADAPTER, PCAP_TSTAMP_ADAPTER_UNSYNCED. This values not supported on all systems, GoReplay will tell you available values of you put wrong one.
-input-raw-track-response
If turned on Gor will track responses in addition to requests, and they will be available to middleware and file output.
-input-tcp value // 用来在多个gor之间流转流量
Used for internal communication between Gor instances. Example:
# Receive requests from other Gor instances on 28020 port, and redirect output to staging
gor --input-tcp :28020 --output-http staging.com
-input-tcp-certificate string
Path to PEM encoded certificate file. Used when TLS turned on.
-input-tcp-certificate-key string
Path to PEM encoded certificate key file. Used when TLS turned on.
-input-tcp-secure
Turn on TLS security. Do not forget to specify certificate and key files.
-memprofile string
write memory profile to this file
-middleware string
Used for modifying traffic using external command
-output-dummy value //用来测试输入,打印出接收的数据.
DEPRECATED: use --output-stdout instead
-output-file value //把进入的请求写入一个文件中
Write incoming requests to file:
gor --input-raw :80 --output-file ./requests.gor
-output-file-append
The flushed chunk is appended to existence file or not.
-output-file-flush-interval duration
Interval for forcing buffer flush to the file, default: 1s. (default 1s)
-output-file-max-size-limit value
Max size of output file, Default: 1TB (default -1)
-output-file-queue-limit int
The length of the chunk queue. Default: 256 (default 256)
-output-file-size-limit value
Size of each chunk. Default: 32mb (default 33554432)
-output-http value //转发进入的请求到一个http地址上
Forwards incoming requests to given http address.
# Redirect all incoming requests to staging.com address
gor --input-raw :80 --output-http http://staging.com
-output-http-compatibility-mode
Use standard Go client, instead of built-in implementation. Can be slower, but more compatible.
-output-http-debug
Enables http debug output.
-output-http-elasticsearch string //把请求和响应状态发送到ElasticSearch
Send request and response stats to ElasticSearch:
gor --input-raw :8080 --output-http staging.com --output-http-elasticsearch 'es_host:api_port/index_name'
-output-http-header --output-http-header
WARNING: --output-http-header DEPRECATED, use `--http-set-header` instead
-output-http-header-filter --output-http-header-filter
WARNING: --output-http-header-filter DEPRECATED, use `--http-allow-header` instead
-output-http-header-hash-filter output-http-header-hash-filter
WARNING: output-http-header-hash-filter DEPRECATED, use `--http-header-hash-limiter` instead
-output-http-method --output-http-method
WARNING: --output-http-method DEPRECATED, use `--http-allow-method` instead
-output-http-queue-len int
Number of requests that can be queued for output, if all workers are busy. default = 1000 (default 1000)
-output-http-redirects int //设置多少次重定向被允许
Enable how often redirects should be followed.
-output-http-response-buffer int
HTTP response buffer size, all data after this size will be discarded.
-output-http-rewrite-url --output-http-rewrite-url
WARNING: --output-http-rewrite-url DEPRECATED, use `--http-rewrite-url` instead
-output-http-stats //每5秒钟输出一次输出队列的状态
Report http output queue stats to console every N milliseconds. See output-http-stats-ms
-output-http-stats-ms int
Report http output queue stats to console every N milliseconds. default: 5000 (default 5000)
-output-http-timeout duration //指定http的request/response超时时间,默认是5秒
Specify HTTP request/response timeout. By default 5s. Example: --output-http-timeout 30s (default 5s)
-output-http-track-response
If turned on, HTTP output responses will be set to all outputs like stdout, file and etc.
-output-http-url-regexp --output-http-url-regexp
WARNING: --output-http-url-regexp DEPRECATED, use `--http-allow-url` instead
-output-http-workers int // gor默认是动态的扩展工作者数量,你也可以指定固定数量的工作者
Gor uses dynamic worker scaling. Enter a number to set a maximum number of workers. default = 0 = unlimited.
-output-http-workers-min int
Gor uses dynamic worker scaling. Enter a number to set a minimum number of workers. default = 1.
-output-kafka-host string
Read request and response stats from Kafka:
gor --input-raw :8080 --output-kafka-host '192.168.0.1:9092,192.168.0.2:9092'
-output-kafka-json-format
If turned on, it will serialize messages from GoReplay text format to JSON.
-output-kafka-topic string
Read request and response stats from Kafka:
gor --input-raw :8080 --output-kafka-topic 'kafka-log'
-output-null
Used for testing inputs. Drops all requests.
-output-stdout
Used for testing inputs. Just prints to console data coming from inputs.
-output-tcp value //用来在多个gor之间流转流量
Used for internal communication between Gor instances. Example:
# Listen for requests on 80 port and forward them to other Gor instance on 28020 port
gor --input-raw :80 --output-tcp replay.local:28020
-output-tcp-secure
Use TLS secure connection. --input-file on another end should have TLS turned on as well.
-output-tcp-stats //每5秒钟报告一次tcp输出队列的状态
Report TCP output queue stats to console every 5 seconds.
-prettify-http
If enabled, will automatically decode requests and responses with: Content-Encodning: gzip and Transfer-Encoding: chunked. Useful for debugging, in conjuction with --output-stdout
-split-output true
By default each output gets same traffic. If set to true it splits traffic equally among all outputs.
-stats //打开输出队列的状态
Turn on queue stats output
-verbose
Turn on more verbose output