流量复制重放工具goreplay

goreplay简介

https://github.com/buger/goreplay

https://goreplay.org

GoReplay是一个开源工具,用于捕获实时HTTP流量并将其重放到测试环境中,以便使用真实数据持续测试系统。 GoReplay不是代理,而是监听网络接口上的流量,不需要更改生产基础架构,而是在与服务相同的计算机上运行GoReplay守护程序。

goreplay工作原理

goreplay常见用法

1. 简单的 HTTP 流量复制:
gor –input-raw :80 –output-http “http://staging.com”

2.HTTP 流量复制频率控制:
gor –input-tcp :28020 –output-http “http://staging.com|10″

3.HTTP 流量复制缩小:
gor –input-raw :80 –output-tcp “replay.local:28020|10%”

4.HTTP 流量记录到本地文件:
gor –input-raw :80 –output-file requests.gor

5.HTTP 流量回放和压测:
gor –input-file “requests.gor|200%” –output-http “staging.com”

6.HTTP 流量过滤复制:
gor –input-raw :8080 –output-http staging.com –output-http-url-regexp ^www.

7.HTTP指定接口流量复制:
gor --input-raw :80 --http-allow-url '/api/v1' --output-stdout      //--output-stdout表示直接在控制台输出

gor参数

[root@~]# gor --help
Gor is a simple http traffic replication tool written in Go. Its main goal is to replay traffic from production servers to staging and dev environments.
Project page: https://github.com/buger/gor
Author: <Leonid Bugaev> leonsbox@gmail.com
Current Version: 1.0.0

  -copy-buffer-size int
    	Set the buffer size for an individual request (default 5M) (default 5242880)
  -cpuprofile string
    	write cpu profile to file
  -debug verbose		//打开debug模式,显示所有接口的流量 
    	Turn on debug output, shows all intercepted traffic. Works only when with verbose flag
  -exit-after duration
    	exit after specified duration
  -http-allow-header value      //用一个正则表达式来匹配http头部,如果请求的头部没有匹配上,则被拒绝
    	A regexp to match a specific header against. Requests with non-matching headers will be dropped:
    		 gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^v1
  -http-allow-method value      // 类似于一个白名单机制来允许通过的http请求方法,除此之外的方法都被拒绝.
    	Whitelist of HTTP methods to replay. Anything else will be dropped:
    		gor --input-raw :8080 --output-http staging.com --http-allow-method GET --http-allow-method OPTIONS
  -http-allow-url value             //一个正则表达式用来匹配url, 用来过滤完全匹配的的url,在此之外的都被过滤掉
    	A regexp to match requests against. Filter get matched against full url with domain. Anything else will be dropped:
    		 gor --input-raw :8080 --output-http staging.com --http-allow-url ^www.
  -http-basic-auth-filter value         //匹配认证头重放
    	A regexp to match the decoded basic auth string against. Requests with non-matching headers will be dropped:
    		 gor --input-raw :8080 --output-http staging.com --http-basic-auth-filter "^customer[0-9].*"
  -http-disallow-header value       //用一个正则表达式来匹配http头部,匹配到的请求会被拒绝掉
    	A regexp to match a specific header against. Requests with matching headers will be dropped:
    		 gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor"
  -http-disallow-url value      //用一个正则表达式来匹配url,如果请求匹配上了,则会被拒绝
    	A regexp to match requests against. Filter get matched against full url with domain. Anything else will be forwarded:
    		 gor --input-raw :8080 --output-http staging.com --http-disallow-url ^www.
  -http-header-limiter value	//读取请求,基于FNV32-1A散列来拒绝一定比例的特殊请求 
    	Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific header:
    		 gor --input-raw :8080 --output-http staging.com --http-header-limiter user-id:25%
  -http-original-host       //在--output-http的输出中,通常gor会使用取代请求的http头,所以应该禁用该选项,保留原始的主机头
    	Normally gor replaces the Host http header with the host supplied with --output-http.  This option disables that behavior, preserving the original Host header.
  -http-param-limiter value
    	Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific GET param:
    		 gor --input-raw :8080 --output-http staging.com --http-param-limiter user_id:25%
  -http-pprof :8181
    	Enable profiling. Starts  http server on specified port, exposing special /debug/pprof endpoint. Example: :8181
  -http-rewrite-header value
    	Rewrite the request header based on a mapping:
    		gor --input-raw :8080 --output-http staging.com --http-rewrite-header Host: (.*).example.com,$1.beta.example.com
  -http-rewrite-url value
    	Rewrite the request url based on a mapping:
    		gor --input-raw :8080 --output-http staging.com --http-rewrite-url /v1/user/([^\/]+)/ping:/v2/user/$1/ping
  -http-set-header value
    	Inject additional headers to http reqest:
    		gor --input-raw :8080 --output-http staging.com --http-set-header 'User-Agent: Gor'
  -http-set-param value
    	Set request url param, if param already exists it will be overwritten:
    		gor --input-raw :8080 --output-http staging.com --http-set-param api_key=1
  -input-dummy value
    	Used for testing outputs. Emits 'Get /' request every 1s
  -input-file value		//从一个文件中读取请求
    	Read requests from file: 
    		gor --input-file ./requests.gor --output-http staging.com
  -input-file-loop
    	Loop input files, useful for performance testing.
  -input-kafka-host string
    	Send request and response stats to Kafka:
    		gor --output-stdout --input-kafka-host '192.168.0.1:9092,192.168.0.2:9092'
  -input-kafka-json-format
    	If turned on, it will assume that messages coming in JSON format rather than  GoReplay text format.
  -input-kafka-topic string
    	Send request and response stats to Kafka:
    		gor --output-stdout --input-kafka-topic 'kafka-log'
  -input-raw value
    	Capture traffic from given port (use RAW sockets and require *sudo* access):
    		# Capture traffic from 8080 port
    		gor --input-raw :8080 --output-http staging.com
  -input-raw-bpf-filter string
    	BPF filter to write custom expressions. Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80'
  -input-raw-buffer-size int
    	Controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value.
  -input-raw-engine libpcap
    	Intercept traffic using libpcap (default), and `raw_socket` (default "libpcap")
  -input-raw-expire duration
    	How much it should wait for the last TCP packet, till consider that TCP message complete. (default 2s)
  -input-raw-immediate-mode
    	Set pcap interface to immediate mode.
  -input-raw-override-snaplen
    	Override the capture snaplen to be 64k. Required for some Virtualized environments
  -input-raw-realip-header string
    	If not blank, injects header with given name and real IP value to the request payload. Usually this header should be named: X-Real-IP
  -input-raw-timestamp-type string
    	Possible values: PCAP_TSTAMP_HOST, PCAP_TSTAMP_HOST_LOWPREC, PCAP_TSTAMP_HOST_HIPREC, PCAP_TSTAMP_ADAPTER, PCAP_TSTAMP_ADAPTER_UNSYNCED. This values not supported on all systems, GoReplay will tell you available values of you put wrong one.
  -input-raw-track-response
    	If turned on Gor will track responses in addition to requests, and they will be available to middleware and file output.
  -input-tcp value		// 用来在多个gor之间流转流量
    	Used for internal communication between Gor instances. Example: 
    		# Receive requests from other Gor instances on 28020 port, and redirect output to staging
    		gor --input-tcp :28020 --output-http staging.com
  -input-tcp-certificate string
    	Path to PEM encoded certificate file. Used when TLS turned on.
  -input-tcp-certificate-key string
    	Path to PEM encoded certificate key file. Used when TLS turned on.
  -input-tcp-secure
    	Turn on TLS security. Do not forget to specify certificate and key files.
  -memprofile string
    	write memory profile to this file
  -middleware string
    	Used for modifying traffic using external command
  -output-dummy value		//用来测试输入,打印出接收的数据.
    	DEPRECATED: use --output-stdout instead
  -output-file value		//把进入的请求写入一个文件中 
    	Write incoming requests to file: 
    		gor --input-raw :80 --output-file ./requests.gor
  -output-file-append
    	The flushed chunk is appended to existence file or not. 
  -output-file-flush-interval duration
    	Interval for forcing buffer flush to the file, default: 1s. (default 1s)
  -output-file-max-size-limit value
    	Max size of output file, Default: 1TB (default -1)
  -output-file-queue-limit int
    	The length of the chunk queue. Default: 256 (default 256)
  -output-file-size-limit value
    	Size of each chunk. Default: 32mb (default 33554432)
  -output-http value		//转发进入的请求到一个http地址上
    	Forwards incoming requests to given http address.
    		# Redirect all incoming requests to staging.com address 
    		gor --input-raw :80 --output-http http://staging.com
  -output-http-compatibility-mode
    	Use standard Go client, instead of built-in implementation. Can be slower, but more compatible.
  -output-http-debug
    	Enables http debug output.
  -output-http-elasticsearch string		//把请求和响应状态发送到ElasticSearch
    	Send request and response stats to ElasticSearch:
    		gor --input-raw :8080 --output-http staging.com --output-http-elasticsearch 'es_host:api_port/index_name'
  -output-http-header --output-http-header
    	WARNING: --output-http-header DEPRECATED, use `--http-set-header` instead
  -output-http-header-filter --output-http-header-filter
    	WARNING: --output-http-header-filter DEPRECATED, use `--http-allow-header` instead
  -output-http-header-hash-filter output-http-header-hash-filter
    	WARNING: output-http-header-hash-filter DEPRECATED, use `--http-header-hash-limiter` instead
  -output-http-method --output-http-method
    	WARNING: --output-http-method DEPRECATED, use `--http-allow-method` instead
  -output-http-queue-len int
    	Number of requests that can be queued for output, if all workers are busy. default = 1000 (default 1000)
  -output-http-redirects int		//设置多少次重定向被允许
    	Enable how often redirects should be followed.
  -output-http-response-buffer int
    	HTTP response buffer size, all data after this size will be discarded.
  -output-http-rewrite-url --output-http-rewrite-url
    	WARNING: --output-http-rewrite-url DEPRECATED, use `--http-rewrite-url` instead
  -output-http-stats		//每5秒钟输出一次输出队列的状态
    	Report http output queue stats to console every N milliseconds. See output-http-stats-ms
  -output-http-stats-ms int
    	Report http output queue stats to console every N milliseconds. default: 5000 (default 5000)
  -output-http-timeout duration		//指定http的request/response超时时间,默认是5秒 
    	Specify HTTP request/response timeout. By default 5s. Example: --output-http-timeout 30s (default 5s)
  -output-http-track-response
    	If turned on, HTTP output responses will be set to all outputs like stdout, file and etc.
  -output-http-url-regexp --output-http-url-regexp
    	WARNING: --output-http-url-regexp DEPRECATED, use `--http-allow-url` instead
  -output-http-workers int		// gor默认是动态的扩展工作者数量,你也可以指定固定数量的工作者
    	Gor uses dynamic worker scaling. Enter a number to set a maximum number of workers. default = 0 = unlimited.
  -output-http-workers-min int
    	Gor uses dynamic worker scaling. Enter a number to set a minimum number of workers. default = 1.
  -output-kafka-host string
    	Read request and response stats from Kafka:
    		gor --input-raw :8080 --output-kafka-host '192.168.0.1:9092,192.168.0.2:9092'
  -output-kafka-json-format
    	If turned on, it will serialize messages from GoReplay text format to JSON.
  -output-kafka-topic string
    	Read request and response stats from Kafka:
    		gor --input-raw :8080 --output-kafka-topic 'kafka-log'
  -output-null
    	Used for testing inputs. Drops all requests.
  -output-stdout
    	Used for testing inputs. Just prints to console data coming from inputs.
  -output-tcp value		//用来在多个gor之间流转流量
    	Used for internal communication between Gor instances. Example: 
    		# Listen for requests on 80 port and forward them to other Gor instance on 28020 port
    		gor --input-raw :80 --output-tcp replay.local:28020
  -output-tcp-secure
    	Use TLS secure connection. --input-file on another end should have TLS turned on as well.
  -output-tcp-stats		//每5秒钟报告一次tcp输出队列的状态
    	Report TCP output queue stats to console every 5 seconds.
  -prettify-http
    	If enabled, will automatically decode requests and responses with: Content-Encodning: gzip and Transfer-Encoding: chunked. Useful for debugging, in conjuction with --output-stdout
  -split-output true
    	By default each output gets same traffic. If set to true it splits traffic equally among all outputs.
  -stats		//打开输出队列的状态
    	Turn on queue stats output
  -verbose
    	Turn on more verbose output
Licensed under CC BY-NC-SA 4.0
Built with Hugo
主题 StackJimmy 设计