镜缘浮影 小人本住在 苏州的城外 家里有屋又有田 生活乐无边

Codis 基础

2015-12-05
wilmosfang
原文地址 http://soft.dog/2015/12/05/codis/

前言

Codis 是一个使用Go语言编写的Redis集群代理,是一种 Redis 的分布式集群解决方案,支持管道和动态弹性扩容

redis集群方案对比

除了 CodisTwemproxyRedis Cluster 也能实现Redis集群,下面是它们的对比

Feature Codis Twemproxy Redis Cluster
resharding without restarting cluster Yes No Yes
pipeline Yes Yes No
hash tags for multi-key operations Yes Yes Yes
multi-key operations while resharding Yes - No(details)
Redis clients supporting Any clients Any clients Clients have to support cluster protocol

架构图

codis_architecture.png

其它特性

  • 具备GUI网页管理界面
  • 支持绝大部分(并非所有)Redis命令,完全兼容Twemproxy
  • 代理可以注册到 zk/etcd ,构建高可用集群

下面分享一下它的基础操作,详细可以参考 官方文档

Tip: 貌似没有自己的版本信息,不过当前的版本基于 Redis 2.8.21 开发的

[root@h101 bin]# ./codis-server --version
Redis server v=2.8.21 sha=4a332efd:0 malloc=jemalloc-3.6.0 bits=64 build=569d3bc356356f21
[root@h101 bin]# 

概要


安装Codis

由于 Codis 是基于Go语言的,所以得先安装Go


安装Go

Tip: Go当前最新版本为 go1.5.2

[root@h102 ~]# wget https://storage.googleapis.com/golang/go1.5.2.linux-amd64.tar.gz
--2015-12-04 09:49:42--  https://storage.googleapis.com/golang/go1.5.2.linux-amd64.tar.gz
Resolving storage.googleapis.com... 216.58.221.80, 2404:6800:4005:80a::2010
Connecting to storage.googleapis.com|216.58.221.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 76350075 (73M) [application/octet-stream]
Saving to: “go1.5.2.linux-amd64.tar.gz”

100%[============================================================================================>] 76,350,075  1.39M/s   in 51s     

2015-12-04 09:50:33 (1.44 MB/s) - “go1.5.2.linux-amd64.tar.gz” saved [76350075/76350075]

[root@h102 ~]# ll go1.5.2.linux-amd64.tar.gz 
-rw-r--r-- 1 root root 76350075 Dec  4 05:32 go1.5.2.linux-amd64.tar.gz
[root@h102 ~]# du -sh go1.5.2.linux-amd64.tar.gz 
73M	go1.5.2.linux-amd64.tar.gz
[root@h102 ~]# tar -C /usr/local/  -xzvf go1.5.2.linux-amd64.tar.gz 
go/
go/AUTHORS
go/CONTRIBUTING.md
go/CONTRIBUTORS
...
...
go/test/undef.go
go/test/utf.go
go/test/varerr.go
go/test/varinit.go
go/test/writebarrier.go
go/test/zerodivide.go
[root@h102 ~]#

Tip: Go语言的详细安装文档可以参考 The Go Programming Language , Go语言各种版本的 下载地址


配置Go环境

[root@h102 ~]# export GOROOT=/usr/local/go
[root@h102 ~]# export PATH=$PATH:$GOROOT/bin
[root@h102 ~]# export GOPATH=/root/go_home
[root@h102 ~]# export PATH=$PATH:$GOPATH/bin
[root@h102 ~]# env | grep -i path
rvm_bin_path=/usr/local/rvm/bin
rvm_path=/usr/local/rvm
PATH=/usr/local/rvm/gems/ruby-2.2.1/bin:/usr/local/rvm/gems/ruby-2.2.1@global/bin:/usr/local/rvm/rubies/ruby-2.2.1/bin:/usr/lib64/qt.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin:/usr/local/go/bin:/root/go_home/bin
GEM_PATH=/usr/local/rvm/gems/ruby-2.2.1:/usr/local/rvm/gems/ruby-2.2.1@global
GOPATH=/root/go_home
[root@h102 ~]# env | grep -i root
USER=root
MAIL=/var/spool/mail/root
PATH=/usr/local/rvm/gems/ruby-2.2.1/bin:/usr/local/rvm/gems/ruby-2.2.1@global/bin:/usr/local/rvm/rubies/ruby-2.2.1/bin:/usr/lib64/qt.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin:/usr/local/go/bin:/root/go_home/bin
PWD=/root
HOME=/root
GOROOT=/usr/local/go
LOGNAME=root
GOPATH=/root/go_home
[root@h102 ~]# 

GOROOT : Go语言的根目录

GOPATH : Go项目的根目录

PATH 里要将上面两者的 /bin 目录加入

go_home 是在家里手动创建的用于构建Go项目的工作目录

查看Go的版本和环境

[root@h102 ~]# go version
go version go1.5.2 linux/amd64
[root@h102 ~]# go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go_home"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
[root@h102 ~]# 

下载codis代码

直接使用 go get -u -d github.com/wandoulabs/codis 来获取codis源码

[root@h102 ~]# go get -u -d github.com/wandoulabs/codis
package github.com/wandoulabs/codis: no buildable Go source files in /root/go_home/src/github.com/wandoulabs/codis
[root@h102 ~]# cd go_home/src/github.com/wandoulabs/
[root@h102 wandoulabs]# du -sh codis/
25M	codis/
[root@h102 wandoulabs]# cd codis/
[root@h102 codis]# ls
cmd  config.ini  doc  docker  Dockerfile  extern  genver.sh  Godeps  Makefile  MIT-LICENSE.txt  pkg  README.md  test  vitess_license
[root@h102 codis]#

Tip: 官方建议只通过go get命令来下载codis,除非自己非常熟悉go语言的目录引用形式从而不会导致代码放错地方,该命令会下载master分支的最新版,作者会确保master分支的稳定


编译codis

[root@h102 codis]# make 
GOPATH=`godep path` godep restore
GOPATH=`godep path`:$GOPATH go build -o bin/codis-proxy ./cmd/proxy
GOPATH=`godep path`:$GOPATH go build -o bin/codis-config ./cmd/cconfig
make -j4 -C extern/redis-2.8.21/
make[1]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21'
cd src && make all
make[2]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src'
rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-dump redis-check-aof *.o *.gcda *.gcno *.gcov redis.info lcov-html
(cd ../deps && make distclean)
make[3]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps'
(cd hiredis && make clean) > /dev/null || true
(cd linenoise && make clean) > /dev/null || true
(cd lua && make clean) > /dev/null || true
(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true
(rm -f .make-*)
make[3]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps'
(rm -f .make-*)
echo STD=-std=c99 -pedantic >> .make-settings
echo WARN=-Wall -W >> .make-settings
echo OPT=-O2 >> .make-settings
echo MALLOC=jemalloc >> .make-settings
echo CFLAGS= >> .make-settings
echo LDFLAGS= >> .make-settings
echo REDIS_CFLAGS= >> .make-settings
echo REDIS_LDFLAGS= >> .make-settings
echo PREV_FINAL_CFLAGS=-std=c99 -pedantic -Wall -W -O2 -g -ggdb   -I../deps/hiredis -I../deps/linenoise -I../deps/lua/src -DUSE_JEMALLOC -I../deps/jemalloc/include >> .make-settings
echo PREV_FINAL_LDFLAGS=  -g -ggdb -rdynamic >> .make-settings
(cd ../deps && make hiredis linenoise lua jemalloc)
make[3]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps'
(cd hiredis && make clean) > /dev/null || true
(cd linenoise && make clean) > /dev/null || true
(cd lua && make clean) > /dev/null || true
(cd jemalloc && [ -f Makefile ] && make distclean) > /dev/null || true
(rm -f .make-*)
(echo "" > .make-ldflags)
(echo "" > .make-cflags)
MAKE hiredis
MAKE linenoise
cd hiredis && make static
cd linenoise && make
MAKE lua
cd lua/src && make all CFLAGS="-O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL " MYLDFLAGS="" AR="ar rcu"
make[4]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/linenoise'
cc  -Wall -Os -g  -c linenoise.c
make[4]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/lua/src'
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lapi.o lapi.c
MAKE jemalloc
cd jemalloc && ./configure --with-jemalloc-prefix=je_ --enable-cc-silence CFLAGS="-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops " LDFLAGS=""
make[4]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/hiredis'
cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  net.c
checking for xsltproc... /usr/bin/xsltproc
checking for gcc... gcc
checking whether the C compiler works... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lcode.o lcode.c
make[4]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/linenoise'
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ldebug.o ldebug.c
yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  hiredis.c
no
checking for suffix of object files... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ldo.o ldo.c
o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ldump.o ldump.c
yes
checking for gcc option to accept ISO C89... ldo.c: In function ‘f_parser’:
ldo.c:496: warning: unused variable ‘c’
none needed
checking how to run the C preprocessor... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lfunc.o lfunc.c
gcc -E
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lgc.o lgc.c
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o llex.o llex.c
yes
checking for sys/types.h... cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  sds.c
yes
checking for sys/stat.h... yes
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lmem.o lmem.c
checking for stdlib.h... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lobject.o lobject.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lopcodes.o lopcodes.c
yes
checking for string.h... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lparser.o lparser.c
yes
checking for memory.h... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lstate.o lstate.c
cc -std=c99 -pedantic -c -O3 -fPIC  -Wall -W -Wstrict-prototypes -Wwrite-strings -g -ggdb  async.c
yes
checking for strings.h... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lstring.o lstring.c
yes
checking for inttypes.h... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ltable.o ltable.c
yes
checking for stdint.h... yes
checking for unistd.h... yes
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ltm.o ltm.c
ar rcs libhiredis.a net.o hiredis.o sds.o async.o
checking whether byte ordering is bigendian... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lundump.o lundump.c
make[4]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/hiredis'
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lvm.o lvm.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lzio.o lzio.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o strbuf.o strbuf.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o fpconv.o fpconv.c
no
checking size of void *... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lauxlib.o lauxlib.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lbaselib.o lbaselib.c
8
checking size of int... 4
checking size of long... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ldblib.o ldblib.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o liolib.o liolib.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lmathlib.o lmathlib.c
8
checking size of intmax_t... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o loslib.o loslib.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o ltablib.o ltablib.c
8
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lstrlib.o lstrlib.c
checking build system type... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o loadlib.o loadlib.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o linit.o linit.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lua_cjson.o lua_cjson.c
x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking whether pause instruction is compilable... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lua_struct.o lua_struct.c
yes
checking whether SSE2 intrinsics is compilable... yes
checking for ar... ar
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lua_cmsgpack.o lua_cmsgpack.c
checking whether __attribute__ syntax is compilable... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lua_bit.o lua_bit.c
yes
checking whether compiler supports -fvisibility=hidden... yes
checking whether compiler supports -Werror... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o lua.o lua.c
cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o luac.o luac.c
yes
checking whether tls_model attribute is compilable... yes
checking for a BSD-compatible install... cc -O2 -Wall -DLUA_ANSI -DENABLE_CJSON_GLOBAL    -c -o print.o print.c
/usr/bin/install -c
checking for ranlib... ranlib
checking for ld... /usr/bin/ld
checking for autoconf... /usr/bin/autoconf
checking for memalign... ar rcu liblua.a lapi.o lcode.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o strbuf.o fpconv.o lauxlib.o lbaselib.o ldblib.o liolib.o lmathlib.o loslib.o ltablib.o lstrlib.o loadlib.o linit.o lua_cjson.o lua_struct.o lua_cmsgpack.o lua_bit.o	# DLL needs all object files
ranlib liblua.a
cc -o lua  lua.o liblua.a -lm 
liblua.a(loslib.o): In function `os_tmpname':
loslib.c:(.text+0x35): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
yes
checking for valloc... cc -o luac  luac.o print.o liblua.a -lm 
make[4]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/lua/src'
yes
checking configured backtracing method... N/A
checking for sbrk... yes
checking whether utrace(2) is compilable... no
checking whether valgrind is compilable... no
checking STATIC_PAGE_SHIFT... 12
checking pthread.h usability... yes
checking pthread.h presence... yes
checking for pthread.h... yes
checking for pthread_create in -lpthread... yes
checking for _malloc_thread_cleanup... no
checking for _pthread_mutex_init_calloc_cb... no
checking for TLS... yes
checking whether a program using ffsl is compilable... yes
checking whether atomic(9) is compilable... no
checking whether Darwin OSAtomic*() is compilable... no
checking whether to force 32-bit __sync_{add,sub}_and_fetch()... no
checking whether to force 64-bit __sync_{add,sub}_and_fetch()... no
checking whether Darwin OSSpin*() is compilable... no
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating doc/html.xsl
config.status: creating doc/manpages.xsl
config.status: creating doc/jemalloc.xml
config.status: creating include/jemalloc/jemalloc_macros.h
config.status: creating include/jemalloc/jemalloc_protos.h
config.status: creating include/jemalloc/internal/jemalloc_internal.h
config.status: creating test/test.sh
config.status: creating test/include/test/jemalloc_test.h
config.status: creating config.stamp
config.status: creating bin/jemalloc.sh
config.status: creating include/jemalloc/jemalloc_defs.h
config.status: creating include/jemalloc/internal/jemalloc_internal_defs.h
config.status: creating test/include/test/jemalloc_test_defs.h
config.status: executing include/jemalloc/internal/private_namespace.h commands
config.status: executing include/jemalloc/internal/private_unnamespace.h commands
config.status: executing include/jemalloc/internal/public_symbols.txt commands
config.status: executing include/jemalloc/internal/public_namespace.h commands
config.status: executing include/jemalloc/internal/public_unnamespace.h commands
config.status: executing include/jemalloc/internal/size_classes.h commands
config.status: executing include/jemalloc/jemalloc_protos_jet.h commands
config.status: executing include/jemalloc/jemalloc_rename.h commands
config.status: executing include/jemalloc/jemalloc_mangle.h commands
config.status: executing include/jemalloc/jemalloc_mangle_jet.h commands
config.status: executing include/jemalloc/jemalloc.h commands
===============================================================================
jemalloc version   : 3.6.0-0-g46c0af68bd248b04df75e4f92d5fb804c3d75340
library revision   : 1

CC                 : gcc
CPPFLAGS           :  -D_GNU_SOURCE -D_REENTRANT
CFLAGS             : -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -fvisibility=hidden
LDFLAGS            : 
EXTRA_LDFLAGS      : 
LIBS               :  -lpthread
RPATH_EXTRA        : 

XSLTPROC           : /usr/bin/xsltproc
XSLROOT            : /usr/share/sgml/docbook/xsl-stylesheets

PREFIX             : /usr/local
BINDIR             : /usr/local/bin
INCLUDEDIR         : /usr/local/include
LIBDIR             : /usr/local/lib
DATADIR            : /usr/local/share
MANDIR             : /usr/local/share/man

srcroot            : 
abs_srcroot        : /root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/jemalloc/
objroot            : 
abs_objroot        : /root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/jemalloc/

JEMALLOC_PREFIX    : je_
JEMALLOC_PRIVATE_NAMESPACE
                   : je_
install_suffix     : 
autogen            : 0
experimental       : 1
cc-silence         : 1
debug              : 0
code-coverage      : 0
stats              : 1
prof               : 0
prof-libunwind     : 0
prof-libgcc        : 0
prof-gcc           : 0
tcache             : 1
fill               : 1
utrace             : 0
valgrind           : 0
xmalloc            : 0
mremap             : 0
munmap             : 0
dss                : 0
lazy_lock          : 0
tls                : 1
===============================================================================
cd jemalloc && make CFLAGS="-std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops " LDFLAGS="" lib/libjemalloc.a
make[4]: Entering directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/jemalloc'
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/jemalloc.o src/jemalloc.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/arena.o src/arena.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/atomic.o src/atomic.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/base.o src/base.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/bitmap.o src/bitmap.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/chunk.o src/chunk.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/chunk_dss.o src/chunk_dss.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/chunk_mmap.o src/chunk_mmap.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ckh.o src/ckh.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/ctl.o src/ctl.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/extent.o src/extent.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/hash.o src/hash.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/huge.o src/huge.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mb.o src/mb.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/mutex.o src/mutex.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/prof.o src/prof.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/quarantine.o src/quarantine.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/rtree.o src/rtree.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/stats.o src/stats.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tcache.o src/tcache.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/util.o src/util.c
gcc -std=gnu99 -Wall -pipe -g3 -O3 -funroll-loops  -c -D_GNU_SOURCE -D_REENTRANT -Iinclude -Iinclude -o src/tsd.o src/tsd.c
ar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/atomic.o src/base.o src/bitmap.o src/chunk.o src/chunk_dss.o src/chunk_mmap.o src/ckh.o src/ctl.o src/extent.o src/hash.o src/huge.o src/mb.o src/mutex.o src/prof.o src/quarantine.o src/rtree.o src/stats.o src/tcache.o src/util.o src/tsd.o
make[4]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps/jemalloc'
make[3]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/deps'
    CC adlist.o
    CC ae.o
    CC anet.o
    CC dict.o
anet.c: In function ‘anetSockName’:
anet.c:565: warning: dereferencing pointer ‘s’ does break strict-aliasing rules
anet.c:563: note: initialized from here
anet.c:569: warning: dereferencing pointer ‘s’ does break strict-aliasing rules
anet.c:567: note: initialized from here
    CC redis.o
anet.c: In function ‘anetPeerToString’:
anet.c:543: warning: dereferencing pointer ‘s’ does break strict-aliasing rules
anet.c:541: note: initialized from here
anet.c:547: warning: dereferencing pointer ‘s’ does break strict-aliasing rules
anet.c:545: note: initialized from here
anet.c: In function ‘anetTcpAccept’:
anet.c:511: warning: dereferencing pointer ‘s’ does break strict-aliasing rules
anet.c:509: note: initialized from here
anet.c:515: warning: dereferencing pointer ‘s’ does break strict-aliasing rules
anet.c:513: note: initialized from here
    CC sds.o
    CC zmalloc.o
    CC lzf_c.o
    CC lzf_d.o
    CC pqsort.o
    CC zipmap.o
    CC sha1.o
    CC ziplist.o
    CC release.o
    CC networking.o
    CC util.o
    CC object.o
    CC db.o
    CC replication.o
    CC rdb.o
    CC t_string.o
db.c: In function ‘scanGenericCommand’:
db.c:454: warning: ‘pat’ may be used uninitialized in this function
db.c:455: warning: ‘patlen’ may be used uninitialized in this function
    CC t_list.o
    CC t_set.o
    CC t_zset.o
    CC t_hash.o
    CC config.o
    CC aof.o
    CC pubsub.o
    CC multi.o
    CC debug.o
    CC sort.o
    CC intset.o
    CC syncio.o
    CC migrate.o
    CC endianconv.o
    CC slowlog.o
    CC scripting.o
    CC bio.o
    CC rio.o
    CC rand.o
    CC memtest.o
    CC crc64.o
    CC crc32.o
    CC bitops.o
    CC sentinel.o
    CC notify.o
    CC setproctitle.o
    CC hyperloglog.o
    CC latency.o
    CC sparkline.o
    CC slots.o
    CC redis-cli.o
    CC redis-benchmark.o
    CC redis-check-dump.o
    CC redis-check-aof.o
    LINK redis-benchmark
    LINK redis-check-aof
    LINK redis-server
    LINK redis-check-dump
    INSTALL redis-sentinel
    LINK redis-cli

Hint: It's a good idea to run 'make test' ;)

make[2]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src'
make[1]: Leaving directory `/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21'
[root@h102 codis]# echo $?
0
[root@h102 codis]# 

可以使用 make gotest 进行测试 (可选操作,非必须)

[root@h102 codis]# make gotest
GOPATH=`godep path`:$GOPATH go test ./pkg/... ./cmd/...
ok  	github.com/wandoulabs/codis/pkg/models	16.501s
ok  	github.com/wandoulabs/codis/pkg/proxy	9.242s
ok  	github.com/wandoulabs/codis/pkg/proxy/redis	12.355s
ok  	github.com/wandoulabs/codis/pkg/proxy/router	1.419s
?   	github.com/wandoulabs/codis/pkg/utils	[no test files]
?   	github.com/wandoulabs/codis/pkg/utils/assert	[no test files]
?   	github.com/wandoulabs/codis/pkg/utils/atomic2	[no test files]
ok  	github.com/wandoulabs/codis/pkg/utils/bytesize	0.003s
?   	github.com/wandoulabs/codis/pkg/utils/errors	[no test files]
?   	github.com/wandoulabs/codis/pkg/utils/log	[no test files]
?   	github.com/wandoulabs/codis/pkg/utils/trace	[no test files]
?   	github.com/wandoulabs/codis/cmd/cconfig	[no test files]
?   	github.com/wandoulabs/codis/cmd/proxy	[no test files]
[root@h102 codis]# echo $?
0
[root@h102 codis]# 

生成可执行文件

[root@h102 codis]# ll bin/
total 37632
drwxr-xr-x 4 root root     4096 Dec  4 15:41 assets
-rwxr-xr-x 1 root root 16195952 Dec  4 15:41 codis-config
-rwxr-xr-x 1 root root 16017064 Dec  4 15:41 codis-proxy
-rwxr-xr-x 1 root root  6311083 Dec  4 15:42 codis-server
[root@h102 codis]# file bin/*
bin/assets:       directory
bin/codis-config: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped
bin/codis-proxy:  ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped
bin/codis-server: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, 
[root@h102 codis]# 

Tip: 执行完上面全部指令后,会在 bin 文件夹内生成 codis-config、codis-proxy、codis-server三个可执行文件;另外, bin/assets 文件夹是 codis-config 的 dashboard http 服务需要的前端资源, 需要和 codis-config 放置在同一文件夹下

三个命令的基本帮助信息如下

[root@h102 codis]# bin/codis-config -h
usage: codis-config  [-c <config_file>] [-L <log_file>] [--log-level=<loglevel>]
		<command> [<args>...]
options:
   -c	set config file
   -L	set output log file, default is stdout
   --log-level=<loglevel>	set log level: info, warn, error, debug [default: info]

commands:
	server
	slot
	dashboard
	action
	proxy
[root@h102 codis]# bin/codis-proxy -h
usage: proxy [-c <config_file>] [-L <log_file>] [--log-level=<loglevel>] [--log-filesize=<filesize>] [--cpu=<cpu_num>] [--addr=<proxy_listen_addr>] [--http-addr=<debug_http_server_addr>]

options:
   -c	set config file
   -L	set output log file, default is stdout
   --log-level=<loglevel>	set log level: info, warn, error, debug [default: info]
   --log-filesize=<maxsize>  set max log file size, suffixes "KB", "MB", "GB" are allowed, 1KB=1024 bytes, etc. Default is 1GB.
   --cpu=<cpu_num>		num of cpu cores that proxy can use
   --addr=<proxy_listen_addr>		proxy listen address, example: 0.0.0.0:9000
   --http-addr=<debug_http_server_addr>		debug vars http server
[root@h102 codis]# bin/codis-server -h
Usage: ./redis-server [/path/to/redis.conf] [options]
       ./redis-server - (read config from stdin)
       ./redis-server -v or --version
       ./redis-server -h or --help
       ./redis-server --test-memory <megabytes>

Examples:
       ./redis-server (run the server with default conf)
       ./redis-server /etc/redis/6379.conf
       ./redis-server --port 7777
       ./redis-server --port 7777 --slaveof 127.0.0.1 8888
       ./redis-server /etc/myredis.conf --loglevel verbose

Sentinel mode:
       ./redis-server /etc/sentinel.conf --sentinel
[root@h102 codis]# 

部署Codis集群

启动zookeeper服务

配置 zookeeper

[root@h101 zookeeper-Standalone]# grep -v "^#" conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper
dataLogDir=/tmp/zookeeper
clientPort=2180
[root@h101 zookeeper-Standalone]# 

启动 zookeeper

[root@h101 zookeeper-Standalone]# bin/zkServer.sh start 
JMX enabled by default
Using config: /root/zk/zookeeper-Standalone/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@h101 zookeeper-Standalone]#

检查 zookeeper 状态

[root@h101 zookeeper-Standalone]# cat /tmp/zookeeper/zookeeper_server.pid 
9632[root@h101 zookeeper-Standalone]# 
[root@h101 zookeeper-Standalone]# ps faux | grep zookee
root      9652  0.0  0.0 103256   828 pts/0    S+   17:53   0:00  |       \_ grep zookee
root      9632  8.8  2.0 2094296 40108 pts/0   Sl   17:53   0:00 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /root/zk/zookeeper-Standalone/bin/../build/classes:/root/zk/zookeeper-Standalone/bin/../build/lib/*.jar:/root/zk/zookeeper-Standalone/bin/../lib/slf4j-log4j12-1.6.1.jar:/root/zk/zookeeper-Standalone/bin/../lib/slf4j-api-1.6.1.jar:/root/zk/zookeeper-Standalone/bin/../lib/netty-3.7.0.Final.jar:/root/zk/zookeeper-Standalone/bin/../lib/log4j-1.2.16.jar:/root/zk/zookeeper-Standalone/bin/../lib/jline-0.9.94.jar:/root/zk/zookeeper-Standalone/bin/../zookeeper-3.4.6.jar:/root/zk/zookeeper-Standalone/bin/../src/java/lib/*.jar:/root/zk/zookeeper-Standalone/bin/../conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /root/zk/zookeeper-Standalone/bin/../conf/zoo.cfg
[root@h101 zookeeper-Standalone]# netstat  -ant | grep 218
tcp        0      0 :::2180                     :::*                        LISTEN      
[root@h101 zookeeper-Standalone]# iptables -L -nv | grep 218
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:2180 
[root@h101 zookeeper-Standalone]#

Note: 不事先启动zk 而启动后面的服务会报错


启动codis dashboard

配置 config.ini

Note: codis-config 和 codis-proxy 在不加 -c 参数的时候, 默认会读取当前目录下的 config.ini 文件

[root@h102 codis]# grep -v "^#" config.ini  | grep -v "^$"
coordinator=zookeeper
zk=192.168.100.101:2180
product=test
dashboard_addr=192.168.100.102:18087
password=codis
backend_ping_period=5
session_max_timeout=1800
session_max_bufsize=131072
session_max_pipeline=1024
zk_session_timeout=30000
proxy_id=proxy_1
[root@h102 codis]# 

启动codis dashboard

使用 bin/codis-config dashboard 启动 dashboard

[root@h102 codis]# bin/codis-config dashboard
2015/12/04 17:56:00 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2015/12/04 17:56:00 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "192.168.100.102:18087", "pid": 35057}
2015/12/04 17:56:00 dashboard.go:144: [WARN] ********** Attention **********
2015/12/04 17:56:00 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2015/12/04 17:56:00 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually
2015/12/04 17:56:00 dashboard.go:147: [WARN] *******************************
...
...
...

会在当前终端启动一个端口为18087的web server


打开防火墙

[root@h102 codis]# iptables -L -nv | grep  18087 
[root@h102 codis]# vim /etc/sysconfig/iptables
[root@h102 codis]# grep  18087  /etc/sysconfig/iptables
-A INPUT -p tcp -m state --state NEW -m tcp --dport 18087 -j ACCEPT 
[root@h102 codis]# /etc/init.d/iptables  reload 
iptables: Trying to reload firewall rules:                 [  OK  ]
[root@h102 codis]# iptables -L -nv | grep  18087 
   0   0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:18087 
[root@h102 codis]# 

访问dashboard

codis_dashboard.png


初始化 slots

[root@h102 codis]# bin/codis-config  slot help
usage:
	codis-config slot init [-f]
	codis-config slot info <slot_id>
	codis-config slot set <slot_id> <group_id> <status>
	codis-config slot range-set <slot_from> <slot_to> <group_id> <status>
	codis-config slot migrate <slot_from> <slot_to> <group_id> [--delay=<delay_time_in_ms>]
	codis-config slot rebalance [--delay=<delay_time_in_ms>]
[root@h102 codis]# bin/codis-config slot init 
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# 

此时通过管理界面就可以看到slot信息(初始化之前是空白的)

codis_slot.png


启动 Codis Server

extern 目录中包含一整个 redis-2.8.21 的版本

[root@h102 codis]# grep -v "^#" extern/redis-2.8.21/redis.conf  | grep -v "^$"
daemonize yes
pidfile /var/run/redis.pid
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel notice
logfile ""
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
[root@h102 codis]# bin/codis-server extern/redis-2.8.21/redis.conf
[root@h102 codis]# 

检查状态

[root@h102 codis]#  ps faux | grep codis |  grep -v grep 
root     35057  1.5  2.5 448184 49352 pts/0    Sl+  17:55   2:32  |       \_ bin/codis-config dashboard
root     35742  0.1  0.5 137392  9592 ?        Ssl  20:39   0:00 bin/codis-server *:6379                        
[root@h102 codis]# cat /var/run/redis.pid
35742
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6379 info 
# Server
redis_version:2.8.21
redis_git_sha1:4a332efd
redis_git_dirty:0
redis_build_id:2aa3ac6d720af7b0
redis_mode:standalone
os:Linux 2.6.32-504.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:35742
run_id:0506f862c0562d04c11c5e018d59f0ec74d72617
tcp_port:6379
uptime_in_seconds:445
uptime_in_days:0
hz:10
lru_clock:6392636
config_file:/root/go_home/src/github.com/wandoulabs/codis/extern/redis-2.8.21/redis.conf

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:2530152
used_memory_human:2.41M
used_memory_rss:9822208
used_memory_peak:2530152
used_memory_peak_human:2.41M
used_memory_lua:36864
mem_fragmentation_ratio:3.88
mem_allocator:jemalloc-3.6.0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1449232767
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:3
total_commands_processed:2
instantaneous_ops_per_sec:0
total_net_input_bytes:42
total_net_output_bytes:3829
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.36
used_cpu_user:0.43
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Keyspace
[root@h102 codis]#

添加 Redis Server Group

准备配置文件

[root@h102 codis]# grep -v "^#" extern/redis-2.8.21/redis.conf  | grep -v "^$"  > /etc/codis/redis_s.conf
[root@h102 codis]# grep -v "^#" extern/redis-2.8.21/redis.conf  | grep -v "^$"  > /etc/codis/redis2.conf
[root@h102 codis]# grep -v "^#" extern/redis-2.8.21/redis.conf  | grep -v "^$"  > /etc/codis/redis2_s.conf
[root@h102 codis]# vim  /etc/codis/redis_s.conf
[root@h102 codis]# vim  /etc/codis/redis2.conf
[root@h102 codis]# vim  /etc/codis/redis2_s.conf

配置文件主要区别

[root@h102 codis]# diff /etc/codis/redis2.conf /etc/codis/redis2_s.conf
2,3c2,3
< pidfile /var/run/redis2.pid
< port 6389
---
> pidfile /var/run/redis2_s.pid
> port 6390
16c16
< dbfilename dump2.rdb
---
> dbfilename dump2_s.rdb
25c25
< appendfilename "appendonly2.aof"
---
> appendfilename "appendonly2_s.aof"
[root@h102 codis]#

启动多个服务

[root@h102 codis]# bin/codis-server /etc/codis/redis_s.conf 
[root@h102 codis]# bin/codis-server /etc/codis/redis2.conf 
[root@h102 codis]# bin/codis-server /etc/codis/redis2_s.conf             
[root@h102 codis]#

服务状态

[root@h102 codis]# ll /var/run/redis*
-rw-r--r-- 1 root root 6 Dec  4 21:00 /var/run/redis2.pid
-rw-r--r-- 1 root root 6 Dec  4 21:00 /var/run/redis2_s.pid
-rw-r--r-- 1 root root 6 Dec  4 20:39 /var/run/redis.pid
-rw-r--r-- 1 root root 6 Dec  4 21:00 /var/run/redis_s.pid
[root@h102 codis]# cat /var/run/redis*
35894
35898
35742
35890
[root@h102 codis]# ps faux | grep codis | grep -v grep 
root     35057  1.5  2.4 448184 47208 pts/0    Sl+  17:55   2:53  |       \_ bin/codis-config dashboard
root     35742  0.1  0.5 137392  9592 ?        Ssl  20:39   0:02 bin/codis-server *:6379                        
root     35890  0.1  0.4 137392  9548 ?        Ssl  21:00   0:00 bin/codis-server *:6380                 
root     35894  0.1  0.4 137392  9544 ?        Ssl  21:00   0:00 bin/codis-server *:6389                
root     35898  0.1  0.4 137392  9544 ?        Ssl  21:00   0:00 bin/codis-server *:6390                  
[root@h102 codis]# 

添加到group

[root@h102 codis]# bin/codis-config server add 1 localhost:6379 master
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# 
[root@h102 codis]# bin/codis-config server add 1 localhost:6380 slave
2015/12/04 21:10:49 main.go:154: [PANIC] run sub-command failed
[error]: http status code 500, ERR Client sent AUTH, but no password is set
    4   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/utils.go:66
            main.callApi
    3   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/server_group.go:94
            main.runAddServerToGroup
    2   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/server_group.go:54
            main.cmdServer
    1   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/main.go:87
            main.runCommand
    0   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/main.go:152
            main.main
        ... ...
[stack]: 
    0   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/main.go:154
            main.main
        ... ...
[root@h102 codis]# echo $?
1
[root@h102 codis]#
报错

原因为client发送了认证信息,但服务并没有设置密码与认证

解决办法是修改配置,注释掉password这一行

[root@h102 codis]# grep pass config.ini 
#password=codis
[root@h102 codis]# 

再次尝试添加

[root@h102 codis]# bin/codis-config server add 1 localhost:6380 slave
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# bin/codis-config server add 2 localhost:6389 master 
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# bin/codis-config server add 2 localhost:6390 slave
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# 

发现此时,slave的同步关系已经自动建立好了

[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6390  info replication
# Replication
role:slave
master_host:localhost
master_port:6389
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:673
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
[root@h102 codis]# 

查看配置信息

[root@h102 codis]# bin/codis-config server list
[
  {
    "id": 1,
    "product_name": "test",
    "servers": [
      {
        "addr": "localhost:6379",
        "group_id": 1,
        "type": "master"
      },
      {
        "addr": "localhost:6380",
        "group_id": 1,
        "type": "slave"
      }
    ]
  },
  {
    "id": 2,
    "product_name": "test",
    "servers": [
      {
        "addr": "localhost:6390",
        "group_id": 2,
        "type": "slave"
      },
      {
        "addr": "localhost:6389",
        "group_id": 2,
        "type": "master"
      }
    ]
  }
]
[root@h102 codis]#

此刻管理界面里已经多了两个组

codis_servergroup.png


设置 server group 服务的 slot 范围

[root@h102 codis]# bin/codis-config slot range-set 0 500 1 online
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# bin/codis-config slot range-set 501 1023 2 online
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# 

执行完第一条命令后,会发现管理界面里slot状态发生了变化

codis_slot1.png

执行完第二条命令后,管理界面里slot状态又发生了变化

codis_slot2.png

双击其中一个slot会展示其slot信息

codis_slotinfo.png


启动 codis-proxy

[root@h102 codis]# bin/codis-proxy -c config.ini -L /tmp/proxy.log --cpu=2 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000

  _____  ____    ____/ /  (_)  _____
 / ___/ / __ \  / __  /  / /  / ___/
/ /__  / /_/ / / /_/ /  / /  (__  )
\___/  \____/  \__,_/  /_/  /____/
...
...
...

修改codis-proxy状态

[root@h102 codis]# bin/codis-config proxy help
usage:
	codis-config proxy list
	codis-config proxy offline <proxy_name>
	codis-config proxy online <proxy_name>
[root@h102 codis]# bin/codis-config -c config.ini  proxy  list
[
  {
    "addr": "h102.temp:19000",
    "debug_var_addr": "h102.temp:11000",
    "description": "",
    "id": "proxy_1",
    "last_event": "",
    "last_event_ts": 0,
    "pid": 36715,
    "start_at": "2015-12-04 22:37:24.576758464 +0800 CST",
    "state": "offline"
  }
]
[root@h102 codis]# bin/codis-config -c config.ini  proxy  online proxy_1
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]#

管理界面中的proxy状态

codis_proxystatus.png


打开防火墙

[root@h102 ~]# vim /etc/sysconfig/iptables
[root@h102 ~]# grep 1.000 /etc/sysconfig/iptables
-A INPUT -p tcp -m state --state NEW -m tcp --dport 19000 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 11000 -j ACCEPT 
[root@h102 ~]# /etc/init.d/iptables reload 
iptables: Trying to reload firewall rules:                 [  OK  ]
[root@h102 ~]# iptables -L -nv | grep 1.000
    3   148 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:19000 
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:11000 
[root@h102 ~]# 

debug信息

打开浏览器进入 http://ip:11000/debug/vars 可以获取一些debug信息

codis_debug.png


数据迁移

有两种方法进行数据迁移

  • 使用web管理界面
  • 使用命令行

使用管理界面迁移数据

codis_GUI_migrate_data.png

点击[OK]后会展示正在迁移的slot队列,一次只能迁移一个

codis_GUI_migrate_data5_15.png

迁移完成后slot的状态

codis_GUI_migrate_slot5_15.png

使用命令行迁移数据

[root@h102 codis]# bin/codis-config slot migrate 7 12 1
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# 

管理界面也会作出相应反映

codis_GUI_migrate_data7_12.png

迁移完成后的状态

codis_GUI_migrate_slot7_12.png

Tip: 迁移的过程对于上层业务来说是安全且透明的, 数据不会丢失, 上层不会中止服务.

Note: 迁移的过程中打断是可以的, 但是如果中断了一个正在迁移某个slot的任务, 下次需要先迁移掉正处于迁移状态的 slot, 否则无法继续 (即迁移程序会检查同一时刻只能有一个 slot 处于迁移状态).


自动均衡

要求:

  • 所有的codis-server都必须设置了maxmemory参数
  • 所有的 slots 都应该处于 online 状态, 即没有迁移任务正在执行
  • 所有 server group 都必须有 Master

系统默认maxmemory值是0

[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6379 CONFIG GET maxmemory
1) "maxmemory"
2) "0"
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6380 CONFIG GET maxmemory
1) "maxmemory"
2) "0"
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6389 CONFIG GET maxmemory
1) "maxmemory"
2) "0"
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6390 CONFIG GET maxmemory
1) "maxmemory"
2) "0"
[root@h102 codis]#

修改此值

[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6379 CONFIG set maxmemory 100
OK
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6389 CONFIG set maxmemory 100
OK
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6390 CONFIG set maxmemory 100
OK
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6380 CONFIG set maxmemory 100
OK
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6379 CONFIG GET maxmemory
1) "maxmemory"
2) "100"
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6380 CONFIG GET maxmemory
1) "maxmemory"
2) "100"
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6389 CONFIG GET maxmemory
1) "maxmemory"
2) "100"
[root@h102 codis]# extern/redis-2.8.21/src/redis-cli -p 6390 CONFIG GET maxmemory
1) "maxmemory"
2) "100"
[root@h102 codis]#

进行自动均衡

[root@h102 codis]# bin/codis-config slot rebalance 
{
  "msg": "OK",
  "ret": 0
}
[root@h102 codis]# 

codis_autorebalance.png


codis dashboard重启问题

codis dashboard由于各种原因可能要重启

比如重读配置文件(将zk端口从2180改为2181),此时dashboard会报错

2015/12/04 22:38:08 Failed to connect to 192.168.100.101:2180: dial tcp 192.168.100.101:2180: i/o timeout
2015/12/04 22:38:09 Failed to connect to 192.168.100.101:2180: dial tcp 192.168.100.101:2180: i/o timeout
2015/12/04 22:38:09 Failed to connect to 192.168.100.101:2180: dial tcp 192.168.100.101:2180: i/o timeout
2015/12/04 22:38:09 Failed to connect to 192.168.100.101:2180: dial tcp 192.168.100.101:2180: i/o timeout
2015/12/04 22:38:09 Failed to connect to 192.168.100.101:2180: dial tcp 192.168.100.101:2180: i/o timeout

另开一个窗口,手动将dashboard进程杀死

[root@h102 codis]# kill  36106 
[root@h102 codis]# ps faux | grep conf
root      1799  0.0  0.1 189408  3784 ?        Ss   09:44   0:00 cupsd -C /etc/cups/cupsd.conf
root     36106  2.8  4.8 310552 93192 pts/0    Sl+  21:17   2:16  |       \_ bin/codis-config dashboard
root     36715  0.5  1.1 320592 22764 pts/1    Sl+  22:37   0:00  |       \_ bin/codis-proxy -c config.ini -L /tmp/proxy.log --cpu=2 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000
root     36728  0.0  0.0 103256   828 pts/2    S+   22:37   0:00          \_ grep conf
[root@h102 codis]# kill -9 36106
[root@h102 codis]# ps faux | grep conf
root      1799  0.0  0.1 189408  3784 ?        Ss   09:44   0:00 cupsd -C /etc/cups/cupsd.conf
root     36715  0.4  1.1 320592 22772 pts/1    Sl+  22:37   0:00  |       \_ bin/codis-proxy -c config.ini -L /tmp/proxy.log --cpu=2 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000
root     36730  0.0  0.0 103256   828 pts/2    S+   22:38   0:00          \_ grep conf
[root@h102 codis]# 

此时,容易导致dashboard 起不来,报错如下

[root@h102 codis]# bin/codis-config dashboard
2015/12/04 22:38:14 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2015/12/04 22:38:14 dashboard.go:234: [PANIC] create zk node failed
[error]: dashboard already exists: {"addr": "192.168.100.102:18087", "pid": 36106}
[stack]: 
    3   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/dashboard.go:234
            main.runDashboard
    2   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/dashboard.go:54
            main.cmdDashboard
    1   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/main.go:85
            main.runCommand
    0   /root/go_home/src/github.com/wandoulabs/codis/cmd/cconfig/main.go:152
            main.main
        ... ...
[root@h102 codis]# 

但是直接使用ps查看当前进程,又发现系统中并没有残存任何dashboard进程,端口也并没有被占用

[root@h102 codis]# netstat -an | grep 18087
[root@h102 codis]#

这种情况是由于zk中留存上一次dashboard信息导致的

解决办法是,进入zk中手动删除

[zk: localhost:2181(CONNECTED) 8] ls /zk/codis/db_test/dashboard
[]
[zk: localhost:2181(CONNECTED) 9] get  /zk/codis/db_test/dashboard
{"addr": "192.168.100.102:18087", "pid": 36106}
cZxid = 0x143a
ctime = Fri Dec 04 21:17:34 CST 2015
mZxid = 0x143a
mtime = Fri Dec 04 21:17:34 CST 2015
pZxid = 0x143a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 47
numChildren = 0
[zk: localhost:2181(CONNECTED) 10] rmr /zk/codis/db_test/dashboard
[zk: localhost:2181(CONNECTED) 11]

之后就可以正常启动了

[root@h102 codis]# bin/codis-config  dashboard 
2015/12/04 22:44:50 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2015/12/04 22:44:50 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "192.168.100.102:18087", "pid": 36799}
2015/12/04 22:44:50 dashboard.go:144: [WARN] ********** Attention **********
2015/12/04 22:44:50 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2015/12/04 22:44:50 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually
2015/12/04 22:44:50 dashboard.go:147: [WARN] *******************************
...
...
...  

原文地址 http://soft.dog/2015/12/05/codis/

类似博客

评论