BOSSzz’s Blog

netty-socketio对于多节点连接的尝试

2019-04-11T16:00:00+00:00

最近公司有个项目使用了netty-socketio来做实时聊天。但是在部署过程中发现当部署多台机器的时候连接会自动断开，无法建立连接。无奈只能只启动一台机器。但是只有一台机器会有一定风险，所以尝试一下能否解决多台机器断开连接的问题。

而且多节点目前只支持轮询（此处吐槽一下某某云，如果能像nginx那样支持sticky就不用那么麻烦了）

所以新建了一个简单的项目来测试。

项目地址在github上https://github.com/BOSSzz/socketio-redis-test

开始

java代码

首先引入依赖, 使用了最新版本的netty-socketio

<dependency>
    <groupId>com.corundumstudio.socketio</groupId>
    <artifactId>netty-socketio</artifactId>
    <version>1.7.17</version>
</dependency>

定义一个消息实体类：HelloUid

public class HelloUid {

    private String uid;

    private String message;

    public String getUid() {
        return uid;
    }

    public void setUid(String uid) {
        this.uid = uid;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

用来与浏览器传递数据

然后建立socketio服务器

public class SocketIOMain {

    private final static Logger logger = LoggerFactory.getLogger(SocketIOMain.class);

    public static void main(String[] args) throws InterruptedException {
        Configuration config = new Configuration();
        config.setHostname("localhost");
        config.setPort(Integer.parseInt(args[0]));
        SocketIOServer server = new SocketIOServer(config);
        server.addConnectListener(new ConnectListener() {// 添加客户端连接监听器
            @Override
            public void onConnect(SocketIOClient client) {
                logger.info(client.getRemoteAddress() + " web客户端接入");
                client.sendEvent("helloPush", "hello");
            }
        });
        // 握手请求
        server.addEventListener("helloevent", HelloUid.class, new DataListener<HelloUid>() {
            @Override
            public void onData(final SocketIOClient client, HelloUid data, AckRequest ackRequest) {
                // 握手
                String userid = data.getUid();
                logger.info(Thread.currentThread().getName() + "web读取到的userid：" + userid);


                // send message back to client with ack callback
                // WITH data
                client.sendEvent("hellopush", new AckCallback<String>(String.class) {
                    @Override
                    public void onSuccess(String result) {
                        logger.info("ack from client: " + client.getSessionId() + " data: " + result);
                    }
                }, data);


            }
        });

        server.start();

        Thread.sleep(Integer.MAX_VALUE);

        server.stop();
    }
}

这样我们的一个简易socketio服务器就完成了。

记得在启动的时候在args参数里指定一下端口号

前端代码

在resource文件夹中新建html文件

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <title>Socketio chat</title>
    <script src="./jquery-3.3.1.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="./socket.io/socket.io.js"></script>
    <style>
        body {
            padding: 20px;
        }
        #console {
            height: 400px;
            overflow: auto;
        }
        .username-msg {
            color: orange;
        }
        .connect-msg {
            color: green;
        }
        .disconnect-msg {
            color: red;
        }
        .send-msg {
            color: #888
        }
    </style>
</head>
<body>
<h1>Netty-socketio chat demo</h1>
<br />
<div id="console" class="well"></div>
<form class="well form-inline" onsubmit="return false;">
    <input id="name" class="input-xlarge" type="text" placeholder="用户名称. . . " />
    <input id="msg" class="input-xlarge" type="text" placeholder="发送内容. . . " />
    <button type="button" onClick="sendMessage()" class="btn">Send</button>
    <button type="button" onClick="sendDisconnect()" class="btn">Disconnect</button>
</form>
</body>
<script type="text/javascript">
    var socket = io.connect('http://localhost:8099');
    socket.on('connect',function() {
        output('<span class="connect-msg">Client has connected to the server!</span>');
    });

    socket.on('hellopush', function(data) {
        output('<span class="username-msg">' + data.uid + ' : </span>'
            + data.message);
    });

    socket.on('disconnect',function() {
        output('<span class="disconnect-msg">The client has disconnected! </span>');
    });

    function sendDisconnect() {
        socket.disconnect();
    }

    function sendMessage() {
        var userName = $("#name").val()
        var message = $('#msg').val();
        $('#msg').val('');
        socket.emit('helloevent', {
            uid : userName,
            message : message
        });
    }

    function output(message) {
        var currentTime = "<span class='time' >" + new Date() + "</span>";
        var element = $("<div>" + currentTime + " " + message + "</div>");
        $('#console').prepend(element);
    }
</script>
</html>

记得在resource目录里放上jquery和socketio的js文件

然后在idea里直接通过浏览器打开即可

之后运行服务端再打开页面就能完成建议的通信了

通过nginx模拟多节点轮询

接下来我们使用nginx来模拟多个节点的轮询，安装nginx的过程就不写了

nginx的配置过程参考nginx+sticky安装及负载均衡使用

注意我们不需要使用sticky，所以可以忽略sticky的安装

最终我的配置如下

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    upstream socket_nodes {
        server localhost:10015;
        server localhost:10016;
        #session_sticky;
    }

    server {
        listen       8099;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }

        location /socket.io {
            proxy_redirect off;
            proxy_buffering off;
            proxy_set_header       Host $host:$server_port;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            client_max_body_size    10m;
            client_body_buffer_size 128k;
            proxy_connect_timeout   1000;
            proxy_send_timeout      1000;
            proxy_read_timeout      1000;
            proxy_buffer_size       256k;
            proxy_buffers           128 256k;
            proxy_busy_buffers_size 256k;
            proxy_temp_file_write_size 256k;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_pass      http://socket_nodes;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}
    include servers/*;
}

有几点需要注意

upstream socket_nodes {
        server localhost:10015;
        server localhost:10016;
        #session_sticky;
    }

这里两个地址是服务端模拟多节点启动的两个端口

server {
        listen       8099;
        server_name  localhost;

这里是nginx监听的端口，也就是页面建立socketio连接所填的端口

多服务器启动

之后分别用10015和10016两个端口启动服务器，再打开页面会发现连接一直失败和项目遇到的场景一致

查阅了socketio连接原理和查看了服务端打印的日志后发现是两次xhr请求发送到了两个服务器导致其中一台在尝试用过sessionId获取client信息时获取不到，从而主动断开了连接。

查阅日志发现了这样一段

17:35:50.231 [nioEventLoopGroup-3-12] WARN com.corundumstudio.socketio.transport.WebSocketTransport - Unauthorized client with sessionId: 4933e61f-5df1-4c5e-a223-de22e876b9e0 with ip: /127.0.0.1:58668. Channel closed!

而打印这日志的调用在WebSocketTransport类中，可见如果没有在clientsBox中获取到ClientHead，就会直接关闭连接。

private void connectClient(final Channel channel, final UUID sessionId) {
        ClientHead client = clientsBox.get(sessionId);
        if (client == null) {
            log.warn("Unauthorized client with sessionId: {} with ip: {}. Channel closed!",
                        sessionId, channel.remoteAddress());
            channel.close();
            return;
        }

        client.bindChannel(channel, Transport.WEBSOCKET);

        authorizeHandler.connect(client);

        if (client.getCurrentTransport() == Transport.POLLING) {
            SchedulerKey key = new SchedulerKey(SchedulerKey.Type.UPGRADE_TIMEOUT, sessionId);
            scheduler.schedule(key, new Runnable() {
                @Override
                public void run() {
                    ClientHead clientHead = clientsBox.get(sessionId);
                    if (clientHead != null) {
                        if (log.isDebugEnabled()) {
                            log.debug("client did not complete upgrade - closing transport");
                        }
                        clientHead.onChannelDisconnect();
                    }
                }
            }, configuration.getUpgradeTimeout(), TimeUnit.MILLISECONDS);
        }

        log.debug("сlient {} handshake completed", sessionId);
    }

而ClientsBox代码是这样的

public class ClientsBox {

    private final Map<UUID, ClientHead> uuid2clients = PlatformDependent.newConcurrentHashMap();
    private final Map<Channel, ClientHead> channel2clients = PlatformDependent.newConcurrentHashMap();

    // TODO use storeFactory
    public HandshakeData getHandshakeData(UUID sessionId) {
        ClientHead client = uuid2clients.get(sessionId);
        if (client == null) {
            return null;
        }

        return client.getHandshakeData();
    }

    public void addClient(ClientHead clientHead) {
        uuid2clients.put(clientHead.getSessionId(), clientHead);
    }

    public void removeClient(UUID sessionId) {
        uuid2clients.remove(sessionId);
    }

    public ClientHead get(UUID sessionId) {
        return uuid2clients.get(sessionId);
    }

    public void add(Channel channel, ClientHead clientHead) {
        channel2clients.put(channel, clientHead);
    }

    public void remove(Channel channel) {
        channel2clients.remove(channel);
    }


    public ClientHead get(Channel channel) {
        return channel2clients.get(channel);
    }

}

这个类将uuid和ClientHead的对应关系保存在uuid2clients这个map中，所以另一个服务器肯定是获取不到的。

解决

那么思路就很简单了，我们将uuid2clients这个map保存在redis中不就好了吗。

但是很可惜，ClientHead这个类没有实现序列化接口，所以虽然思路不变，但是序列化保存到redis的过程需要修改一下。

所以我们将ClientHead这个类拆开，一些成员变量其实是socketio server初始化的，可以不需要保存在redis，将跟随uuid变化的变量保存起来

但是很不幸，我发现其中一个成员变量HandshakeData虽然实现了Serializable接口，但是他的成员变量HttpHeaders是没法序列化的, 无奈只好把header转成map保存

我们通过一个类ClientHeadData来保存这些需要存入redis的信息

public class ClientHeadData implements Serializable {

    private static final long serialVersionUID = 238790253873656662L;

    private UUID sessionId;

    private Map<String, List<String>> headers;
    private InetSocketAddress address;
    private Date time;
    private InetSocketAddress local;
    private String url;
    private Map<String, List<String>> urlParams;
    private boolean xdomain;

    private String transport;

    public ClientHeadData(UUID sessionId, Map<String, List<String>> headers, Map<String, List<String>> urlParams, InetSocketAddress address, Date time, InetSocketAddress local, String url, boolean xdomain, String transport) {
        this.sessionId = sessionId;
        this.address = address;
        this.time = time;
        this.local = local;
        this.url = url;
        this.urlParams = urlParams;
        this.headers = new HashMap<>();
        this.headers = headers;
        this.transport = transport;
    }

    public UUID getSessionId() {
        return sessionId;
    }

    public void setSessionId(UUID sessionId) {
        this.sessionId = sessionId;
    }

    public String getTransport() {
        return transport;
    }

    public void setTransport(String transport) {
        this.transport = transport;
    }

    public Map<String, List<String>> getHeaders() {
        return headers;
    }

    public void setHeaders(Map<String, List<String>> headers) {
        this.headers = headers;
    }

    public InetSocketAddress getAddress() {
        return address;
    }

    public void setAddress(InetSocketAddress address) {
        this.address = address;
    }

    public Date getTime() {
        return time;
    }

    public void setTime(Date time) {
        this.time = time;
    }

    public InetSocketAddress getLocal() {
        return local;
    }

    public void setLocal(InetSocketAddress local) {
        this.local = local;
    }

    public String getUrl() {
        return url;
    }

    public void setUrl(String url) {
        this.url = url;
    }

    public Map<String, List<String>> getUrlParams() {
        return urlParams;
    }

    public void setUrlParams(Map<String, List<String>> urlParams) {
        this.urlParams = urlParams;
    }

    public boolean isXdomain() {
        return xdomain;
    }

    public void setXdomain(boolean xdomain) {
        this.xdomain = xdomain;
    }
}

然后我们新建一个类ClientsRedisBox继承ClientsBox, 里面使用jedis来保存和获取数据

public class ClientsRedisBox extends ClientsBox {

    private final static String CLIENT_HEAD = "clientheads";

    private AckManager ackManager;

    private DisconnectableHub disconnectable;

    private StoreFactory storeFactory;

    private CancelableScheduler disconnectScheduler;

    private Configuration configuration;

    private Jedis jedis;

    public ClientsRedisBox() {
        jedis = new Jedis("localhost", 6379);
    }

    public ClientsRedisBox(AckManager ackManager, DisconnectableHub disconnectable, StoreFactory storeFactory, CancelableScheduler disconnectScheduler, Configuration configuration) {
        this.ackManager = ackManager;
        this.disconnectable = disconnectable;
        this.storeFactory = storeFactory;
        this.disconnectScheduler = disconnectScheduler;
        this.configuration = configuration;
        jedis = new Jedis("localhost", 6379);

    }

    public AckManager getAckManager() {
        return ackManager;
    }

    public void setAckManager(AckManager ackManager) {
        this.ackManager = ackManager;
    }

    public DisconnectableHub getDisconnectable() {
        return disconnectable;
    }

    public void setDisconnectable(DisconnectableHub disconnectable) {
        this.disconnectable = disconnectable;
    }

    public StoreFactory getStoreFactory() {
        return storeFactory;
    }

    public void setStoreFactory(StoreFactory storeFactory) {
        this.storeFactory = storeFactory;
    }

    public CancelableScheduler getDisconnectScheduler() {
        return disconnectScheduler;
    }

    public void setDisconnectScheduler(CancelableScheduler disconnectScheduler) {
        this.disconnectScheduler = disconnectScheduler;
    }

    public Configuration getConfiguration() {
        return configuration;
    }

    public void setConfiguration(Configuration configuration) {
        this.configuration = configuration;
    }

    @Override
    public HandshakeData getHandshakeData(UUID sessionId) {
        return super.getHandshakeData(sessionId);
    }

    @Override
    public void addClient(ClientHead clientHead) {
        super.addClient(clientHead);
        Map<String, List<String>> headers = new HashMap<>();
        for (String name : clientHead.getHandshakeData().getHttpHeaders().names()) {
            List<String> values = new ArrayList<>(clientHead.getHandshakeData().getHttpHeaders().getAll(name));
            headers.put(name, values);
        }
        ClientHeadData clientHeadData = new ClientHeadData(clientHead.getSessionId(), headers,
                clientHead.getHandshakeData().getUrlParams(),
                clientHead.getHandshakeData().getAddress(),
                clientHead.getHandshakeData().getTime(),
                clientHead.getHandshakeData().getLocal(),
                clientHead.getHandshakeData().getUrl(),
                clientHead.getHandshakeData().isXdomain(),
                clientHead.getCurrentTransport().getValue());
        try {
            ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
            ObjectOutputStream objectOutputStream = new ObjectOutputStream(byteArrayOutputStream);
            objectOutputStream.writeObject(clientHeadData);
            jedis.hset(CLIENT_HEAD.getBytes(), clientHead.getSessionId().toString().getBytes(), byteArrayOutputStream.toByteArray());
            byteArrayOutputStream.close();
            objectOutputStream.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Override
    public void removeClient(UUID sessionId) {
        super.removeClient(sessionId);
    }

    @Override
    public ClientHead get(UUID sessionId) {
        ClientHead clientHead = super.get(sessionId);
        if (clientHead == null) {
            try {
                if (!jedis.hexists(CLIENT_HEAD.getBytes(), sessionId.toString().getBytes())) {
                    return null;
                }
                byte[] bytes = jedis.hget(CLIENT_HEAD.getBytes(), sessionId.toString().getBytes());
                ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
                ObjectInputStream objectInputStream = new ObjectInputStream(byteArrayInputStream);
                ClientHeadData clientHeadData = (ClientHeadData) objectInputStream.readObject();
                HttpHeaders httpHeaders = new DefaultHttpHeaders();
                clientHeadData.getHeaders().forEach((name, values) -> {
                    httpHeaders.add(name, values);
                });
                HandshakeData handshakeData = new HandshakeData(httpHeaders, clientHeadData.getUrlParams(), clientHeadData.getAddress(),
                        clientHeadData.getLocal(), clientHeadData.getUrl(), clientHeadData.isXdomain());
                clientHead = new ClientHead(sessionId, ackManager, disconnectable, storeFactory, handshakeData,
                        this, Transport.byName(clientHeadData.getTransport()), disconnectScheduler, configuration);
                super.addClient(clientHead);

            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        return clientHead;
    }

    @Override
    public void add(Channel channel, ClientHead clientHead) {
        super.add(channel, clientHead);
    }

    @Override
    public void remove(Channel channel) {
        super.remove(channel);
    }

    @Override
    public ClientHead get(Channel channel) {
        return super.get(channel);
    }
}

在内存中get不到时，会尝试从redis再获取一次

然后我们新建一个继承了SocketIOChannelInitializer的类来替换服务器的初始化过程

@ChannelHandler.Sharable
public class MyPipeLineFactory extends SocketIOChannelInitializer {

    public MyPipeLineFactory() {
        super();
        try {
            Field field = SocketIOChannelInitializer.class.getDeclaredField("clientsBox");
            field.setAccessible(true);
            field.set(this, new ClientsRedisBox());
        } catch (Exception e) {

        }
    }

    @Override
    public void handlerAdded(ChannelHandlerContext ctx) {
        super.handlerAdded(ctx);
    }

    @Override
    public void start(Configuration configuration, NamespacesHub namespacesHub) {
        super.start(configuration, namespacesHub);
        try {
            AckManager ackManager;
            CancelableScheduler disconnectScheduler;
            Field field = SocketIOChannelInitializer.class.getDeclaredField("ackManager");
            field.setAccessible(true);
            ackManager = (AckManager) field.get(this);
            field = SocketIOChannelInitializer.class.getDeclaredField("scheduler");
            field.setAccessible(true);
            disconnectScheduler = (CancelableScheduler) field.get(this);
            field = SocketIOChannelInitializer.class.getDeclaredField("clientsBox");
            field.setAccessible(true);
            ClientsRedisBox clientsRedisBox = (ClientsRedisBox) field.get(this);
            clientsRedisBox.setAckManager(ackManager);
            clientsRedisBox.setConfiguration(configuration);
            clientsRedisBox.setDisconnectable(this);
            clientsRedisBox.setDisconnectScheduler(disconnectScheduler);
            clientsRedisBox.setStoreFactory(configuration.getStoreFactory());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Override
    protected void initChannel(Channel ch) throws Exception {
        super.initChannel(ch);
    }

    @Override
    protected void addSslHandler(ChannelPipeline pipeline) {
        super.addSslHandler(pipeline);
    }

    @Override
    protected void addSocketioHandlers(ChannelPipeline pipeline) {
        super.addSocketioHandlers(pipeline);
    }

    @Override
    public void onDisconnect(ClientHead client) {
        super.onDisconnect(client);
    }

    @Override
    public void stop() {
        super.stop();
    }
}

可以看到我使用了反射的方法来替换成员变量，因为clientsBox是private的

然后我们在SocketIOMain中添加一行代码来使用我们自定义的PipeLineFactory

server.setPipelineFactory(new MyPipeLineFactory());

至此，大功告成！

在启动两个端口，打开页面，就可以看到已经不会自动断开连接了，发送消息也正常，可喜可贺。

其他问题

过程中还遇到了一个序列化问题

java.io.NotSerializableException: io.netty.handler.codec.HeadersUtils$1

这让我非常奇怪，因为这个类根本就没有再需要序列化的过程中。

查阅了资料后发现这是指在HeadersUtils类中的第一个匿名内部类（参考Java 里 Hashmap 序列化的一个坑）

也就是源码的这一段

    public static <K, V> List<String> getAllAsString(Headers<K, V, ?> headers, K name) {
        final List<V> allNames = headers.getAll(name);
        return new AbstractList<String>() {
            public String get(int index) {
                V value = allNames.get(index);
                return value != null ? value.toString() : null;
            }

            public int size() {
                return allNames.size();
            }
        };
    }

可以看到返回了一个AbstractList，就是这个list不能序列化索性这是个list，只要new 一个ArrayList就行了

List<String> values = new ArrayList<>(clientHead.getHandshakeData().getHttpHeaders().getAll(name));

之前我是没有new ArrayList的，所以会序列化失败

另一种思路

写完博客后不久，我发现可以不修改java代码，只需修改前端js代码就可以简单的避免这个问题。

var socket = io.connect('http://localhost:8099', {transports:['websocket']});

没错，只要在建立socket连接的时候，指定transports为websocket，就可以跳过enginio的xhr请求步骤，直接建立websocket连接

比起改动java代码可谓是简单的多

不得不感叹一下自己对前端的知识了解的太少，还需多学习

参考链接

本地搭建kafka和zookeeper

2018-05-11T15:00:00+00:00

因为工作需要借助kafak和spark读取日志进行处理，上一次记录了搭建spark和hdfs的过程：本地搭建hdfs以及spark环境

在这一篇，将记录一下搭建kafka和zookeeper的过程

一、zookeeper搭建

因为对于kafka的启动zookeeper是必须的，所以首先搭建zookeeper

1.准备

从zookeeper官网下载tar包

http://zookeeper.apache.org/releases.html#download

直接下载最新版

这里搭建的是本地模拟集群环境，所以新建三个文件夹

$ mkdir /Users/zhoujy/zookeeper/zookeeper-1
$ mkdir /Users/zhoujy/zookeeper/zookeeper-2
$ mkdir /Users/zhoujy/zookeeper/zookeeper-3

之后分别解压到三个文件夹

2.配置

为每个目录的conf目录下创建zoo.cfg文件

对于zookeeper-1 添加如下内容

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/zhoujy/hadoop/hadoop/tmp/zk1/data
dataLogDir=/Users/zhoujy/hadoop/hadoop/tmp/zk1/log
clientPort=2181
server.1=localhost:2287:3387
server.2=localhost:2288:3388
server.3=localhost:2289:3389

对于zookeeper-2 添加如下内容

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/zhoujy/hadoop/hadoop/tmp/zk2/data
dataLogDir=/Users/zhoujy/hadoop/hadoop/tmp/zk2/log
clientPort=2182
server.1=localhost:2287:3387
server.2=localhost:2288:3388
server.3=localhost:2289:3389

对于zookeeper-3 添加如下内容

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/zhoujy/hadoop/hadoop/tmp/zk3/data
dataLogDir=/Users/zhoujy/hadoop/hadoop/tmp/zk3/log
clientPort=2183
server.1=localhost:2287:3387
server.2=localhost:2288:3388
server.3=localhost:2289:3389

因为是模拟集群，所以端口号互相错开

之后再配置的每个data目录下添加myid文件，内容对应server.x中的x

/Users/zhoujy/hadoop/hadoop/tmp/zk1/data/myid 中的内容为1，对应server.1的1
/Users/zhoujy/hadoop/hadoop/tmp/zk2/data/myid 中的内容为2，对应server.2的2
/Users/zhoujy/hadoop/hadoop/tmp/zk3/data/myid 中的内容为3，对应server.3的3

3.启动

分别运行bin下面的zkServer.sh脚本

$ sh bin/zkServer.sh start

然后可以使用

$ sh bin/zkCli.sh -server localhost:2181

来进行连接测试

成功后应该会得到类似的提示

[zk: localhost:2181(CONNECTED) 0]

二、kafka搭建

搭建完zookeeper就可以搭建kafka了

1.准备

从kafka官网下载tar包

http://kafka.apache.org/downloads

这里使用的是0.11.0.2的版本

下载后解压，进入kafka目录

2.配置

修改config目录下的server.properties配置文件

$ vim server.properties

添加或修改以下内容：

listeners=PLAINTEXT://localhost:9092

zookeeper.connect=localhost:2181

保存

修改config目录下的zookeeper.properties配置文件

$ vim zookeeper.properties

添加以下内容：

dataDir=/Users/zhoujy/kafka_2.11-0.11.0.2/tmp/zookeeper

clientPort=2181

保存

3.启动

通过以下命令启动：

$ sh bin/kafka-server-start.sh config/server.properties

成功启动kafak

4.一些常用命令

(1) 创建一个topic

$ sh kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic-1

(2) 查看所有topic

$ sh kafka-topics.sh --list --zookeeper localhost:2181

(3) 启动生产者客户端

$ sh kafka-console-producer.sh --broker-list localhost:9092 --topic topic-1

可以往该topic生产数据

(4) 查看topic中的日志

$ sh kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-1 --from-beginning

(5) 查看group的状态

$ sh sh kafka-consumer-offset-checker.sh --topic topic-1 --group group-1 --zookeeper localhost:2181

结尾

至此本地搭建zookeeper和kafka完成，可以在本地spark streaming程序中读取kafka中的数据来进行本地调试了

参考资料

本地搭建hdfs以及spark环境

2018-05-09T14:00:00+00:00

因为工作需要搭建本地spark运行环境，同时需要访问hdfs，记录一下搭建过程和遇到的问题

一、开始

1.hdfs准备

首先去官网下载hadoop的tar包 http://hadoop.apache.org/releases.html

我下载的是2.9.0版本的hadoop

(1) 下载到本地之后解压，进入相应的文件夹，修改一些参数：

$ vim etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/home/users/hadoop/hadoop/tmp</value>
        </property>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://localhost:8200</value>
        </property>
</configuration>

这里的端口号最好设置为8200，不然可能在运行本地spark程序时发生Call to localhost/127.0.0.1:8020 failed on connection exception错误

$ vim etc/hadoop/hdfs-site.xml

<configuration>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>/home/users/hadoop/hadoop/data</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/home/users/hadoop/hadoop/name</value>
        </property>
        <property>
                <name>dfs.http.address</name>
                <value>0.0.0.0:8100</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
</configuration>

(2) namenode格式化

$ bin/hdfs namenode -format

成功输出如下:

如果在hosts文件中修改过0.0.0.0 这个地址，可能会有错误，建议去掉这层修改

(3) 开启NameNode和DataNode

$ sbin/start-dfs.sh

在mac系统中可能会遇到如下问题:

原因是mac没有打开远程访问权限，在系统偏好设置->共享中打开远程登录即可

成功运行结果如下：

执行成功后，输入jps查看开启状态

$ jps

可以看到启动了一个datanode和两个namenode

(4) web页面查看hdfs服务情况

http://localhost:8100

8100对应hdfs-site.xml配置文件中的dfs.http.address端口号

至此，hdfs本地环境搭建成功

(5) 一些命令

停止hdfs：

$ sbin/stop-dfs.sh

查看hdfs目录：

$ bin/hadoop fs -ls /

创建文件夹：

$ bin/hadoop fs -mkdir /test

(6) 其他问题

如果发现namenode启动但是datanode没有启动，一般是因为重新format了namenode，解决方法参考：

hadoop伪分布式下无法启动datanode的原因及解决办法

2.spark准备

从官网下载spark的包

http://spark.apache.org/downloads.html

这里下载是2.2版本

解压即可使用

bin和sbin目录下有各种启动的脚本

(1) 命令行启动

$ sh bin/spark-shell.sh

可以进入spark命令行

(2) 本地启动spark集群

$ sbin/start-master.sh

可以启动master，只后再浏览器输入http://localhost:8080/ 可以看到master的地址（因人而异）:

可以看到master地址为spark://zhoujydeMacBook-Pro.local:7077

根据这个地址，启动worker

$ bin/spark-class org.apache.spark.deploy.worker.Worker spark://zhoujydeMacBook-Pro.local:7077

看到成功的信息：成功启动worker后，再看页面，可以看到启动的alive workers 为1

至此spark启动成功，可以在ide中编写spark程序本地运行

参考资料

crontab执行jar包版本不匹配问题

2018-04-26T10:35:00+00:00

在linux环境上使用java -jar 命令运行jar包时遇到这样一个问题

正常运行没报错但将命令写成shell脚本并且添加到crontab定时执行时报错，报错如下：

Exception in thread “main” java.lang.UnsupportedClassVersionError:
com/kingnetdc/streaming/Main : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:648) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:272) at java.net.URLClassLoader.access$000(URLClassLoader.java:68) at java.net.URLClassLoader$1.run(URLClassLoader.java:207) at java.net.URLClassLoader$1.run(URLClassLoader.java:201) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:200) at java.lang.ClassLoader.loadClass(ClassLoader.java:325) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296) at java.lang.ClassLoader.loadClass(ClassLoader.java:270) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406)

一开始以为是本地jdk版本过高导致的，但是使用jdk1.7编译依然是同样的问题

报错原因其实很简单，是因为没有加载正确的配置导致，在shell脚本前加上

source /etc/profile

让配置立即生效即可

第一篇blog

2018-04-24T18:17:00+00:00

尝试搭建自己的github pages

在这里我将不定期更新遇到的问题和解决方法，记录工作心得等

参考博客如何使用Jekyll+GitHub Pages搭建个人博客站点

主题来源NexT