play with 31z4 storm 1.0.2

22 September 2016

steps

  1. pull images

    1. storm

       $ docker pull 31z4/storm:1.0.2

    2. zookeeper

       $ docker pull zookeeper:3.4

  2. setup a minimal storm cluster

    1. run zookeeper

       $ docker run -d --restart always --name zookeeper zookeeper:3.4

    2. run nimbus daemon and connect with zookeeper

       $ docker run -d --restart always --name nimbus --net container:zookeeper 31z4/storm:1.0.2 storm nimbus

    3. run supervisor

       $ docker run -d --restart always --name supervisor --net container:nimbus --net container:zookeeper 31z4/storm:1.0.2 storm supervisor

  3. run topology

    1. submit a topology to cluster

       # word count demo
 $ docker run -it --net container:nimbus -v $(pwd)/storm-starter-topology-1.0.2.jar:/topology.jar 31z4/storm:1.0.2 storm jar /topology.jar org.apache.storm.starter.WordCountTopology topology

 # my tick demo
 $ docker run -it --rm --name submit --net container:nimbus -v /c/Users/realtime-tick-1.0-SNAPSHOT.jar:/topology.jar 31z4/storm:1.0.2 storm jar /topology.jar com.jd.o2o.realtime.tick.topology.TickTopology local

    2. list

       $ docker run -it --rm --name list --net container:nimbus 31z4/storm:1.0.2 storm list

  4. demo

    1. shell scripts

    2. test code

    3. steps

       # 0. pull images first

 # 1. setup a storm cluster
 $ ./run.setup.sh

 # 2. put your jar file under C:/Users/

 # 3. submit jar file
 $ ./storm.jar.sh

 # 4. list topology
 $ ./storm.list.sh

 # 5. kill topology
 $ ./storm.kill.sh

 # 6. cleanup docker containers
 $ ./run.cleanup.sh

reference

  1. how-to-deploy-dockerfile-and-application-files-to-boot2docker

    1. virtualbox-guest-additions

       The first of the following share names that exists (if any) will be automatically mounted at the location specified:

 Users share at /Users
 /Users share at /Users
 c/Users share at /c/Users
 /c/Users share at /c/Users
 c:/Users share at /c/Users

  2. how-to-get-bash-or-ssh-into-a-running-container-in-background-mode

    1. docker command

       # from docker 1.3 onwards
 $ docker exec -it supervisor bash

    2. demo

       $ cd /apache-storm-1.0.2/logs/workers-artifacts/tick_topology-1-1477531112/6700
 # your topology folder may not be `tick_topology-1-1477531112`

 $ tail -F worker.log
 ...
 2016-10-27 01:44:18.260 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:44:18
 2016-10-27 01:44:18.260 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:44:18
 2016-10-27 01:44:48.217 c.j.o.r.t.b.TickSleepBolt [INFO] tick done at 2016-10-27 01:44:48
 2016-10-27 01:44:48.217 c.j.o.r.t.b.TickSleepBolt [INFO] tick init at 2016-10-27 01:44:48
 2016-10-27 01:44:48.260 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:44:48
 2016-10-27 01:44:48.261 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:44:48
 2016-10-27 01:45:18.261 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:45:18
 2016-10-27 01:45:18.262 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:45:18
 2016-10-27 01:45:48.262 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:45:48
 2016-10-27 01:45:48.262 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:45:48
 2016-10-27 01:46:18.263 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:46:18
 2016-10-27 01:46:18.263 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:46:18
 2016-10-27 01:46:48.217 c.j.o.r.t.b.TickSleepBolt [INFO] tick done at 2016-10-27 01:46:48
 2016-10-27 01:46:48.218 c.j.o.r.t.b.TickSleepBolt [INFO] tick init at 2016-10-27 01:46:48
 2016-10-27 01:46:48.263 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:46:48
 2016-10-27 01:46:48.264 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:46:48
 2016-10-27 01:47:18.264 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:47:18
 2016-10-27 01:47:18.265 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:47:18
 2016-10-27 01:47:48.265 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:47:48
 2016-10-27 01:47:48.266 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:47:48
 2016-10-27 01:48:18.266 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:48:18
 2016-10-27 01:48:18.266 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:48:18
 2016-10-27 01:48:48.218 c.j.o.r.t.b.TickSleepBolt [INFO] tick done at 2016-10-27 01:48:48
 2016-10-27 01:48:48.218 c.j.o.r.t.b.TickSleepBolt [INFO] tick init at 2016-10-27 01:48:48
 2016-10-27 01:48:48.267 c.j.o.r.t.b.TickBolt [INFO] tick done at 2016-10-27 01:48:48
 2016-10-27 01:48:48.269 c.j.o.r.t.b.TickBolt [INFO] tick init at 2016-10-27 01:48:48
 ...

  3. storm-docker

    1. hub.docker.com

    2. github.com

fixed

  1. java.net.UnknownHostException: nimbus

    1. error info

       30259 [main] WARN  o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from nimbus. will retry with a different seed host.
 java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.UnknownHostException: nimbus
     at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:106) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:66) [storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.command.list$_
 main.invoke(list.clj:22) [storm-core-1.0.2.jar:1.0.2]
     at clojure.lang.AFn.applyToHelper(AFn.java:152) [clojure-1.7.0.jar:?]
     at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
     at org.apache.storm.command.list.main(Unknown Source) [storm-core-1.0.2.jar:1.0.2]
 Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.UnknownHostException: nimbus
     at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) ~[storm-core-1.0.2.jar:1.0.2]
     ... 8 more
 Caused by: java.net.UnknownHostException: nimbus
     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImp
 l.java:184) ~[?:1.8.0_92-internal]
     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_92-internal]
     at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_92-internal]
     at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-1.0.2.jar:1.0.2]
     at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) ~[storm-core-1.0.2.jar:1.0.2]
     ... 8 more
 Exception in thread "main" org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["nimbus"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
     at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90)
     at org.apach
 e.storm.command.list$_main.invoke(list.clj:22)
     at clojure.lang.AFn.applyToHelper(AFn.java:152)
     at clojure.lang.AFn.applyTo(AFn.java:144)
     at org.apache.storm.command.list.main(Unknown Source)

    2. not fixed yet on windows and mac

blog comments powered by Disqus