storm parallelism

main components

1. nodes - machines

2. workers - jvms

3. executors - threads

4. tasks - bolt/spout instance

topology execution

1. config workers

   1. 1 sentence spout 1 split bolt 1 count bolt 1 report bolt

       Config config = new Config();
       config.setNumWorkers(2);
       builder.setSpout(SENTENCE_SPOUT_ID, spout);
       builder.setBolt(SPLIT_BOLT_ID, splitBolt);
       builder.setBolt(COUNT_BOLT_ID, countBolt)
           .fieldGrouping(SPLIT_BOLT_ID, new Fields("word");
       builder.setBolt(REPORT_BOLT_ID, reportBolt)
           .globalGrouping(COUNT_BOLT_ID);
      

   2. using 1 worker

       +-----------------------------------------------------------------+
       |                              node                               |
       | +-------------------------------------------------------------+ |
       | |                          worker (JVM)                       | |
       | | +------------+ +------------+ +------------+ +------------+ | |
       | | |  executor  | |  executor  | |  executor  | |  executor  | | |
       | | |  (thread)  | |  (thread)  | |  (thread)  | |  (thread)  | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | | |  task  | | | /  task  \ | | /  task  \ | | /  task  \ | | |
       | | | |  spout | | | |  bolt  | | | |  bolt  | | | |  bolt  | | | |
       | | | |        | | | |        | | | |        | | | |        | | | |
       | | | |sentence| | | |sentence| | | |  word  | | | | report | | | |
       | | | |        | | | \  split / | | \  count / | | \        / | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | +------------+ +------------+ +------------+ +------------+ | |
       | +-------------------------------------------------------------+ |
       +-----------------------------------------------------------------+
      

adding workers to a topology

1. config workers

   1. code

       Config config = new Config();
       config.setNumWorkers(2);
      

2. config executors and tasks

   1. code

       builder.setSpout(SENTENCE_SPOUT_ID, spout, 2);
      

   2. using one worker

       +-----------------------------------------------------------------+
       |                              node                               |
       | +-------------------------------------------------------------+ |
       | |                          worker (JVM)                       | |
       | | +------------+ +------------+ +------------+ +------------+ | |
       | | |  executor  | |  executor  | |  executor  | |  executor  | | |
       | | |  (thread)  | |  (thread)  | |  (thread)  | |  (thread)  | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | | |  task  | | | /  task  \ | | /  task  \ | | /  task  \ | | |
       | | | |  spout | |>| |  bolt  | |>| |  bolt  | |>| |  bolt  | | | |
       | | | |        | | | |        | | | |        | | | |        | | | |
       | | | |sentence| | | |sentence| | | |  word  | | | | report | | | |
       | | | |        | | | \  split / | | \  count / | | \        / | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | +------------+ +------------+ +------------+ +------------+ | |
       | | +------------+        ^                                     | |
       | | |  executor  |        |                                     | |
       | | |  (thread)  |        |                                     | |
       | | | +--------+ |        |                                     | |
       | | | |  task  | |        |                                     | |
       | | | |  spout | |--------                                      | |
       | | | |        | |                                              | |
       | | | |sentence| |                                              | |
       | | | |        | |                                              | |
       | | | +--------+ |                                              | |
       | | +------------+                                              | |
       | +-------------------------------------------------------------+ |
       +-----------------------------------------------------------------+
      

3. setup split sentence bolt and word count bolt

   1. split sentence bolt execute as 4 tasks with 2 executors

       builder.setBolt(SPLIT_BOLT_ID, splitBolt, 2)
           .setNumTask(4)
           .shuffleGrouping(SENTENCE_SPOUT_ID);
      

   2. word count bolt execute as 4 tasks each with its own executor thread

       builder.setBolt(COUNT_BOLT_ID, countBolt, 4)
           .fieldGrouping(SPLIT_BOLT_ID, new Fields("word"));
      

   3. using 2 workers

       +-----------------------------------------------------------------+
       |                              node                               |
       | +-------------------------------------------------------------+ |
       | |                          worker (JVM)                       | |
       | | +------------+ +------------+ +------------+ +------------+ | |
       | | |  executor  | |  executor  | |  executor  | |  executor  | | |
       | | |  (thread)  | |  (thread)  | |  (thread)  | |  (thread)  | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | | |  task  | | | /  task  \ | | /  task  \ | | /  task  \ | | |
       | | | |  spout | | | |  bolt  | | | |  bolt  | | | |  bolt  | | | |
       | | | |        | | | |        | | | |        | | | |        | | | |
       | | | |sentence| | | |sentence| | | |  word  | | | | report | | | |
       | | | |        | | | \  split / | | \  count / | | \        / | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | +------------+ |  --------  | +------------+ +------------+ | |
       | |                | /  task  \ | +------------+                | |
       | |                | |  bolt  | | |  executor  |                | |
       | |                | |        | | |  (thread)  |                | |
       | |                | |sentence| | |  --------  |                | |
       | |                | \  split / | | /  task  \ |                | |
       | |                |  --------  | | |  bolt  | |                | |
       | |                +------------+ | |        | |                | |
       | |                               | |  word  | |                | |
       | |                               | \  count / |                | |
       | |                               |  --------  |                | |
       | |                               +------------+                | |
       | +-------------------------------------------------------------+ |
       | +-------------------------------------------------------------+ |
       | |                          worker (JVM)                       | |
       | | +------------+ +------------+ +------------+ +------------+ | |
       | | |  executor  | |  executor  | |  executor  | |  executor  | | |
       | | |  (thread)  | |  (thread)  | |  (thread)  | |  (thread)  | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | | |  task  | | | /  task  \ | | /  task  \ | | /  task  \ | | |
       | | | |  spout | | | |  bolt  | | | |  bolt  | | | |  bolt  | | | |
       | | | |        | | | |        | | | |        | | | |        | | | |
       | | | |sentence| | | |sentence| | | |  word  | | | | report | | | |
       | | | |        | | | \  split / | | \  count / | | \        / | | |
       | | | +--------+ | |  --------  | |  --------  | |  --------  | | |
       | | +------------+ |  --------  | +------------+ +------------+ | |
       | |                | /  task  \ | +------------+                | |
       | |                | |  bolt  | | |  executor  |                | |
       | |                | |        | | |  (thread)  |                | |
       | |                | |sentence| | |  --------  |                | |
       | |                | \  split / | | /  task  \ |                | |
       | |                |  --------  | | |  bolt  | |                | |
       | |                +------------+ | |        | |                | |
       | |                               | |  word  | |                | |
       | |                               | \  count / |                | |
       | |                               |  --------  |                | |
       | |                               +------------+                | |
       | +-------------------------------------------------------------+ |
       +-----------------------------------------------------------------+