======================================================================
Performance updates 2011-10-18
======================================================================
These updates refer to revision 483 of plasma.
The questions here:
1. How many db commits can the system do per second?
2. How many db commits are done in a typical map/red run?
3. What follows for the total capacity of the system?
----------------------------------------------------------------------
1. How many db commits can the system do per second?
----------------------------------------------------------------------
In this Plasma version commits can be parallelized so far different
files are modified by the concurrently running commits.
This continues perf111013, test 5. We run it again with eight
namenode workers serving requests, and seven parallel test clients
generating requests (the other parameters are unchanged):
office3 $ time { ./ps_create -n 10000 /dir1 & ./ps_create -n 10000 /dir2 & ./ps_create -n 10000 /dir3 & ./ps_create -n 10000 /dir4 & ./ps_create -n 10000 /dir5 & ./ps_create -n 10000 /dir6 & ./ps_create -n 10000 /dir7 & wait; }
[1] 4975
[2] 4976
[3] 4977
[4] 4978
[5] 4979
[6] 4980
[7] 4981
Done
Done
[1] Done ./ps_create -n 10000 /dir1
[2] Done ./ps_create -n 10000 /dir2
Done
Done
Done
Done
[3] Done ./ps_create -n 10000 /dir3
[4] Done ./ps_create -n 10000 /dir4
[6]- Done ./ps_create -n 10000 /dir6
[7]+ Done ./ps_create -n 10000 /dir7
Done
[5]+ Done ./ps_create -n 10000 /dir5
real 3m13.861s
user 0m39.702s
sys 0m6.744s
Each ps_create client creates 10000 files in the given directory, and
each file creation is immediately committed. We get a total
performance of
70000 files / 193.861s = 362 commits/s
That's over the whole distance. In the log file we can find much
higher peak rates of around 550 commits/s! (By counting the commit
messages for the period of one second.)
----------------------------------------------------------------------
2. How many db commits are done in a typical map/red run?
----------------------------------------------------------------------
This continues perf111010. I repeated the run (with parameters according
to run #4), and counted the number of commits by analyzing the log file.
We got:
--------------------------------------------------------
Job name: mr_cffa64eecebfd154c85ecea909570356
Job ID: mr_cffa64eecebfd154c85ecea909570356
PlasmaFS input dir: /input
PlasmaFS output dir: /output
PlasmaFS work dir: /work
PlasmaFS log dir: /log
Number task nodes: 2
Task server dir: /tmp/mapred
--------------------------------------------------------
Checking...
Planning...
Pre-start hook...
Starting...
[Tue Oct 18 16:12:01 2011] [info] Checking executable versions
...
[Tue Oct 18 17:07:49 2011] [info] Finished job
[Tue Oct 18 17:07:49 2011] [info] Number successful tasks: 437
[Tue Oct 18 17:07:49 2011] [info] Number misallocated tasks: 9
[Tue Oct 18 17:07:49 2011] [info] Number input files: 830
[Tue Oct 18 17:07:49 2011] [info] Number non-local input files: 157
[Tue Oct 18 17:07:49 2011] [info] Avg number of running tasks: 10.098553
[Tue Oct 18 17:07:49 2011] [info] Wallclock runtime: 3348.170546 seconds
Job_status: successful
The important number is now: This run needed 7399 commits. This means
(we have two nodes with four CPU cores each):
Number of commits: 7399
Number of commits/second: 2.21
Number of commits/second and core: 0.28
The number of transactions depends on the amount of RAM. The machines
had 8 G of RAM, and we used a value for buffer_size of 256 M. By
extending the buffers one can reduce the number of transactions
because more data can be read and written in one go. Note that there
is practically no dependency on the size of the data blocks.
Also note that this run created 2195 log files, and according to the
code each log file is written in 2 transactions. This means more than
half of the commits was used for log files. (This will be improved in
a future Plasma version.)
----------------------------------------------------------------------
3. What follows for the total capacity of the system?
----------------------------------------------------------------------
So the total capacity of this system under ideal conditions would be:
- 550 commits/s namenode capacity
- 0.28 commits/s needed for each CPU core
- ==> 550/0.28 = 1964 cores
Or around 500 of the used quad-core machines!
It is very likely that one runs into other bottlenecks first,
especially network bandwidth becomes quickly a scarce resource when a
network is scaled up.
