![Performance Gain
4©DICE
0 1 2 3 4 5
average read latency (µs) ×10
4
0
20
40
60
80
100
120
140
160
observations
1000 1200 1400 1600 1800 2000
average read latency (µs)
0
10
20
30
40
50
60
70
observations
1
1
×10
4
(a) cass-20 (b) cass-10
question explicitly in the paper. Instead, here we define
the questions and provide evidences based on the results re-
ported in the paper. Here, we only focus on the research
questions that we believe are most important for software
engineering community.
RQ1: How large is the potential di↵erence and as a result
potential impact of a wrong choice of parameter settings?
We demonstrated for both SPS in Figure 2 (in the paper)
and for NoSQL systems in Figure 6 and also in Figure 7 that
the performance di↵erence between di↵erent configurations
is significant. The error measures in the experimental results
in the paper also demonstrate this large di↵erence. The
performance gain between the worst and best settings are
measured for each datasets in Table 4.
Table 4: Performance gain between best and worst settings.
Dataset Best (ms) Worst(ms) Gain (%)
1 wc(6D) 55209 3.3172 99%
2 sol(6D) 40499 1.2000 100%
3 rs(6D) 34733 1.9000 99%
4 wc(3D) 94553 1.2994 100%
5 wc(5D) 405.5 47.387 88%
6 cass-10 1.955 1.206 38%
7 cass-20 40.011 2.045 95%
Di↵erent parameter settings cause very large vari-
ance in the performance of the system under test.
RQ2: How does a “default” setting compare to the worst
and optimum configuration in terms of performance?
Figure 16 reports the relative performance for the de-
fault setting, in terms of throughput and latency with the
worst and best configuration as well as the one prescribed
by NoSQL experts. As opposed our expectations, a default
configuration is worst that majority of other configurations,
it performs better than only 10% of other settings for read
operations. However, for write operations the default con-
figurations performs better than half of the other settings.
This was expected because Apache Cassandra is designed to
be write e cient. Even the configuration that is prescribed
pert’s prescription is also far from optimal on indi-
vidual problem instances.
RQ3: How is the performance of a configuration optimiza-
tion on a version when it has been tuned on another version?
If one makes tuning on a specific version of the system
under test, then we would expect it would be optimum for
other versions of the system after it has gone under some
changes. The results for both SPS and NoSQL in Figures
2 and 7 show that the optimum setting will be di↵erent for
di↵erent versions. We also observed that parameter settings
that should work well on average can be particularly ine -
cient on new versions compared to the best configuration for
those versions. In other words, there is a very high variance
in the performance of parameter settings. This has been
also observed by another study in search-based software en-
gineering community [1]. One interesting observations that
has not reported previously is that the performance data
have correlations among di↵erent versions.
Tuned parameters can improve upon default values
on average, but they are far from optimal on individ-
ual problem instances.
RQ4: How much can transfer learning improve tuning with
a larger number of observations from other versions?
The larger the training set is, the more accurate the model
predictions will likely be. In the context of configuration
tuning, carrying over a larger observation set will result in
better tuning, but it would be more expensive to carry out
the optimization process as we have shown in Section 4.6.
The results in Figure 18(b) and Figure 19(b) show that al-
though the models become more accurate slightly, the gain
in performance tuning does not pay o↵ the costs.
Although transferring more observations improves
performance and reduce the entropy, the improve-
ment is low.
3. REFERENCES
[1] A. Arcuri and G. Fraser. Parameter tuning or default
values? an empirical investigation in search-based
software engineering. Empirical Software Engineering,
18(3):594–623, 2013.
6 cass-10 1.955 1.206 38%
7 cass-20 40.011 2.045 95%
Di↵erent parameter settings cause very large vari-
ance in the performance of the system under test.
RQ2: How does a “default” setting compare to the worst
and optimum configuration in terms of performance?
Figure 16 reports the relative performance for the de-
fault setting, in terms of throughput and latency with the
worst and best configuration as well as the one prescribed
by NoSQL experts. As opposed our expectations, a default
configuration is worst that majority of other configurations,
it performs better than only 10% of other settings for read
operations. However, for write operations the default con-
figurations performs better than half of the other settings.
This was expected because Apache Cassandra is designed to
be write e cient. Even the configuration that is prescribed
a larger nu
The larg
predictions
tuning, ca
better tun
the optimi
The result
though the
in perform
Althou
perform
ment is
3. REF
[1] A. Arc
values?
softwar
18(3):5
3](https://image.slidesharecdn.com/configurationoptimization-pooyan-160322164026/75/Configuration-Optimization-for-Big-Data-Software-4-2048.jpg)
![How does a “default” setting compare to the worst and
optimum configuration in terms of performance?
10©DICE
0 500 1000 1500
Throughput (ops/sec)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Averagereadlatency(µs)
×104
TL4CO
BO4CO
BO4CO after
20 iterations TL4CO after
20 iterations
TL4CO after
100 iterations
0 500 1000 1500
Throughput (ops/sec)
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Averagewritelatency(µs)
TL4CO
BO4CO
Default configuration
Configuration
recommended
by expert
TL4CO after
100 iterations
BO4CO after
100 iterations
Default configuration
Configuration
recommended
by expert
IGHTS
to eval-
ch ques-
fied the
e define
sults re-
research
software
a result
ings?
e paper)
e 7 that
urations
l results
e. The
ings are
ettings.
(%)
by practitioners and NoSQL experts is far from the optimum
setting TL4CO found after only 100 iterations.
Default parameter settings perform poorly. The ex-
pert’s prescription is also far from optimal on indi-
vidual problem instances.
RQ3: How is the performance of a configuration optimiza-
tion on a version when it has been tuned on another version?
If one makes tuning on a specific version of the system
under test, then we would expect it would be optimum for
other versions of the system after it has gone under some
changes. The results for both SPS and NoSQL in Figures
2 and 7 show that the optimum setting will be di↵erent for
di↵erent versions. We also observed that parameter settings
that should work well on average can be particularly ine -
cient on new versions compared to the best configuration for
those versions. In other words, there is a very high variance
in the performance of parameter settings. This has been
also observed by another study in search-based software en-
gineering community [1]. One interesting observations that
has not reported previously is that the performance data](https://image.slidesharecdn.com/configurationoptimization-pooyan-160322164026/75/Configuration-Optimization-for-Big-Data-Software-10-2048.jpg)
Uploaded byPooyan Jamshidi
Configuration Optimization for Big Data Software
The document discusses configuration optimization for big data software using an approach developed in the DICE project funded by the European Union's Horizon 2020 program. It describes optimizing configurations for Apache Storm and Cassandra to significantly reduce configuration time. Experiments showed large performance variations between configurations and that default settings often performed poorly compared to optimized settings. Tuning on one version did not guarantee good performance on other versions, but transferring more observations from other versions improved performance, though with diminishing returns due to increased optimization costs.
More Related Content
Similar to Configuration Optimization for Big Data Software
Configuration Optimization for Big Data Software
- 1.
- 2. Configuration Optimization (WP5) 2©DICE • T5.1; Lead: IMP • Contributors: XLAB, IEAT •Experimentally produce an optimal configuration • Parameters not assigned by WP3 optimization
- 3.
- 4. Performance Gain 4©DICE 0 1 23 4 5 average read latency (µs) ×10 4 0 20 40 60 80 100 120 140 160 observations 1000 1200 1400 1600 1800 2000 average read latency (µs) 0 10 20 30 40 50 60 70 observations 1 1 ×10 4 (a) cass-20 (b) cass-10 question explicitly in the paper. Instead, here we define the questions and provide evidences based on the results re- ported in the paper. Here, we only focus on the research questions that we believe are most important for software engineering community. RQ1: How large is the potential di↵erence and as a result potential impact of a wrong choice of parameter settings? We demonstrated for both SPS in Figure 2 (in the paper) and for NoSQL systems in Figure 6 and also in Figure 7 that the performance di↵erence between di↵erent configurations is significant. The error measures in the experimental results in the paper also demonstrate this large di↵erence. The performance gain between the worst and best settings are measured for each datasets in Table 4. Table 4: Performance gain between best and worst settings. Dataset Best (ms) Worst(ms) Gain (%) 1 wc(6D) 55209 3.3172 99% 2 sol(6D) 40499 1.2000 100% 3 rs(6D) 34733 1.9000 99% 4 wc(3D) 94553 1.2994 100% 5 wc(5D) 405.5 47.387 88% 6 cass-10 1.955 1.206 38% 7 cass-20 40.011 2.045 95% Di↵erent parameter settings cause very large vari- ance in the performance of the system under test. RQ2: How does a “default” setting compare to the worst and optimum configuration in terms of performance? Figure 16 reports the relative performance for the de- fault setting, in terms of throughput and latency with the worst and best configuration as well as the one prescribed by NoSQL experts. As opposed our expectations, a default configuration is worst that majority of other configurations, it performs better than only 10% of other settings for read operations. However, for write operations the default con- figurations performs better than half of the other settings. This was expected because Apache Cassandra is designed to be write e cient. Even the configuration that is prescribed pert’s prescription is also far from optimal on indi- vidual problem instances. RQ3: How is the performance of a configuration optimiza- tion on a version when it has been tuned on another version? If one makes tuning on a specific version of the system under test, then we would expect it would be optimum for other versions of the system after it has gone under some changes. The results for both SPS and NoSQL in Figures 2 and 7 show that the optimum setting will be di↵erent for di↵erent versions. We also observed that parameter settings that should work well on average can be particularly ine - cient on new versions compared to the best configuration for those versions. In other words, there is a very high variance in the performance of parameter settings. This has been also observed by another study in search-based software en- gineering community [1]. One interesting observations that has not reported previously is that the performance data have correlations among di↵erent versions. Tuned parameters can improve upon default values on average, but they are far from optimal on individ- ual problem instances. RQ4: How much can transfer learning improve tuning with a larger number of observations from other versions? The larger the training set is, the more accurate the model predictions will likely be. In the context of configuration tuning, carrying over a larger observation set will result in better tuning, but it would be more expensive to carry out the optimization process as we have shown in Section 4.6. The results in Figure 18(b) and Figure 19(b) show that al- though the models become more accurate slightly, the gain in performance tuning does not pay o↵ the costs. Although transferring more observations improves performance and reduce the entropy, the improve- ment is low. 3. REFERENCES [1] A. Arcuri and G. Fraser. Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Software Engineering, 18(3):594–623, 2013. 6 cass-10 1.955 1.206 38% 7 cass-20 40.011 2.045 95% Di↵erent parameter settings cause very large vari- ance in the performance of the system under test. RQ2: How does a “default” setting compare to the worst and optimum configuration in terms of performance? Figure 16 reports the relative performance for the de- fault setting, in terms of throughput and latency with the worst and best configuration as well as the one prescribed by NoSQL experts. As opposed our expectations, a default configuration is worst that majority of other configurations, it performs better than only 10% of other settings for read operations. However, for write operations the default con- figurations performs better than half of the other settings. This was expected because Apache Cassandra is designed to be write e cient. Even the configuration that is prescribed a larger nu The larg predictions tuning, ca better tun the optimi The result though the in perform Althou perform ment is 3. REF [1] A. Arc values? softwar 18(3):5 3
- 5. Configuration Optimization features • A traditionally time-costly operation considerably reduced in duration • Configurations get more efficient with each build •Support for Apache Storm and Cassandra • Metrics: latency, throughput 5©DICE 0 5 10 15 20 Number of counters 100 120 140 160 180 200 220 240 Latency(ms) splitters=2 splitters=3 number of counters number of splitters latency(ms) 100 150 1 200 250 2 300 Cubic Interpolation Over Finer Grid 243 684 10125 14166 18
- 6. Overview of our approach 6©DICE -1.5 -1 -0.50 0.5 1 1.5 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Configuration Space Empirical Model 2 4 6 8 10 12 1 2 3 4 5 6 160 140 120 100 80 60 180 Experiment (exhastive) Experiment Experiment 0 20 40 60 80 100 120 140 160 180 200 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Selection Criteria (b) Sequential Design (a) Design of Experiment
- 7. BO4CO Architecture 7©DICE Configuration Optimisation Tool (BO4CO) performance repository Monitoring Deployment Service DataPreparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester
- 8. TL4CO (DevOps compatible) 8©DICE Configuration Optimisation Tool (TL4CO) performance repository Monitoring DeploymentService Data Preparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester experiment time polling interval previous versions configuration parameters GP model Kafka Vn V1 V2 System Under Test historical data Workload Generator Technology Interface Storm Cassandra Spark 0 4 0.5 1 4 ×104 Latency(µs) 3 1.5 3 2 2 2 1 1 -1 4 0 1 2 4 Latency(µs) ×105 3 3 4 3 5 2 2 1 1 1000 4 1500 2000 2500 4 3000 Latency(µs) 3 3500 4000 3 4500 2 2 1 1 1300 4 1350 1400 4 Latency(µs) 3 1450 1500 3 1550 2 2 1 1 (a) cass-20 v1 (b) cass-20 v2 concurrent_reads concurrent_writes concurrent_reads concurrent_writes concurrent_reads concurrent_writes concurrent_reads concurrent_writes
- 9. Experiments (Storm+Cassandra) 9©DICE sults that werenot included in the main text. 1. FURTHER DETAILS ON SETTINGS 1.1 Configuration Parameters The Apache Storm parameters (cf. Table 1): • max spout (topology.max.spout.pending). The maximum number of tuples that can be pending on a spout. • spout wait (topology.sleep.spout.wait.strategy.time.ms). Time in ms the SleepEmptyEmitStrategy should sleep. • netty min wait (storm.messaging.netty.min wait ms). The min time netty waits to get the control back from OS. • spouts, splitters, counters, bolts. Parallelism level. • heap. The size of the worker heap. • bu↵er size (storm.messaging.netty.bu↵er size). The size of the transfer queue. • emit freq (topology.tick.tuple.freq.secs). The frequency at which tick tuples are received. • top level. The length of a linear topology. • message size, chunk size. The size of tuples and chunk of messages sent across PEs respectively. The Apache Cassandra parameters (cf. Table 1): • trickle fsync: when enabled it forces OS to flush the write bu↵er at a regular time. • auto snapshot: enables the system to take snapshots, avoiding to lose data during truncation or table drop. • concurrent reads: number of threads dedicated to the read operations. • concurrent writes: number of threads dedicated to the write operations. • file cache size in mb: the memory used as reading bu↵ers. 1.2 Dataset Table 1: Experimental datasets, note that this is the complete set of datasets that we experimentally collected over the course of 3 month of 6 di↵erent cloud infrastructures. We only have shown part of this in the paper. Dataset Parameters Size Testbed 1 wc(6D) 1-spouts: {1,3}, 2-max spout: {1,2,10,100,1000,10000}, 3-spout wait:{1,2,3,10,100}, 4-splitters:{1,2,3,6}, 5-counters:{1,3,6,12}, 6-netty min wait:{10,100,1000} 2880 C1 2 sol(6D) 1-spouts:{1,3}, 2-max spout:{1,10,100,1000,10000}, 3-top level:{2,3,4,5}, 4-netty min wait:{10,100,1000}, 5-message size: {10,100,1e3,1e4,1e5,1e6}, 6-bolts: {1,2,3,6} 2866 C2 3 rs(6D) 1-spouts:{1,3}, 2-max spout:{10,100,1000,10000}, 3-sorters:{1,2,3,6,9,12,15,18}, 4-emit freq:{1,10,60,120,300}, 5-chunk size:{1e5,1e6,2e6,1e7}, 6-message size:{1e3,1e4,1e5} 3840 C3 4 wc(3D) 1-max spout:{1,10,100,1e3, 1e4,1e5,1e6}, 2-splitters:{1,2,3,4,5,6}, 3-counters:{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18} 756 C4 5 wc+rs 1-max spout:{1,10,100,1e3, 1e4,1e5,1e6}, 2-splitters:{1,2,3,6}, 3-counters:{1,3,6,9,12,15,18} 196 C4 6 wc+sol 1-max spout:{1,10,100,1e3, 1e4,1e5,1e6}, 2-splitters:{1,2,3,6}, 3-counters:{1,3,6,9,12,15,18} 196 C4 7 wc+wc 1-max spout:{1,10,100,1e3, 1e4,1e5,1e6}, 2-splitters:{1,2,3,6}, 3-counters:{1,3,6,9,12,15,18} 196 C4 8 wc(5D) 1-spouts:{1,2,3}, 2-splitters:{1,2,3,6}, 3-counters:{1,2,3,6,9,12}, 4-bu↵er-size:{256k,1m,5m,10m,100m}, 5-heap:{“-Xmx512m”, “-Xmx1024m”, “-Xmx2048m”} 1080 C5 9 wc-c1 1-spout wait:{1,2,3,4,5,6,7,8,9,10,100,1e3,1e1,6e4}, 2-splitters:{1,2,3,4,5,6}, 3-counters:{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18} 1343 C1 10 wc-c3 1-spout wait:{1,2,3,4,5,6,7,8,9,10,100,1e3,1e4,6e4}, 2-splitters:{1,2,3,4,5,6}, 3-counters:{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18} 1512 C3 11 12 cass-10 cass-20 1-trickle fsync:{false,true}, 2-auto snapshot:{false,true}, 3-con. reads:{16,24,32,40}, 4-con. writes:{16,24,32,40}, 5-file cache size in mb:{256,384,512,640}, 6-con. compactors:{1,3,5,7} 1024 C6 - Over 4 months (24/7) of experimental efforts - on 6 different cloud platforms with 4 benchmark systems (3 Storm, 1 NoSQL)
- 10. How does a “default” setting compare to the worst and optimum configuration in terms of performance? 10©DICE 0 500 10001500 Throughput (ops/sec) 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Averagereadlatency(µs) ×104 TL4CO BO4CO BO4CO after 20 iterations TL4CO after 20 iterations TL4CO after 100 iterations 0 500 1000 1500 Throughput (ops/sec) 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Averagewritelatency(µs) TL4CO BO4CO Default configuration Configuration recommended by expert TL4CO after 100 iterations BO4CO after 100 iterations Default configuration Configuration recommended by expert IGHTS to eval- ch ques- fied the e define sults re- research software a result ings? e paper) e 7 that urations l results e. The ings are ettings. (%) by practitioners and NoSQL experts is far from the optimum setting TL4CO found after only 100 iterations. Default parameter settings perform poorly. The ex- pert’s prescription is also far from optimal on indi- vidual problem instances. RQ3: How is the performance of a configuration optimiza- tion on a version when it has been tuned on another version? If one makes tuning on a specific version of the system under test, then we would expect it would be optimum for other versions of the system after it has gone under some changes. The results for both SPS and NoSQL in Figures 2 and 7 show that the optimum setting will be di↵erent for di↵erent versions. We also observed that parameter settings that should work well on average can be particularly ine - cient on new versions compared to the best configuration for those versions. In other words, there is a very high variance in the performance of parameter settings. This has been also observed by another study in search-based software en- gineering community [1]. One interesting observations that has not reported previously is that the performance data
- 11.