Download as PDF, PPTX
Uploaded byPooyan Jamshidi
Configuration Optimization Tool
The document describes a configuration optimization tool that aims to automatically optimize the configuration of big data technologies. It does this by running experiments on data intensive applications, measuring performance under different configurations, and using this data to recommend optimal configurations. The tool implements two approaches for optimization - Bayesian optimization and transfer learning. It consists of several components, including an experimental suite to run tests, an optimization module, interfaces to various big data technologies, and a performance repository to store results. The goal is to help users like SMEs reduce the time and cost of testing and configuring big data applications between releases.
More Related Content
Configuration Optimization Tool
- 1.
- 2. Big Data Technologies Cloud (Priv/Pub) ` DICE Framework 2©DICE DICE IDE Profile Plugins Sim Ver Opt DPIM DTSM DDSM TOSCAMethodology DeployConfig Test M o n Anomaly Trace Iter. Enh. Data Intensive Application (DIA) Cont.Int. Fault Inj. WP4 WP3 WP2 WP5 WP1 WP6 - Demonstrators
- 3. Configuration Optimization Tool The problem: o Big Data technologies have 100s oftuneable parameters o A knowledge gap is faced by SMEs in configuring these technologies What does the tool do? o Automatically runs experiments on DIAs o Returns recommended configuration parameters for Big Data technologies 3©DICE Innovation: o Automate DIA configuration across release cycles o Prior art focuses on manual configuration Impact & stakeholders: o Reduce cost and time of testing between releases o Support operators of DIAs
- 4. CO Tool Architecture 4©DICE Configuration Optimisation Tool performance repository Monitoring Deployment Service DataPreparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester experiment time polling interval previous versions configuration parameters GP model Kafka Vn V1 V2 System Under Test historical data Workload Generator Technology Interface Storm Cassandra Spark Two impementations of CO: BO4CO, TL4CO
- 5. Tool Input (Parameters and Options) 5©DICE 1- Information about the experiment:budget, config file, duration of each experiment 2- Information about the configuration parameters and their options that testers determine
- 6. CO Tool Architecture 6©DICE Configuration Optimisation Tool performance repository Monitoring Deployment Service DataPreparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester experiment time polling interval previous versions configuration parameters GP model Kafka Vn V1 V2 System Under Test historical data Workload Generator Technology Interface Storm Cassandra Spark
- 7. Optimization Component (Matlab) 7©DICE - This componentselect the next configuration to experiment considering the current measurements, - This continues until optimum configuration located or experimental budget finished. The optimization overhead is negligable comparing with measurements This componly relies on rayality free MCR component
- 8. CO Tool Architecture 8©DICE Configuration Optimisation Tool performance repository Monitoring Deployment Service DataPreparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester experiment time polling interval previous versions configuration parameters GP model Kafka Vn V1 V2 System Under Test historical data Workload Generator Technology Interface Storm Cassandra Spark
- 9.
- 10. CO Tool Architecture 10©DICE Configuration Optimisation Tool performance repository Monitoring Deployment Service DataPreparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester experiment time polling interval previous versions configuration parameters GP model Kafka Vn V1 V2 System Under Test historical data Workload Generator Technology Interface Storm Cassandra Spark
- 11. Performance Repository 11©DICE spouts max_spout sortersemit_freq chunk_size message_size throughput latency 1 10 1 1 1.00E+05 1000 22657 3.9833 1 10 1 1 1.00E+05 10000 3596.3 18.415 1 10 1 1 1.00E+05 1.00E+05 112.56 217.63 1 10 1 1 1.00E+06 1000 12273 5.1952 1 10 1 1 1.00E+06 10000 1174.9 24.247 1 10 1 1 1.00E+06 1.00E+05 111.88 205.49 1 10 1 1 2.00E+06 1000 12024 5.2935 1 10 1 1 2.00E+06 10000 1151.3 25.039 1 10 1 1 2.00E+06 1.00E+05 94.294 220.62 1 10 1 1 1.00E+07 1000 11552 6.2867 1 10 1 1 1.00E+07 10000 1228.1 24.975 1 10 1 1 1.00E+07 1.00E+05 102.29 236.19 1 10 1 10 1.00E+05 1000 25978 3.4782 1 10 1 10 1.00E+05 10000 10112 9.2847 1 10 1 10 1.00E+05 1.00E+05 1023.8 83.236 1 10 1 10 1.00E+06 1000 24147 3.6594 1 10 1 10 1.00E+06 10000 8400.2 11.804 1 10 1 10 1.00E+06 1.00E+05 1197.4 73.786 1 10 1 10 2.00E+06 1000 22858 3.7151 1 10 1 10 2.00E+06 10000 7141.3 10.755 1 10 1 10 2.00E+06 1.00E+05 1095.1 78.624 1 10 1 10 1.00E+07 1000 22693 4.3637 1 10 1 10 1.00E+07 10000 6281.5 14.308 1 10 1 10 1.00E+07 1.00E+05 951.27 71.492 1 10 1 60 1.00E+05 1000 25862 3.8521 1 10 1 60 1.00E+05 10000 10859 8.6452 1 10 1 60 1.00E+05 1.00E+05 1128.8 79.862 1 10 1 60 1.00E+06 1000 23553 3.9048 1 10 1 60 1.00E+06 10000 9734.3 9.345 1 10 1 60 1.00E+06 1.00E+05 982 66.852 1 10 1 60 2.00E+06 1000 25408 3.5738 1 10 1 60 2.00E+06 10000 7993.9 9.2784 Configuration Metrics Measured The performance repository mediates between the optimization component and experimental suite
- 12.
- 13. Technology Support 13©DICE Configuration Optimisation Tool performance repository Monitoring Deployment Service DataPreparation configuration parameters values configuration parameters values Experimental Suite Testbed Doc Data Broker Tester experiment time polling interval previous versions configuration parameters GP model Kafka Vn V1 V2 System Under Test historical data Workload Generator Technology Interface Storm Cassandra Spark Technologies: Storm, Spark, Cassandra