[B! Spark SQL] wlbhiroã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯

wlbhiro id:wlbhiro

Spark SQLã«é–¢ã™ã‚‹wlbhiroã®ãƒ–ãƒƒã‚¯ãƒžãƒ¼ã‚¯ (6)

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

${{author_name}}$
{{author_name}}{{created}}
{{ #comment }}{{ comment }}{{ /comment }}
- {{ label }}

${{author_name}}$

{{{comment_expanded}}}

{{label}}

{{#is_bookmark}}ãƒªã‚¹ãƒˆ{{/is_bookmark}}{{^is_bookmark}}ãƒªãƒ³ã‚¯{{/is_bookmark}}

Apache Spark 2.0æ£å¼ç‰ˆãŒãƒªãƒªãƒ¼ã‚¹ã€‚ANSI SQLæ¨™æº–ã‚µãƒãƒ¼ãƒˆã€10å€ä»¥ä¸Šã®é«˜é€ŸåŒ–ãªã©
Apache Spark 2.0æ£å¼ç‰ˆãŒãƒªãƒªãƒ¼ã‚¹ã€‚ANSI SQLæ¨™æº–ã‚µãƒãƒ¼ãƒˆã€10å€ä»¥ä¸Šã®é«˜é€ŸåŒ–ãªã© åˆ†æ•£å‡¦ç†ãƒ•ãƒ¬ãƒ¼ãƒ ãƒ¯ãƒ¼ã‚¯ã®ã€ŒApache Spark 2.0ã€æ£å¼ç‰ˆã®ãƒªãƒªãƒ¼ã‚¹ãŒã€é–‹ç™ºå…ƒã®Databricksã‹ã‚‰ç™ºè¡¨ã•ã‚Œã¾ã—ãŸã€‚ã“ã‚Œã¾ã§Apache Sparkã¯ãƒãƒ¼ã‚¸ãƒ§ãƒ³1.xï¼ˆç›´å‰ã®æœ€æ–°ç‰ˆã¯1.6ï¼‰ã§ã—ãŸã®ã§ã€ãƒ¡ã‚¸ãƒ£ãƒ¼ãƒãƒ¼ã‚¸ãƒ§ãƒ³ã‚¢ãƒƒãƒ—ã¨ãªã‚Šã¾ã™ã€‚ Spark 2.0ã§æœ€å¤§ã®æ–°æ©Ÿèƒ½ã¯ã€æ–°ã—ã„SQLãƒ‘ãƒ¼ã‚µãƒ¼ã‚’æŽ¡ç”¨ã—ãŸã“ã¨ã«ã‚ˆã‚‹ANSI SQLï¼ˆSQL 2003ï¼‰ã¸ã®å¯¾å¿œã§ã™ã€‚ãƒ“ãƒƒã‚°ãƒ‡ãƒ¼ã‚¿ã®ãƒ™ãƒ³ãƒãƒžãƒ¼ã‚¯ã®1ã¤ã§ã‚ã‚‹TPC-DSã®99ç¨®é¡žã®ã‚¯ã‚¨ãƒªãŒãã®ã¾ã¾å®Ÿè¡Œå¯èƒ½ã¨èª¬æ˜Žã•ã‚Œã¦ãŠã‚Šã€ãƒ—ãƒã‚°ãƒ©ãƒžãŒæ…£ã‚Œè¦ªã—ã‚“ã ä¸€èˆ¬çš„ãªSQLæ–‡ã¯ã™ã¹ã¦å®Ÿè¡Œå¯èƒ½ã«ãªã‚Šã¾ã™ã€‚ ã¾ãŸã€DataFrameã¨Datasetã¯çµ±åˆã•ã‚ŒãŸAPIã¨ãªã‚Šã¾ã—ãŸã€‚ ã“ã†ã—ãŸAPIã®å¤‰æ›´ã‚„æ”¹å–„ãŒè¡Œã‚ã‚ŒãŸä¸€æ–¹ã§ã€Spark 2.0ã§ã¯ãƒ‘ãƒ•
wlbhiro 2017/03/31
Spark

Spark SQL

ANSI

SQL
ãƒªãƒ³ã‚¯
ã‚«ã‚¹ã‚¿ãƒ DataSourceã‚’ä½œæˆã™ã‚‹æ–¹æ³• - Qiita
Apache Spark Advent Calendar 2015 20æ—¥ç›®ã®è¨˜äº‹ã§ã™ã€‚ï¼ˆå¤§é…åˆ»ã§ã™ã¿ã¾ã›ã‚“ã€‚ã€‚ã€‚ï¼‰ã“ã®è¨˜äº‹ã§ã¯ã€DataFrame APIã‚’é€šã—ã¦ã‚«ã‚¹ã‚¿ãƒ DataSourceã¸æŽ¥ç¶šã™ã‚‹æ–¹æ³•ã‚’è§£èª¬ã—ã¾ã™ã€‚ 0. æ–°ãŸãªDataSourceã‚’æ›¸ãå§‹ã‚ã‚‹å‰ã« æ—¢ã«å®Ÿè£…æ¸ˆã¿ã®DataSourceã§ã¯ãªã„ã‹ç¢ºèªã™ã‚‹ã®ãŒè‰¯ã„ã§ã—ã‚‡ã†ã€‚Sparkçµ„ã¿è¾¼ã¿ã®DataSourceã¨ã—ã¦ä¸‹è¨˜ã®4ã¤ãŒã‚ã‚Šã¾ã™ã€‚ JSON Parquet JDBC/ODBC Hive(ä»Šå›žã®è§£èª¬ã¨ã¯ç•°ãªã‚‹å®Ÿè£…) http://spark.apache.org/docs/latest/sql-programming-guide.html#data-sources ã¾ãŸã€spark-packages.orgã«ã¯Sparkãƒ¦ãƒ¼ã‚¶ãƒ¼ã‚³ãƒŸãƒ¥ãƒ‹ãƒ†ã‚£ã§é–‹ç™ºã•ã‚ŒãŸãƒ‘ãƒƒã‚±ãƒ¼ã‚¸ãŒç™»éŒ²ã•ã‚Œã¦ã„ã¾ã™ã€‚cassandraã‚„redshif
wlbhiro 2017/03/06
Spark

Spark SQL

DataSource
ãƒªãƒ³ã‚¯
Spark sql queries vs dataframe functions
wlbhiro 2017/02/17
ä¸èº«ã¯ã‚„ã£ã±ã‚ŠåŒã˜ã‚‚ã®ã€‚

Spark

Spark SQL

DataFrame
ãƒªãƒ³ã‚¯
Spark RDDs vs DataFrames vs SparkSQL
wlbhiro 2017/02/17
Spark

Spark SQL

DataFrame

RDD

Compare
ãƒªãƒ³ã‚¯
Spark SQL and DataFrames - Spark 4.1.1 Documentation
Spark SQL, DataFrames and Datasets Guide Spark SQL is a Spark module for structured data processing. Unlike the basic Spark RDD API, the interfaces provided by Spark SQL provide Spark with more information about the structure of both the data and the computation being performed. Internally, Spark SQL uses this extra information to perform extra optimizations. There are several ways to interact wit
wlbhiro 2016/07/07
ã“ã‚Œã§Spark-Shell(Scala)ä¸Šã§ä½œæˆã—ãŸStringã«å¯¾ã™ã‚‹ä»»æ„ã®DataFrameã‚’ä½œæˆã§ãã‚‹ã€‚

Spark

Schema

Row
ãƒªãƒ³ã‚¯
Spark SQLãƒ¡ãƒ¢(Hishidama's Apache Spark SQL Memo)
æ¦‚è¦ Spark SQLã¯ã€Sparkï¼ˆRDDï¼‰ã®æ“ä½œã‚’SQLï¼ˆSELECTæ–‡ï¼‰ã§è¡Œã†ç‚ºã®ã‚¯ãƒ©ã‚¹ç¾¤ã€‚ Sparkã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã™ã‚Œã°ãã®ã¾ã¾ä½¿ãˆã‚‹ã€‚ï¼ˆä»–ã«ç‰¹ã«ä½•ã‹ã‚’ã‚¤ãƒ³ã‚¹ãƒˆãƒ¼ãƒ«ã™ã‚‹å¿…è¦ã¯ç„¡ã„ï¼‰ Spark SQLã‚’ä½¿ã†å ´åˆã€SQLContextã¨SchemaRDDã‚¯ãƒ©ã‚¹ã‚’ä½¿ã†ã€‚ import org.apache.spark._ import org.apache.spark.sql._ val sc: SparkContext = ï½ž val sqlContext = new SQLContext(sc) import sqlContext._ SQLContextã‚¤ãƒ³ã‚¹ã‚¿ãƒ³ã‚¹ã¯SparkContextã‚’ä½¿ã£ã¦ç”Ÿæˆã™ã‚‹ã€‚ SQLContextã«ã¯é€šå¸¸ã®RDDã‹ã‚‰SchemaRDDã¸å¤‰æ›ã™ã‚‹æš—é»™å¤‰æ›ãƒ¡ã‚½ãƒƒãƒ‰ãŒå®šç¾©ã•ã‚Œã¦ã„ã‚‹ã®ã§ã€ã‚¤ãƒ³ãƒãƒ¼ãƒˆã—ã¦ãŠãã€‚ SchemaRDDã¯RDD[Row]ã€ã™
wlbhiro 2015/06/24
Spark

Spark SQL
ãƒªãƒ³ã‚¯
1