100字范文,内容丰富有趣,生活中的好帮手!
100字范文 > Idea配置spark环境及编写scala代码

Idea配置spark环境及编写scala代码

时间:2023-03-15 12:15:41

相关推荐

Idea配置spark环境及编写scala代码

由于之前在往上没有找到比较好的关于IDEA配置spark环境,现在分享一下

1、打开IDEA

create New Project

2、创建项目,如下图所示

3、配置maven文件,即使你本地没有spark、scala、hadoop环境也没有关系,maven会自动下载及加载到maven里面,只需要你指定相应版本就好,指定好maven之后,自动下载过程需要等待一段时间,如下图所示:

4、New scala class编写代码测试,如下图所示:

注:附maven文件,你可以和我的maven文件一样,下次再创建项目工程的时候,还可以配置相同版本的环境maven就不会重新下载啦,它会自动加载已经下载好的环境!

<projectxmlns="/POM/4.0.0"xmlns:xsi="/2001/XMLSchema-instance"xsi:schemaLocation="/POM/4.0.0/maven-v4_0_0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com_bupt_ai</groupId><artifactId>bupt_at</artifactId><version>1.0-SNAPSHOT</version><inceptionYear></inceptionYear><!--声明子项目公用的配置属性--><properties><scala.version>2.11.8</scala.version><hadoop.version>2.6.0</hadoop.version><spark.version>2.0.1</spark.version><hbase.version>1.2.0</hbase.version><kafka.version>0.8.2.2</kafka.version><jedis.version>2.9.0</jedis.version><mysql.version>5.1.6</mysql.version><log4j.version>1.2.17</log4j.version><slf4j.version>1.7.22</slf4j.version><pool.version>2.4.2</pool.version><jackson.version>2.8.6</jackson.version><net.sf.json.version>2.4</net.sf.json.version><joda.version>2.9.9</joda.version><postgresql.version>42.1.4</postgresql.version><javax.mail>1.6.2</javax.mail><qconf.version>1.2.2</qconf.version><xgboost4j.version>0.72</xgboost4j.version><elasticsearch.version>6.0.1</elasticsearch.version><druid.version>1.1.10</druid.version></properties><!--声明并引入子项目共有的依赖--><dependencies><!--引入Spark相关的Jar包--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.11</artifactId><version>${spark.version}</version><!--<scope>provided</scope>--></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_2.11</artifactId><version>${spark.version}</version><!--<scope>provided</scope>--></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_2.11</artifactId><version>${spark.version}</version><!--<scope>provided</scope>--></dependency><dependency><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId><version>${scala.version}</version><!--<scope>provided</scope>--></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive_2.11</artifactId><version>${spark.version}</version><!--<scope>provided</scope>--></dependency><!--/artifact/org.apache.hbase/hbase-client--><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-client</artifactId><version>${hbase.version}</version><!--<scope>provided</scope>--></dependency><!--/artifact/org.apache.hbase/hbase-server--><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-server</artifactId><version>${hbase.version}</version><!--<scope>provided</scope>--></dependency><!--用来连接Kafka的工具类--><dependency><groupId>org.apache.kafka</groupId><artifactId>kafka-clients</artifactId><version>${kafka.version}</version><!--<scope>provided</scope>--></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming-kafka-0-8_2.11</artifactId><version>${spark.version}</version><!--<scope>provided</scope>--></dependency><dependency><groupId>redis.clients</groupId><artifactId>jedis</artifactId><version>${jedis.version}</version></dependency><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId><version>${mysql.version}</version></dependency><!--/artifact/org.postgresql/postgresql--><dependency><groupId>org.postgresql</groupId><artifactId>postgresql</artifactId><version>${postgresql.version}</version></dependency><dependency><groupId>com.sun.mail</groupId><artifactId>javax.mail</artifactId><version>${javax.mail}</version></dependency><!--/artifact/ml.dmlc/xgboost4j--><dependency><groupId>ml.dmlc</groupId><artifactId>xgboost4j-spark</artifactId><version>${xgboost4j.version}</version></dependency><!--/artifact/org.apache.spark/spark-graphx--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-graphx_2.11</artifactId><version>${spark.version}</version></dependency><!--/artifact/org.apache.spark/spark-mllib--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-mllib_2.11</artifactId><version>${spark.version}</version></dependency><!--/artifact/org.scalanlp/breeze-viz--><dependency><groupId>org.scalanlp</groupId><artifactId>breeze-viz_2.10</artifactId><version>0.12</version></dependency><!--/artifact/org.elasticsearch/elasticsearch-spark-20--><dependency><groupId>org.elasticsearch</groupId><artifactId>elasticsearch-spark-20_2.11</artifactId><version>${elasticsearch.version}</version></dependency><!--/artifact/jfree/jfreechart--><dependency><groupId>jfree</groupId><artifactId>jfreechart</artifactId><version>1.0.13</version></dependency><!--/artifact/com.alibaba/druid--><dependency><groupId>com.alibaba</groupId><artifactId>druid</artifactId><version>${druid.version}</version></dependency><dependency><groupId>com.zaxxer</groupId><artifactId>HikariCP</artifactId><version>2.7.8</version></dependency></dependencies><build><plugins><plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>3.2.2</version><executions><execution><goals><goal>compile</goal><goal>testCompile</goal></goals></execution></executions></plugin></plugins><pluginManagement><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-assembly-plugin</artifactId><version>3.0.0</version><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></pluginManagement></build></project>

本内容不代表本网观点和政治立场,如有侵犯你的权益请联系我们处理。
网友评论
网友评论仅供其表达个人看法,并不表明网站立场。