将博客搬至CSDN

2018-04-07T12:00:00+08:00

将博客搬至CSDN

ethereum私有链环境测试环境搭建

2018-04-07T12:00:00+08:00

ethereum私有链环境测试环境搭建

1.新建目录及创世文件

#私有链数据存储目录

mkdir -p /Users/mac/software/blockchain/ethereum/private_chain/PrivateChain

cd /Users/mac/software/blockchain/ethereum/private_chain
touch genesis.json
vi genesis.json

加入下面内容

{
   "coinbase": "0x0000000000000000000000000000000000000000",
   "config": {
          "homesteadBlock": 5
    },
   "difficulty": "0x20000",
   "extraData": "0x",
   "gasLimit": "0x2FEFD8",
   "mixhash": "0x00000000000000000000000000000000000000647572616c65787365646c6578",
   "nonce": "0x0",
   "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
   "timestamp": "0x00",
   "alloc": {
          "dbdbdb2cbd23b783741e8d7fcf51e459b497e4a6":
          {
              "balance":"100000000000000000000000000000"
          }
   }
}

2.启动geth，初始化创世文件

geth --datadir /Users/mac/software/blockchain/ethereum/private_chain/PrivateChain init /Users/mac/software/blockchain/ethereum/private_chain/genesis.json

3.启动eth控制台

geth --datadir  /Users/mac/software/blockchain/ethereum/private_chain/PrivateChain console

4.创建账户，挖矿

personal.newAccount()
Passphrase:
Repeat passphrase:
"0x63c2d7bcff151c9f8821602425c99aeb55827c49"

my = eth.accounts[0]
eth.getBalance(my)

miner.start()

过一段时间后停止挖矿，并查看账户余额

miner.stop()
eth.getBalance(my)

5.转账

personal.newAccount('123')
"0xc32fb24dc8c62ddde9e9389b4d9abb0415faf73b"

eth.getBalance(eth.accounts[1])
other=eth.accounts[1]

首先解锁被转移账号
personal.unlockAccount(my)
Unlock account 0x63c2d7bcff151c9f8821602425c99aeb55827c49
Passphrase:
true

创建交易

eth.sendTransaction({from:my,to:other,value:1})
INFO [04-07|10:43:00] Submitted transaction                    fullhash=0x1c74e91304324b721a906a67ed337cd90850ddcba13bfd00bdda6a86999b8539 recipient=0xc32FB24dC8c62dDDE9e9389B4D9AbB0415faf73b
"0x1c74e91304324b721a906a67ed337cd90850ddcba13bfd00bdda6a86999b8539"


> eth.getBalance(my)
245000000000000000000
> eth.getBalance(other)
0

此时还是0，因为需要挖矿,查看余额

miner.start()
miner.stop()
eth.getBalance(other)
1

rocketmq环境搭建

2017-11-12T12:00:00+08:00

rocketmq环境搭建

1.下载rocketmq

git clone -b develop https://github.com/apache/rocketmq.git
cd rocketmq
mvn -Prelease-all -DskipTests clean install -U
cd distribution/target/apache-rocketmq

2.启动rocketmq

A.启动rocketmq的nameserver

nohup ./bin/mqnamesrv &

B.启动rockermq的broker代理

nohup ./bin/mqbroker -n localhost:9876 &

2.下载rocketmq管理控制台

https://github.com/apache/rocketmq-externals.git

cd rocketmq-externals/rocketmq-console

mvn install -DskipTests  

3.启动rocketmq-console.jar包

cd /Users/mac/software/flaginfo/software/rocketmq/rocketmq-externals/rocketmq-console/target

NAMESRV_ADDR=localhost:9876 java -jar rocketmq-console-ng-1.0.0.jar   

4.访问localhost:8080即可查看管理控制rocketmq

如果8080被占用，可以在启动时候修改端口  

NAMESRV_ADDR=localhost:9876 java -jar rocketmq-console-ng-1.0.0.jar --server.port=9090  

diamond-server环境搭建

2017-11-12T10:37:00+08:00

diamond-server环境搭建

1.下载diamond-server

目前diamond-server的淘宝地址需要通过淘宝svn账号才能登陆，这里是从github上找到的之前版本。

A.   git clone https://github.com/CodeToSurvive1/diamond.git

B.   cd diamond  

C.   git branch -a
        * bbg-dev
        remotes/origin/HEAD -> origin/bbg-dev
        remotes/origin/bbg-dev
        remotes/origin/bbg-dev-github
        remotes/origin/master
D.  可以看到远程分支上总共存在四个分支，本地默认分支是bbg-dev，下面需要切换到master分支，即为淘宝的最初代码

        git checkout -b master origin/master
        意思是切换到远程分支origin/master，并在本地起名为master

E.  可以通过git branch查看是否本地已经切换到master分支

2.新建数据库并创建表结构

create database config_center;

use config_center;

-- auto-generated definition
CREATE TABLE config_info
(
  id           BIGINT(64) UNSIGNED AUTO_INCREMENT
    PRIMARY KEY,
  data_id      VARCHAR(255) DEFAULT ''                NOT NULL,
  group_id     VARCHAR(128) DEFAULT ''                NOT NULL,
  content      LONGTEXT                               NOT NULL,
  md5          VARCHAR(32) DEFAULT ''                 NOT NULL,
  gmt_create   DATETIME DEFAULT '2010-05-05 00:00:00' NOT NULL,
  gmt_modified DATETIME DEFAULT '2010-05-05 00:00:00' NOT NULL,
  CONSTRAINT uk_config_datagroup
  UNIQUE (data_id, group_id)
);

3.修改配置文件打包war包

修改diamond-server目录下的文件/Users/mac/software/github/diamond/diamond-server/src/main/resources/jdbc.properties

db.url=jdbc:mysql://localhost:3306/config_center?characterEncoding=utf8&connectTimeout=1000&autoReconnect=true
db.user=root
db.password=root
db.initialSize=10
db.maxActive=10
db.maxIdle=5
db.maxWait=5
db.poolPreparedStatements=true

修改diamond-server目录下的文件/Users/mac/software/github/diamond/diamond-server/src/main/resources/user.properties

自定义登录用户名及密码

abc=123
root=root

修改/Users/mac/software/github/diamond/pom.xml文件中的mysql驱动版本，否则会在后面6处报错

    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>6.0.6</version>
    </dependency>

编译打war包


mvn clean package -DskipTests

4.下载tomcat并部署diamond-server.war包

下载tomcat并解压
/Users/mac/software/flaginfo/alisoftware/apache-tomcat-8.5.23

部署diamond-server.war包到webapps下
启动tomcat
登录localhost:8080/diamond-server账号root/root即可登录

5.百度网盘分享地址

https://pan.baidu.com/s/1o8HsqDW

6.如果数据库版本太新，可能会在添加配置的时候报错

Request processing failed; nested exception is com.taobao.diamond.server.exception.ConfigServiceException: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [insert into config_info (data_id,group_id,content,md5,gmt_create,gmt_modified) values(?,?,?,?,?,?)]; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1

kafka集群搭建

2017-05-21T09:17:00+08:00

kafka集群搭建

1.下载kafka

http://kafka.apache.org/downloads

2.下载slf4j软件包

https://www.slf4j.org/download.html

3.将两个压缩包全部上传到集群的某台机器上，这里是centos1

4.解压两个软件包，并重新命令kafka

tar -zxvf kafka_2.10-0.8.2.1.tgz

mv kafka_2.10-0.8.2.1 kafka

tar -zxvf slf4j-1.7.25.tar.gz

mv slf4j-1.7.25/slf4j-nop-1.7.25.jar /opt/cdh/kafka/libs/

5.修改kafka中的config目录下的server.properties文件中的zookeeper配置


zookeeper.connect=centos1:2181,centos2:2181,centos3:2181

6.复制压缩包kafka到集群中的centos2和centos3中

scp -r kafka/ centos@centos2:/opt/cdh/

scp -r kafka/ centos@centos3:/opt/cdh/

7.修改centos2和centos3中的kafka目录下的server.properties中的broker.id分别为1和2，保证集群中的id均不同

8.分别启动三台机器中的启动zookeeper

9.分别启动三台机器中的kafka

nohup ./bin/kafka-server-start.sh config/server.properties &

验证集群

1.创建主题

./bin/kafka-topics.sh --zookeeper centos1:2181,centos2:2181,centos3:2181 --topic testTopic --replication-factor 1 --partitions 1 --create

Created topic "testTopic".

2.创建生产者

./bin/kafka-console-producer.sh --broker-list centos1:9092,centos2:9092,centos3:9092 --topic testTopic

[2017-05-21 10:12:41,802] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cdh/kafka/libs/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh/kafka/libs/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

3.新建窗口，创建消费者

 ./bin/kafka-console-consumer.sh --zookeeper centos1:2181,centos2:2181,centos3:2181 --topic testTopic --from-beginning

4.这时候在生产者窗口中输入helloworld，这时候在消费者窗口会看到消息

oozie安装

2017-03-28T19:17:00+08:00

oozie安装

1.解压oozie

tar -zxvf ../software/oozie-4.0.0-cdh5.3.6.tar.gz -C .

cd oozie-4.0.0-cdh5.3.6/

2.关闭hadoop集群，然后配置hadoop配置目录下的core-site.xml文件

vi etc/hadoop/core-site.xml

<!-- OOZIE -->
<property>
  <name>hadoop.proxyuser.mac.hosts</name>
  <value>localhost,127.0.0.1,centos1,centos2,centos3</value>
</property>
<property>
  <name>hadoop.proxyuser.mac.groups</name>
  <value>*</value>
</property>

然后复制该文件到集群中的其他两个机器中centos2和centos3中

scp etc/hadoop/core-site.xml centos2:/opt/cdh/hadoop-2.5.0-cdh5.3.6/etc/hadoop/
core-site.xml                                 100% 1260     1.2KB/s   00:00    

[mac@centos1 hadoop-2.5.0-cdh5.3.6]$ scp etc/hadoop/core-site.xml centos3:/opt/cdh/hadoop-2.5.0-cdh5.3.6/etc/hadoop/
core-site.xml                                 100% 1260     1.2KB/s   00:00    
[mac@centos1 hadoop-2.5.0-cdh5.3.6]$

3.启动hadoop集群
4.解压oozie目录下的hadooplibs的tar包,在oozie-4.0.0-cdh5.3.6/文件夹中包含了hadooplibs文件夹,其中的hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6是hadoop2.x所使用的包

tar -zxvf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz -C .
cd oozie-4.0.0-cdh5.3.6/

[mac@centos1 oozie-4.0.0-cdh5.3.6]$ ll
总用量 4
drwxr-xr-x. 4 mac mac 4096 7月  29 2015 hadooplibs

ll hadooplibs/
总用量 8
drwxr-xr-x. 2 mac mac 4096 7月  29 2015 hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6
drwxr-xr-x. 2 mac mac 4096 7月  29 2015 hadooplib-2.5.0-mr1-cdh5.3.6.oozie-4.0.0-cdh5.3.6

5.在oozie目录下创建libext目录

pwd
/opt/cdh/oozie-4.0.0-cdh5.3.6

mkdir libext
/opt/cdh/oozie-4.0.0-cdh5.3.6

6.将4部解压的所使用的hadoop的版本中的所有的jars包及下载好的extjs包拷贝到libext目录下

cp oozie-4.0.0-cdh5.3.6/hadooplibs/hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6/* ./libext/

cp /opt/software/ext-2.2.zip ./libext/

7.生成war包，用来监控oozie定时程序的web端页面

./bin/oozie-setup.sh  prepare-war
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/activation-1.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/apacheds-i18n-2.0.0-M15.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/apacheds-kerberos-codec-2.0.0-M15.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/api-asn1-api-1.0.0-M20.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/api-util-1.0.0-M20.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/avro-1.7.6-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/aws-java-sdk-1.7.4.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-beanutils-1.7.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-beanutils-core-1.8.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-cli-1.2.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-codec-1.4.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-collections-3.2.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-compress-1.4.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-configuration-1.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-digester-1.8.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-httpclient-3.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-io-2.4.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-lang-2.4.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-logging-1.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-math3-3.1.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/commons-net-3.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/curator-client-2.6.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/curator-framework-2.6.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/curator-recipes-2.5.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/gson-2.2.4.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/guava-11.0.2.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-annotations-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-auth-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-aws-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-client-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-common-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-hdfs-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-mapreduce-client-app-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-mapreduce-client-common-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-mapreduce-client-core-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-yarn-api-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-yarn-client-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-yarn-common-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/hadoop-yarn-server-common-2.5.0-cdh5.3.6.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/httpclient-4.2.5.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/httpcore-4.2.5.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-annotations-2.2.3.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-core-2.2.3.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-core-asl-1.8.8.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-databind-2.2.3.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-jaxrs-1.8.8.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-mapper-asl-1.8.8.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jackson-xc-1.8.8.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jaxb-api-2.2.2.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jersey-client-1.9.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jersey-core-1.9.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jetty-util-6.1.26.cloudera.2.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/jsr305-1.3.9.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/leveldbjni-all-1.8.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/log4j-1.2.16.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/netty-3.6.2.Final.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/paranamer-2.3.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/protobuf-java-2.5.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/servlet-api-2.5.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/slf4j-api-1.7.5.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/slf4j-log4j12-1.7.5.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/snappy-java-1.0.4.1.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/stax-api-1.0-2.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/xmlenc-0.52.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/xz-1.0.jar
INFO: Adding extension: /opt/cdh/oozie-4.0.0-cdh5.3.6/libext/zookeeper-3.4.5-cdh5.3.6.jar


New Oozie WAR file with added 'ExtJS library, JARs' at /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/webapps/oozie.war


INFO: Oozie is ready to be started

8.将oozie跑各种定时程序所使用的jar包拷贝到hdfs上，比如hive，pig，sqoop

./bin/oozie-setup.sh sharelib create  -fs hdfs://centos1:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz

 setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cdh/oozie-4.0.0-cdh5.3.6/libtools/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh/oozie-4.0.0-cdh5.3.6/libtools/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cdh/oozie-4.0.0-cdh5.3.6/libext/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
the destination path for sharelib is: /user/mac/share/lib/lib_20170328202237

9.创建oozie的数据库，初始化表信息

./bin/oozie-setup.sh db create -run
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"

Validate DB Connection
DONE
Check DB schema does not exist
DONE
Check OOZIE_SYS table does not exist
DONE
Create SQL schema
DONE
Create OOZIE_SYS table
DONE

Oozie DB has been created for Oozie version '4.0.0-cdh5.3.6'


The SQL commands have been written to: /tmp/ooziedb-827033505718254178.sql

10.启动oozie进程

./bin/oozied.sh start

Setting OOZIE_HOME:          /opt/cdh/oozie-4.0.0-cdh5.3.6
Setting OOZIE_CONFIG:        /opt/cdh/oozie-4.0.0-cdh5.3.6/conf
Sourcing:                    /opt/cdh/oozie-4.0.0-cdh5.3.6/conf/oozie-env.sh
  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE:   oozie-site.xml
Setting OOZIE_DATA:          /opt/cdh/oozie-4.0.0-cdh5.3.6/data
Setting OOZIE_LOG:           /opt/cdh/oozie-4.0.0-cdh5.3.6/logs
Setting OOZIE_LOG4J_FILE:    oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD:  10
Setting OOZIE_HTTP_HOSTNAME: centos1.com
Setting OOZIE_HTTP_PORT:     11000
Setting OOZIE_ADMIN_PORT:     11001
Setting OOZIE_HTTPS_PORT:     11443
Setting OOZIE_BASE_URL:      http://centos1.com:11000/oozie
Setting CATALINA_BASE:       /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE:     /home/mac/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS:     password
Setting OOZIE_INSTANCE_ID:       centos1.com
Setting CATALINA_OUT:        /opt/cdh/oozie-4.0.0-cdh5.3.6/logs/catalina.out
Setting CATALINA_PID:        /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp/oozie.pid

Using   CATALINA_OPTS:        -Xmx1024m -Dderby.stream.error.file=/opt/cdh/oozie-4.0.0-cdh5.3.6/logs/derby.log
Adding to CATALINA_OPTS:     -Doozie.home.dir=/opt/cdh/oozie-4.0.0-cdh5.3.6 -Doozie.config.dir=/opt/cdh/oozie-4.0.0-cdh5.3.6/conf -Doozie.log.dir=/opt/cdh/oozie-4.0.0-cdh5.3.6/logs -Doozie.data.dir=/opt/cdh/oozie-4.0.0-cdh5.3.6/data -Doozie.instance.id=centos1.com -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=centos1.com -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://centos1.com:11000/oozie -Doozie.https.keystore.file=/home/mac/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=

Using CATALINA_BASE:   /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server
Using CATALINA_HOME:   /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server
Using CATALINA_TMPDIR: /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp
Using JRE_HOME:        /opt/modules/jdk1.7.0_67
Using CLASSPATH:       /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/bin/bootstrap.jar
Using CATALINA_PID:    /opt/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp/oozie.pid

11.进入tomcat监控页面，端口为11000http://centos1:11000/oozie/或者通过命令查看

看到上面页面证明环境搭建成功

验证环境的可用性

1.解压例子程序

tar -zxvf oozie-examples.tar.gz

2.将解压好的example文件上传到hdfs中

../hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put examples/ examples

3.修改配置内容


vi examples/apps/map-reduce/job.properties

内容如下:

nameNode=hdfs://centos1:8020
jobTracker=centos2:8032
queueName=default
examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml
outputDir=map-reduce

3.运行mapreduce的job

./bin/oozie job -oozie http://centos1:11000/oozie -config examples/apps/map-reduce/job.properties -run
job: 0000000-170328202540397-oozie-mac-W

4.报错如下所示

File /user/mac/share/lib does not exist

修改配置文件oozie-site.xml中的属性oozie.service.HadoopAccessorService.hadoop.configurations

<property>
  <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
  <value>*=hadoop-conf</value>
  <description>
      Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
      the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
      used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
      the relevant Hadoop *-site.xml files. If the path is relative is looked within
      the Oozie configuration directory; though the path can be absolute (i.e. to point
      to Hadoop client conf/ directories in the local filesystem.
  </description>
</property>

修改为

<property>
    <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
    <value>*=/opt/cdh/hadoop-2.5.0-cdh5.3.6/etc/hadoop</value>
    <description>
        Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
        the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is
        used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
        the relevant Hadoop *-site.xml files. If the path is relative is looked within
        the Oozie configuration directory; though the path can be absolute (i.e. to point
        to Hadoop client conf/ directories in the local filesystem.
    </description>
</property>

5.重启oozie

./bin/oozied.sh stop
./bin/oozied.sh start

重启后重新提交job

./bin/oozie job -oozie http://centos1:11000/oozie -config examples/apps/map-reduce/job.properties -run
job: 0000000-170328221752755-oozie-mac-W

验证结果是否正确

flume安装

2017-03-28T19:17:00+08:00

flume安装

1.解压flume

tar -zxvf ../software/flume-ng-1.5.0-cdh5.3.6.tar.gz -C .

mv apache-flume-1.5.0-cdh5.3.6-bin/ flume-1.5.0-cdh5.3.6-bin/

2.修改配置文件

cd conf/
cp flume-env.sh.template flume-env.sh

vi flume-env.sh

添加java_home环境变量
export JAVA_HOME=/opt/modules/jdk1.7.0_67

3.创建日志目录

mkdir logs

flume使用

zookeeper集群安装

2017-03-27T22:17:00+08:00

zookeeper集群搭建

1.在一台机器上解压

tar -zxvf /opt/software/zookeeper-3.4.5-cdh5.3.6.tar.gz -C ./
cd zookeeper-3.4.5-cdh5.3.6

2.修改配置文件,并创建数据目录

cp conf/zoo_sample.cfg conf/zoo.cfg

mkdir -p data/zookeeper

vi conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/cdh/zookeeper-3.4.5-cdh5.3.6/data/zookeeper

# the port at which the clients will connect
clientPort=2181
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=centos1:2888:3888
server.2=centos2:2888:3888
server.3=centos3:2888:3888

3.在上面创建的/opt/cdh/zookeeper-3.4.5-cdh5.3.6/data/zookeeper目录下创建myid的文件,并填写内容1

touch myid
[mac@centos1 zookeeper]$ vi myid
[mac@centos1 zookeeper]$ pwd
/opt/cdh/zookeeper-3.4.5-cdh5.3.6/data/zookeeper

4.将配置好的zookeeper复制到另外两台机器上，centos2和centos3上

scp -r zookeeper-3.4.5-cdh5.3.6 centos2:/opt/cdh/
scp -r zookeeper-3.4.5-cdh5.3.6 centos3:/opt/cdh/

5.进入到centos2中的对应data/zookeeper目录下修改myid内容为2，centos3的myid内容为3
6.分别进入三台系统中启动zookeeper

./bin/zkServer.sh start
JMX enabled by default
Using config: /opt/cdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

7.三台都启动成功后才能通过命令查看状态，没启动成功是无法查看状态的

centos1状态

./bin/zkServer.sh status
JMX enabled by default
Using config: /opt/cdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Mode: follower

centos2状态

./bin/zkServer.sh status
JMX enabled by default
Using config: /opt/cdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Mode: leader

centos3状态

./bin/zkServer.sh status
JMX enabled by default
Using config: /opt/cdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Mode: follower

从角色中可以看出centos2目前是领导者状态，该领导者是集群通过选举策略选择后决定的

没启动完成的时候查看状态如下所示

./bin/zkServer.sh status
JMX enabled by default
Using config: /opt/cdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

通过zkdash查看

通过连接访问服务http://localhost:8888

hive DDL操作

2017-03-25T10:52:00+08:00

创建表

create table student(id int ,name string);

create table log(id INT,text string) partitioned by (year string);

log表有id和text列，还有一个year的虚拟列
默认情况下，表被设定为文本内容格式并且是有^A也就是ctrl-a来分割

查看表

 show tables;
+-----------+--+
| tab_name  |
+-----------+--+
| log       |
| student   |
| xx        |
+-----------+--+

show tables '.*';
+-----------+--+
| tab_name  |
+-----------+--+
| log       |
| student   |
| xx        |
+-----------+--+

show tables '*ent';
+-----------+--+
| tab_name  |
+-----------+--+
| student   |
+-----------+--+

 describe log;
+--------------------------+-----------------------+-----------------------+--+
|         col_name         |       data_type       |        comment        |
+--------------------------+-----------------------+-----------------------+--+
| id                       | int                   |                       |
| text                     | string                |                       |
| year                     | string                |                       |
|                          | NULL                  | NULL                  |
| # Partition Information  | NULL                  | NULL                  |
| # col_name               | data_type             | comment               |
|                          | NULL                  | NULL                  |
| year                     | string                |                       |
+--------------------------+-----------------------+-----------------------+--+
8 rows selected (0.144 seconds)

修改和删除表结构

alter table log rename to log_table;

alter table log_table add columns (date string);

alter table log_table add columns(comment string comment 'a comment');

 describe log_table;
+--------------------------+-----------------------+-----------------------+--+
|         col_name         |       data_type       |        comment        |
+--------------------------+-----------------------+-----------------------+--+
| id                       | int                   |                       |
| text                     | string                |                       |
| date                     | string                |                       |
| comment                  | string                | a comment             |
| year                     | string                |                       |
|                          | NULL                  | NULL                  |
| # Partition Information  | NULL                  | NULL                  |
| # col_name               | data_type             | comment               |
|                          | NULL                  | NULL                  |
| year                     | string                |                       |
+--------------------------+-----------------------+-----------------------+--+
10 rows selected (0.153 seconds)


alter table log_table replace columns(comment string comment 'two comment');

 describe log_table;
+--------------------------+-----------------------+-----------------------+--+
|         col_name         |       data_type       |        comment        |
+--------------------------+-----------------------+-----------------------+--+
| comment                  | string                | two comment           |
| year                     | string                |                       |
|                          | NULL                  | NULL                  |
| # Partition Information  | NULL                  | NULL                  |
| # col_name               | data_type             | comment               |
|                          | NULL                  | NULL                  |
| year                     | string                |                       |
+--------------------------+-----------------------+-----------------------+--+
7 rows selected (0.118 seconds)

replace执行之后表的所有列会被替换为新指定的这些列，分区没有被替换 

drop table log_table;
No rows affected (0.717 seconds)

hiveserver2服务端及客户端使用

2017-03-21T10:52:00+08:00

hiveserver2服务端

hiveserver2是一个服务端接口，能够使远程客户端链接并执行查询操作并返回结果，目前的实现版本是基于thrift rpc调用，hiveserver2是hiveserver的改良版本，更好的支持并发及安全认证，能够更好的支持jdbc及odbc等开放的客户端api

配置：

hive.server2.thrift.min.worker.threads – 最小工作线程, 默认是 5.
hive.server2.thrift.max.worker.threads – 最大工作线程, 默认是 500.
hive.server2.thrift.port – 监听的tcp端口, 默认是 10000.
hive.server2.thrift.bind.host – 绑定的ip，默认是localhost.

启动：

$HIVE_HOME/bin/hiveserver2
或者  
$HIVE_HOME/bin/hive --service hiveserver2

Beeline

hiveserver2支持命令行模式beeline，这是一个基于sqlline cli的jdbc客户端。beeline shell可以工作在内嵌模式或者远程模式中。在内嵌模式中，他运行在内嵌的hive中，远程模式下是通过thrift链接独立的hiveserver2进程，生产环境下推荐远程模式，因为他更加安全，并且不需要获取hdfs元数据的控制权限。在远程模式下，hiveserver2仅仅接受有效的thrift调用，即使是http模式下，消息体包含thrift payloads

bin/beeline 

!connect jdbc:hive2://centos1:10000
scan complete in 33ms
Connecting to jdbc:hive2://centos1:10000
Enter username for jdbc:hive2://centos1:10000: 
Enter password for jdbc:hive2://centos1:10000: 
Connected to: Apache Hive (version 0.13.1)
Driver: Hive JDBC (version 0.13.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://centos1:10000> show databases;
+----------------+
| database_name  |
+----------------+
| default        |
+----------------+
1 row selected (1.412 seconds)
0: jdbc:hive2://centos1:10000> 

或者通过-u参数

 ./bin/beeline -u jdbc:hive2://centos1:10000
scan complete in 3ms
Connecting to jdbc:hive2://centos1:10000
Connected to: Apache Hive (version 0.13.1)
Driver: Hive JDBC (version 0.13.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 0.13.1 by Apache Hive
0: jdbc:hive2://centos1:10000> show databases;
+----------------+
| database_name  |
+----------------+
| default        |
+----------------+
1 row selected (0.17 seconds)

Beeline 命令行参数

-u <database url>       jdbc url
-r hive2.1.0之后支持  重新连接上次连接过的url
-n 用户名
-p 密码
-d 驱动类
-e 查询语句
-f 执行文件
...

输出格式：

Beeline中结果展示可以以不同的形式展示，格式通过outputformat选项设置，支持table，vertical，xmlattr，xmlelements，separated-value formats(csv,tsv,csv2,tsv2,dsv)

beeline --outputformat=tsv

执行最后结果为：

 ./bin/beeline --outputformat=tsv
Beeline version 0.13.1 by Apache Hive
beeline> !connect jdbc:hive2://centos1:10000
scan complete in 5ms
Connecting to jdbc:hive2://centos1:10000
Enter username for jdbc:hive2://centos1:10000: 
Enter password for jdbc:hive2://centos1:10000: 
Connected to: Apache Hive (version 0.13.1)
Driver: Hive JDBC (version 0.13.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://centos1:10000> show databases;
'database_name'
'default'
1 row selected (0.196 seconds)
0: jdbc:hive2://centos1:10000> 

hive客户端

2017-03-16T10:52:00+08:00

hive客户端说明

1.运行Hive CLI,过时了（缺乏多用户，安全及很多其他的特性），由hiveserver2的Beeline客户端代替

$HIVE_HOME/bin/hive

2.hiveserver2的Beeline

$HIVE_HOME/bin/hiveserver2

或者$HIVE_HOME/bin/hive --service hiveserver2

Beeline是通过hiveserver2的jdbc url开始，通过hiveserver2服务启动时候的地址+端口，默认情况下是localhost:10000,所以完整地址为jdbc:hive2://localhost:10000

$HIVE_HOME/bin/beeline -u jdbc:hive2://localhost:10000

hive客户端使用

hive cli使用

./bin/hive -help
usage: hive
 -d,--define <key=value>          Variable subsitution to apply to hive
                                  commands. e.g. -d A=B or --define A=B
    --database <databasename>     Specify the database to use
 -e <quoted-query-string>         SQL from command line
 -f <filename>                    SQL from files
 -H,--help                        Print help information
 -h <hostname>                    connecting to Hive Server on remote host
    --hiveconf <property=value>   Use value for given property
    --hivevar <key=value>         Variable subsitution to apply to hive
                                  commands. e.g. --hivevar A=B
 -i <filename>                    Initialization SQL file
 -p <port>                        connecting to Hive Server on port number
 -S,--silent                      Silent mode in interactive shell
 -v,--verbose                     Verbose mode (echo executed SQL to the
                                  console)

比较重要的有：     
hive -e '查询条件'  

$HIVE_HOME/bin/hive -e 'select a.col from tab1 a'      

hive -f '本地文件系统/hdfs文件系统中的sql脚本文件'   

$HIVE_HOME/bin/hive -f /home/my/hive-script.sql
$HIVE_HOME/bin/hive -f hdfs://<namenode>:<port>/hive-script.sql

hive -i 进入交互式命令行之前先进行脚本初始化操作  
$HIVE_HOME/bin/hive -i /home/my/hive-init.sql

hive -S 安静模式下，通过查询导出数据到文件中  
$HIVE_HOME/bin/hive -S -e 'select a.col from tab1 a' > a.txt

hive --hiveconf(-hiveconf)设置配置参数 

$HIVE_HOME/bin/hive -e 'select a.col from tab1 a' --hiveconf hive.exec.scratchdir=/home/my/hive_scratch  --hiveconf mapred.reduce.tasks=32

修改日志等级及输出目标     
$HIVE_HOME/bin/hive --hiveconf hive.root.logger=INFO,console（过时）
$HIVE_HOME/bin/hiveserver2 --hiveconf hive.root.logger=INFO,console

如果通过set命令设置hive.root.logger级别是不会生效的，因为命令启动的时候已经初始化了

hive交互式命令行

quit/exit  退出命令行 
reset   重置所有配置为默认配置
set key=value   设置配置属性，如果拼错key，是不会报错的
set 输出所有用户或者hive配置    
set -v  输出所有的hadoop及hive配置
add FILE[S] <filepath> <filepath>* 
add JAR[S] <filepath> <filepath>* 
add ARCHIVE[S] <filepath> <filepath>* 添加文件，jars或者archives到分布式文件缓存中
list FILE[S] 
list JAR[S] 
list ARCHIVE[S] 显示所有缓存中的文件，jars或者archives
delete FILE[S] <filepath>* 
delete JAR[S] <filepath>* 
delete ARCHIVE[S] <filepath>*删除文件 jars或者archives
!command 在hive命令行中执行shell命令
dfs dfs-command 执行hdfs命令
source <filepath> 在hive命令行中执行脚本文件

hive resources

hive可以在session期间管理额外的资源文件，并且在查询执行期间可以获取这些资源文件，资源可以是文件，jar文件以及archive文件，任何本地有效的文件都可以添加到session中 
一旦资源加入到session中，hive查询可以在map/reduce/transform条件中通过名字进行引用，并且资源能够在整个hadoop集群中使用，hive使用hadoop的分布式文件缓存来存储资源以便整个集群在查询过程中都可用。
FILE资源仅仅添加到分布式文件缓存系统中
JAR文件还会添加到java的classpath中，这在udf使用对象的引用时候是必须的
ARCHIVE文件会自动解压

Beeline

后续总结

配置管理

1.hive默认从/conf/hive-default.xml文件中获取默认值
2.hive可以通过HIVE_CONF_DIR来配置变量
3.可以通过/conf/hive-site.xml文件来覆盖变量
4.log4j的配置放在/conf/hive-log4j.properties文件中
5.hive会默认从hadoop环境变量中继承过来
6.通过set命令或者hiveconf或者HIVE_OPTS来设置

hive元数据

元数据metadata存储在内嵌的derby数据库中，存储位置由hive的配置变量javax.jdo.option.ConnectionURL决定，默认情况下的位置是./metadata_db

使用内置的derby数据库一次最多有一个用户可以使用。当然可以配置derby为server模式，当然也可以配置其他数据库入mysql

hive中的表和分区的所有的元数据信息都可以通过hive metastore获取，metadata使用了JPOX的orm解决方案来实现持久化，因此任何支持该解决方案的数据库都可以支持hive，大多数的商业关系型数据库都支持

有两种hive的配置方式能够实现搭建metastore database和metastore server

metastore database配置分为本地和远程两种方式

metastore database，元数据持久化的位置  
    本地/内嵌元数据 数据库(derby)
        javax.jdo.option.ConnectionURL:jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true 
        javax.jdo.option.ConnectionDriverName:org.apache.derby.jdbc.EmbeddedDriver 
        hive.metastore.warehouse.dir:file://${user.dir}/../build/ql/test/data/warehouse
    远程 元数据 数据库
        javax.jdo.option.ConnectionURL:jdbc:mysql://<host name>/<database name>?createDatabaseIfNotExist=true
        javax.jdo.option.ConnectionDriverName:com.mysql.jdbc.Driver
        javax.jdo.option.ConnectionUserName:<user name>
        javax.jdo.option.ConnectionPassword:<password>

metastore server配置也分为本地和远程两种方式

在本地或内嵌的metastore环境下，metastore server组件被当做是hive 客户端的一个组件，每个hive客户端都会打开一个数据库连接，然后执行sql查询操作。由于使用的是本地存储模式，所以确保执行sql的机器上能够方式数据库。当然也要确连接数据库的驱动已经配置在hive客户端的环境变零中了.

默认情况下是本地模式，不需要进行任何设置 
hive.metastore.uris:not needed because this is local store
hive.metastore.local:true
hive.metastore.warehouse.dir:hive的非外部表的存储位置 

远程模式配置，这种模式下，所有的客户端连接到metastore服务端中，服务端进行元数据存储的查询操作。客户端与服务端通过thrift进行通信，可以通过下面命令启动服务端：

hive --service metastore


当然客户端也要进行相关的配置：

hive.metastore.uris:thrift://centos1:9083
hive.metastore.local:false 
hive.metastore.warehouse.dir:/user/hive/warehouse
配置完成后直接启动./bin/hive后就会链接到server端，然后执行sql相关命令即可 

具体配置可以参考链接：

hive安装，内嵌模式及本地模式

hive安装，远程模式

最后批注hiveserver2和metastore的区别：

HiveServer2和MetaStore本质上都是Thrift Service，虽然可以启动在同一个进程内，但不建议这么做。
建议是拆成不同的服务进程来启动
具体可以看代码，看看 HiveServer2到底启动了些什么： https://www.codatlas.com/github.com/apache/hive/master/service/src/java/org/apache/hive/service/server/HiveServer2.java?line=112
 
一般来讲，我们认为HiveServer2是用来提交查询的，也就是用来访问数据的。
而MetaStore才是用来访问元数据的。
如果你把两者混了，起在同一个进程内，就会产生你的问题类的疑问。
 
CliDriver是SQL本地直接编译，然后访问MetaStore，提交作业，是重客户端。
BeeLine是把SQL提交给HiveServer2，由HiveServer2编译，然后访问MetaStore，提交作业，是轻客户端。
 
具体写业务脚本两种都行，数据量大的话，建议用CliDriver

参考地址： http://wenda.chinahadoop.cn/question/4489

https://community.mapr.com/thread/8519

macaca相关api整理汇总

2017-03-10T10:52:00+08:00

链接地址 https://macacajs.github.io/macaca-wd/

macaca相关api整理汇总

1.方法：acceptAlert()

支持平台: Android iOS

功能：捕捉当前显示的弹出框

2.方法：alertKeys(keys)

支持平台: iOS

功能：给javascript中的prompt发送按键

3.方法：alertText()

支持平台: iOS

功能：获取JavaScript的alert(), confirm(), 或者 prompt()格式的对话框中的值

4.方法：back()

支持平台:Android Web(WebView)

功能：如果可能的话，将通过浏览器历史导航返回上一层

5.方法：clear()

支持平台：Android iOS Web(WebView)

功能：将TEXTAREA 或者 text 的输入框元素的值清空掉

6.方法：click()

支持平台：Android iOS Web(WebView)

功能：点击元素

7.方法：close()

支持平台：Web(WebView)

功能：关闭当前窗口

8.方法：context(contextRef)

支持平台：Android iOS

功能：设置当前上下文

9.方法：contexts()

支持平台：Android iOS

功能：获取可用的上下文列表

10.方法：currentContext()

支持平台：Android iOS

功能：获取当前的上下文

11.方法：dismissAlert()

支持平台：Android iOS

功能：关闭当前显示的弹出框

12.element(using, value)

支持平台：Android iOS Web(WebView)

功能：根据特定策略查找页面的某个目标元素

参数：using要使用的定位策略 value搜索目标

13.elementByClassName(value)

支持平台：Android iOS Web(WebView)

功能：根据类名称查找元素

14.elementByClassNameIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据类名称查找元素，如果不存在返回undefined

15.elementByClassNameOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据类名称查找元素，如果不存在返回null

16.elementByCss(value)

支持平台：Web(WebView)

功能：根据样式定位器查找元素

17.elementByCssIfExists

支持平台：Web(WebView)

功能：根据样式定位器查找元素，如果不存在返回undefined

18.elementByCssOrNull(value)

支持平台：Web(WebView)

功能：根据样式定位器查找元素，如果不存在返回null

19.elementByCssSelector(value)

支持平台：Web(WebView)

功能：根据样式定位器查找元素

20.elementByCssSelectorIfExists(value)

支持平台：Web(WebView)

功能：根据样式定位器查找元素，如果不存在返回undefined

21.elementByCssSelectorOrNull(value)

支持平台：Web(WebView)

功能：根据样式定位器查找元素，如果不存在返回null

22.elementById(value)

支持平台：Android iOS Web(WebView)

功能：根据元素id查找元素

23.elementByIdIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据元素id查找元素，如果不存在返回undefined

24.elementByIdOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据元素id查找元素，如果不存在返回null

25.elementByLinkText(value)

支持平台：Android iOS Web(WebView)

功能：根据链接的文本内容查找元素

26.elementByLinkTextIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据链接的文本内容查找元素，如果不存在返回undefined

27.elementByLinkTextOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据链接的文本内容查找元素，如果不存在返回null

28.elementByName(value)

支持平台：Android iOS Web(WebView)

功能：根据文本内容属性查找内容

29.elementByNameIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据文本内容属性查找内容，如果不存在返回undefined

30.elementByNameOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据文本内容属性查找内容，如果不存在返回null

31.elementByPartialLinkText(value)

支持平台：Android iOS Web(WebView)

功能：根据部分链接内容查找内容

32.elementByPartialLinkTextIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据部分链接内容查找内容，如果不存在返回undefined

33.elementByPartialLinkTextOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据部分链接内容查找内容，如果不存在返回null

34.elementByTagName(value)

支持平台：Android iOS Web(WebView)

功能：根据标签名字获取元素

35.elementByTagNameIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据标签名字获取元素，如果不存在，返回undefined

36.elementByTagNameOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据标签名字获取元素，如果不存在，返回null

37.elementByXPath(value)

支持平台：Android iOS Web(WebView)

功能：根据xpath路径获取元素

38.elementByXPathIfExists(value)

支持平台：Android iOS Web(WebView)

功能：根据xpath路径获取元素，如果不存在，返回undefined

39.elementByXPathOrNull(value)

支持平台：Android iOS Web(WebView)

功能：根据xpath路径获取元素，如果不存在，返回null

40.elementIfExists(using, value)

支持平台：Android iOS Web(WebView)

功能：判断元素是否存在，using定位策略 value查找内容

41.elementOrNull(using, value)

支持平台：Android iOS Web(WebView)

功能：

42.elements(using, value)

支持平台：Android iOS Web(WebView)

功能：根据给定策略查找元素列表

43.elementsByClassName(value)

支持平台： Android iOS Web(WebView)

功能：根据样式名称查询元素数组

44.elementsByCss(value)

支持平台：Web(WebView)

功能：根据样式查询元素数组

45.elementsByCssSelector(value)

支持平台：Web(WebView)

功能：根据样式选择器查找元素数组

46.elementsById(value)

支持平台：Android iOS Web(WebView)

功能：根据元素id查找元素数组

47.elementsByLinkText(value)

支持平台：Android iOS Web(WebView)

功能：根据链接内容查询元素数组

48.elementsByName(value)

支持平台：Android iOS Web(WebView)

功能：根据名字查找元素数组

49.elementsByPartialLinkText(value)

支持平台：Android iOS Web(WebView)

功能：根据部分链接内容查找元素数组

50.elementsByTagName(value)

支持平台：Android iOS Web(WebView)

功能：根据标签名字获取元素数组

51.elementsByXPath(value)

支持平台：Android iOS Web(WebView)

功能：根据xpath获取元素数组

52.execute(code, argsopt)

支持平台：Web(WebView)

功能：在当前选中的frame的上下文中注入JavaScript代码片段，code为脚本代码片段，argsopt为脚本参数(可选)

53.forward()

支持平台：Web(WebView)

功能：如果可能，导航浏览器前进

54.frame(frameRef)

支持平台：Web(WebView)

功能：将焦点转移到页面中的另外一个frame

55.get(url)

支持平台：Web(WebView)

功能：打开url页面

56.getComputedCss(propertyName)

支持平台：Web(WebView)

功能：Query the value of an element’s computed CSS property.

57.getProperty(name)

支持平台：Android iOS Web(WebView)

功能：获取某个元素的属性值
iOS: ‘isVisible’, ‘isAccessible’, ‘isEnabled’, ‘type’, ‘label’, ‘name’, ‘value’
Android: ‘selected’, ‘description’, ‘text’

58.getRect()

支持平台：Android iOS

功能：查询特定元素的像素和边界，返回值为一个包含x，y，height，width的对象

59.getWindowSize(handleopt)

支持平台：Android iOS Web(WebView)

功能：获取特定窗体的大小,参数handleopt为window handle to set size for (optional, default: ‘current’)

60.hasElement(using, value)

支持平台：Android iOS Web(WebView)

功能：检查元素是否存在，using定位策略，value查找内容

61.hasElementByClassName(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在类名称为value的元素

62.hasElementByCss(value)

支持平台：Web(WebView)

功能：判断是否存在样式为value的元素

63.hasElementByCssSelector(value)

支持平台：Web(WebView)

功能：判断是否存在样式定位格式是value的元素

64.hasElementById(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在id为value的元素

65.hasElementByLinkText(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在链接内容为value的元素

66.hasElementByName(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在名称是value的元素

67.hasElementByPartialLinkText(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在链接内容中包含value的元素

68.hasElementByTagName(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在标签名称为value的元素

69.hasElementByXPath(value)

支持平台：Android iOS Web(WebView)

功能：判断是否存在xpath为value的元素

70.init(desired)

支持平台：Android iOS Web(WebView)

功能：初始化驱动程序，创建session

71.isDisplayed()

支持平台：Android Web(WebView)

功能：判断某个元素是否显示存在

72.keys(keys)

支持平台： Android Web(WebView)

功能：往当前焦点所在的元素发送键盘事件

73.maximize(handle)

支持平台：Web(WebView)

功能：handle如果没有最大化，则最大化指定的handle窗体

74.quit()

支持平台：Android iOS Web(WebView)

功能：删除session

75.refresh()

支持平台：Web(WebView)

功能：刷新当前页面

76.saveScreenshot(filepath)

支持平台：Android iOS Web(WebView)

功能：保存当前页面的截图信息到指定的filepath路径下

77.sendKeys(keys)

支持平台：Android iOS Web(WebView)

功能：发送键盘事件到当前激活焦点的元素

78.sessions()

支持平台：Android iOS Web(WebView)

功能：返回当前有效的所有session列表

79.setWindowSize(width, height, handleopt)

支持平台：Web(WebView)

功能：修改指定窗体的大小 handleopt为window handle to set size for (optional, default: ‘current’)

80.sleep(ms)

支持平台： Android iOS Web(WebView)

功能：设置driver的等待时间，单位ms

81.source()

支持平台：Android iOS Web(WebView)

功能：获取当前页面的source文件

82.status()

支持平台：Android iOS Web(WebView)

功能：获取server的当前状态

83.takeScreenshot()

支持平台： Android iOS Web(WebView)

功能：获取当前页面的截图，返回的是base64字节码，需要自己处理保存

84.text()

支持平台：Android iOS Web(WebView)

功能：返回元素上能看到的文本内容

85.title()

支持平台：Web(WebView)

功能：获取当前页面的标题

86.touch(action, objectopt)

支持平台：iOS, Android

功能：在设备上应用触摸事件

action动作，支持tap/doubleTap/press/pinch/rotate/drag

object，可选参数，指定action的参数
比如driver.touch(‘doubleTap’, {x: 100, y: 100});

87.url()

支持平台：Web(WebView)

功能：获取当前页面的url链接地址

88.waitForElement(using, value, asserteropt, timeoutopt, intervalopt)

支持平台： Android iOS Web(WebView)

功能： All the element-related methods above (except which suffixed with OrNull, IfExists) could be prefixed with the “waitFor-“ (need to capitalize the ‘e’, e.g., waitForElementByClassName)

using string The locator strategy to use, omitted when using specific method like waitForElementByClassName. value string The css selector asserter function The asserter function (commonly used asserter function can be found at wd.asserters) (optional) timeout number 1000ms The timeout before find the element (optional) interval number 200ms The interval between each searching (optional)

例子： waitForElementByClassName(‘btn’, 2000, 100) Search for element which class name is ‘btn’ at intervals of 100ms, last for 2000ms.

89.window()

支持平台： Web(WebView)

功能：修改焦点到另外的窗体

90.windowHandle()

支持平台：Web(WebView)

功能：获取当前窗体句柄

91.windowHandles()

支持平台：Web(WebView)

功能：获取当前session中有效的窗体句柄

centos mysql5.6.24安装

2017-03-06T20:52:00+08:00

先决条件

下载centos系统版本mysql程序，包括server端，client端，以及java 驱动程序，下载地址，http://pan.baidu.com/s/1nvvS7PZ

安装步骤

1.首先查看系统中有没有自带的mysql安装程序

rpm -qa | grep mysql
qt-mysql-4.6.2-25.el6.x86_64
mysql-server-5.1.66-2.el6_3.x86_64
mysql-libs-5.1.66-2.el6_3.x86_64
mysql-5.1.66-2.el6_3.x86_64

2.如果有，首先卸载mysql程序

[mac@centos1 ~]$ sudo rpm -e --nodeps qt-mysql-4.6.2-25.el6.x86_64
[sudo] password for mac: 
[mac@centos1 ~]$ sudo rpm -e --nodeps mysql-server-5.1.66-2.el6_3.x86_64
[mac@centos1 ~]$ sudo rpm -e --nodeps mysql-libs-5.1.66-2.el6_3.x86_64
[mac@centos1 ~]$ sudo rpm -e --nodeps mysql-5.1.66-2.el6_3.x86_64
[mac@centos1 ~]$ 

3.查看是否卸载成功

[mac@centos1 ~]$ rpm -qa| grep mysql

4.将mysql 安装程序拷贝到centos系统中的software目录下，并解压切换到software目录下安装

mac:~ mac$ scp ~/Downloads/mysql-libs.zip centos1:/opt/software
mac@centos1's password: 
mysql-libs.zip                                100%   74MB  36.3MB/s   00:02

[mac@centos1 software]$ unzip mysql-libs.zip 
Archive:  mysql-libs.zip
   creating: mysql-libs/
  inflating: mysql-libs/MySQL-client-5.6.24-1.el6.x86_64.rpm  
  inflating: mysql-libs/mysql-connector-java-5.1.27.tar.gz  
  inflating: mysql-libs/MySQL-server-5.6.24-1.el6.x86_64.rpm 

[mac@centos1 mysql-libs]$ ls
MySQL-client-5.6.24-1.el6.x86_64.rpm  MySQL-server-5.6.24-1.el6.x86_64.rpm
mysql-connector-java-5.1.27.tar.gz
[mac@centos1 mysql-libs]$ sudo rpm -ivh *.rpm
[sudo] password for mac: 
Preparing...                ########################################### [100%]
   1:MySQL-server           ########################################### [ 50%]
2017-03-05 11:37:29 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-03-05 11:37:29 0 [Note] /usr/sbin/mysqld (mysqld 5.6.24) starting as process 2195 ...
2017-03-05 11:37:29 2195 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-03-05 11:37:29 2195 [Note] InnoDB: The InnoDB memory heap is disabled
2017-03-05 11:37:29 2195 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-03-05 11:37:29 2195 [Note] InnoDB: Memory barrier is not used
2017-03-05 11:37:29 2195 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-03-05 11:37:29 2195 [Note] InnoDB: Using Linux native AIO
2017-03-05 11:37:29 2195 [Note] InnoDB: Using CPU crc32 instructions
2017-03-05 11:37:29 2195 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-03-05 11:37:29 2195 [Note] InnoDB: Completed initialization of buffer pool
2017-03-05 11:37:29 2195 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2017-03-05 11:37:29 2195 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2017-03-05 11:37:29 2195 [Note] InnoDB: Database physically writes the file full: wait...
2017-03-05 11:37:29 2195 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2017-03-05 11:37:29 2195 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2017-03-05 11:37:30 2195 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2017-03-05 11:37:30 2195 [Warning] InnoDB: New log files created, LSN=45781
2017-03-05 11:37:30 2195 [Note] InnoDB: Doublewrite buffer not found: creating new
2017-03-05 11:37:30 2195 [Note] InnoDB: Doublewrite buffer created
2017-03-05 11:37:30 2195 [Note] InnoDB: 128 rollback segment(s) are active.
2017-03-05 11:37:30 2195 [Warning] InnoDB: Creating foreign key constraint system tables.
2017-03-05 11:37:30 2195 [Note] InnoDB: Foreign key constraint system tables created
2017-03-05 11:37:30 2195 [Note] InnoDB: Creating tablespace and datafile system tables.
2017-03-05 11:37:30 2195 [Note] InnoDB: Tablespace and datafile system tables created.
2017-03-05 11:37:30 2195 [Note] InnoDB: Waiting for purge to start
2017-03-05 11:37:30 2195 [Note] InnoDB: 5.6.24 started; log sequence number 0
A random root password has been set. You will find it in '/root/.mysql_secret'.
2017-03-05 11:37:30 2195 [Note] Binlog end
2017-03-05 11:37:30 2195 [Note] InnoDB: FTS optimize thread exiting.
2017-03-05 11:37:30 2195 [Note] InnoDB: Starting shutdown...
2017-03-05 11:37:32 2195 [Note] InnoDB: Shutdown completed; log sequence number 1625977


2017-03-05 11:37:32 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-03-05 11:37:32 0 [Note] /usr/sbin/mysqld (mysqld 5.6.24) starting as process 2217 ...
2017-03-05 11:37:32 2217 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-03-05 11:37:32 2217 [Note] InnoDB: The InnoDB memory heap is disabled
2017-03-05 11:37:32 2217 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-03-05 11:37:32 2217 [Note] InnoDB: Memory barrier is not used
2017-03-05 11:37:32 2217 [Note] InnoDB: Compressed tables use zlib 1.2.3
2017-03-05 11:37:32 2217 [Note] InnoDB: Using Linux native AIO
2017-03-05 11:37:32 2217 [Note] InnoDB: Using CPU crc32 instructions
2017-03-05 11:37:32 2217 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-03-05 11:37:32 2217 [Note] InnoDB: Completed initialization of buffer pool
2017-03-05 11:37:32 2217 [Note] InnoDB: Highest supported file format is Barracuda.
2017-03-05 11:37:32 2217 [Note] InnoDB: 128 rollback segment(s) are active.
2017-03-05 11:37:32 2217 [Note] InnoDB: Waiting for purge to start
2017-03-05 11:37:32 2217 [Note] InnoDB: 5.6.24 started; log sequence number 1625977
2017-03-05 11:37:32 2217 [Note] Binlog end
2017-03-05 11:37:32 2217 [Note] InnoDB: FTS optimize thread exiting.
2017-03-05 11:37:32 2217 [Note] InnoDB: Starting shutdown...
2017-03-05 11:37:34 2217 [Note] InnoDB: Shutdown completed; log sequence number 1625987




A RANDOM PASSWORD HAS BEEN SET FOR THE MySQL root USER !
You will find that password in '/root/.mysql_secret'.

You must change that password on your first connect,
no other statement but 'SET PASSWORD' will be accepted.
See the manual for the semantics of the 'password expired' flag.

Also, the account for the anonymous user has been removed.

In addition, you can run:

  /usr/bin/mysql_secure_installation

which will also give you the option of removing the test database.
This is strongly recommended for production servers.

See the manual for more instructions.

Please report any problems at http://bugs.mysql.com/

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

New default config file was created as /usr/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

   2:MySQL-client           ########################################### [100%]

5.安装mysql 服务端，安装完成后注意提示信息，密码自动放在/root/.mysql_secret文件中，并且需要在第一次连接的时候通过set password来修改密码

[mac@centos1 mysql-libs]$ su
密码：
[root@centos1 mysql-libs]# cat /root/.mysql_secret
# The random password set for the root user at Sun Mar  5 11:37:30 2017 (local time): qr2xuNRQnrh48eZA

[root@centos1 mysql-libs]# service mysql start
Starting MySQL.                                            [  OK  ]
[root@centos1 mysql-libs]# 

[root@centos1 mysql-libs]# mysql -uroot -pqr2xuNRQnrh48eZA
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.24

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

6.mysql连接成功后通过set password修改密码,然后退出重新登录，查询数据库验证是否修改成功

mysql> set password=password('123456');
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[root@centos1 mysql-libs]# mysql -uroot -p123456
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.24 MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.00 sec)

7.查看user表中的数据，显示了用户名对应的登陆主机设置,然后修改设置为远程ip都可以访问.

mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed


mysql> select host,user,password from user;
+-----------+------+-------------------------------------------+
| host      | user | password                                  |
+-----------+------+-------------------------------------------+
| localhost | root | *6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9 |
| centos1   | root | *E0781EC3BF7F5F4B9DCA25B2D84C259741AEE1E5 |
| 127.0.0.1 | root | *E0781EC3BF7F5F4B9DCA25B2D84C259741AEE1E5 |
| ::1       | root | *E0781EC3BF7F5F4B9DCA25B2D84C259741AEE1E5 |
+-----------+------+-------------------------------------------+
4 rows in set (0.00 sec)

mysql>update user set host='%' where user='root' and host='localhost';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

最后不要忘记重启mysql 

sudo service mysql restart

hadoop分布式环境搭建二(集群搭建)

2017-02-28T22:52:00+08:00

安装准备

1.在/opt目录下创建software及modules文件夹，并修改到为mac用户下

sudo mkdir /opt/software
[sudo] password for mac: 
[mac@centos1 ~]$ sudo mkdir /opt/modules
[mac@centos1 ~]$ sudo chown -R mac:mac /opt/software/
[mac@centos1 ~]$ sudo chown -R mac:mac /opt/modules/
[mac@centos1 ~]$ ls -al /opt/
总用量 20
drwxr-xr-x.  5 root root 4096 2月  28 21:42 .
dr-xr-xr-x. 25 root root 4096 2月  28 21:18 ..
drwxr-xr-x.  2 mac  mac  4096 2月  28 21:42 modules
drwxr-xr-x.  2 root root 4096 2月  22 2013 rh
drwxr-xr-x.  2 mac  mac  4096 2月  28 21:41 software

2.上传软件到/opt/software目录下

首先下载jdk及hadoop软件，下载地址为链接: https://pan.baidu.com/s/1nu8IrbV 密码: g9te

在自己的mac电脑上使用scp命令将两个文件上传到centos1系统的opt/software目录下

mac:~ mac$ scp ~/Downloads/jdk-7u67-linux-x64.tar.gz ~/Downloads/hadoop-2.5.0.tar.gz centos1:/opt/software
mac@centos1's password: 
jdk-7u67-linux-x64.tar.gz                     100%  136MB  37.5MB/s   00:03    
hadoop-2.5.0.tar.gz                           100%  297MB  35.4MB/s   00:08  

进入centos1的/opt/software目录下查看是否上传成功

[mac@centos1 ~]$ ls -al /opt/software/
总用量 443180
drwxr-xr-x. 2 mac  mac       4096 2月  28 21:53 .
drwxr-xr-x. 5 root root      4096 2月  28 21:42 ..
-rwxr-xr-x. 1 mac  mac  311430119 2月  28 21:53 hadoop-2.5.0.tar.gz
-rwxr-xr-x. 1 mac  mac  142376665 2月  28 21:53 jdk-7u67-linux-x64.tar.gz

将jdk复制到centos2和centos3，hadoop软件临时不需要上传到centos2和centos3中，等最后配置文件全部配置好了之后再复制即可

mac:~ mac$ scp ~/Downloads/jdk-7u67-linux-x64.tar.gz  centos2:/opt/software
mac@centos2's password: 
jdk-7u67-linux-x64.tar.gz                     100%  136MB  45.2MB/s   00:03    
mac:~ mac$ scp ~/Downloads/jdk-7u67-linux-x64.tar.gz  centos3:/opt/software
mac@centos3's password: 
jdk-7u67-linux-x64.tar.gz                     100%  136MB  38.2MB/s   00:03    
mac:~ mac$ 

3.解压jdk并配置环境变量，该操作需要在三台电脑中都要进行配置

[mac@centos1 ~]$ tar -zxvf /opt/software/jdk-7u67-linux-x64.tar.gz -C /opt/modules/

配置环境变量


vi ~/.bashrc

添加下面内容

export JAVA_HOME=/opt/modules/jdk1.7.0_67
export CLASS_PATH=$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$CLASS_PATH

最后执行source操作使其生效

source ~/.bashrc

验证jdk是否配置正确

java

javac


java -version
java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

同样需要在centos2和centos3中进行配置

4.解压hadoop压缩包到modules目录下,并删除hadoop目录下占用很多空间的的doc帮助文档

[mac@centos1 modules]$ tar -zxvf /opt/software/hadoop-2.5.0.tar.gz -C /opt/modules/

rm -rf /opt/modules/hadoop-2.5.0/share/doc/

5.切换到hadoop配置文件目录,修改env文件及site配置文件

cd /opt/modules/hadoop-2.5.0/etc/hadoop

首先注意这里有两类配置文件，只读性质的default配置文件及可修改的Site配置文件

这里使用sublime的package control插件sftp进行远程修改文件

安装完成后进入file->sftp/ftp->SetUp Server

进行centos1相关配置

{
    // The tab key will cycle through the settings when first created
    // Visit http://wbond.net/sublime_packages/sftp/settings for help
    
    // sftp, ftp or ftps
    "type": "sftp",

    "sync_down_on_open": true,
    "sync_same_age": true,
    
    "host": "centos1",
    "user": "mac",
    "password": "qwer1234",
    "port": "22",
    
    "remote_path": "/opt/",
    //"file_permissions": "664",
    //"dir_permissions": "775",
    
    //"extra_list_connections": 0,

    "connect_timeout": 30,
    //"keepalive": 120,
    //"ftp_passive_mode": true,
    //"ftp_obey_passive_host": false,
    //"ssh_key_file": "~/.ssh/id_rsa",
    //"sftp_flags": ["-F", "/path/to/ssh_config"],
    
    //"preserve_modification_times": false,
    //"remote_time_offset_in_hours": 0,
    //"remote_encoding": "utf-8",
    //"remote_locale": "C",
    //"allow_config_upload": false,
}

同样配置centos2和centos3的相关配置

配置完成后就可以通过file->sftp/ftp->Browser file进行远程修改文件了

比如对centos1的文件进行修改

6.主机角色分配

namenode centos1	

datanode centos1 centos2 centos3

resourcemanager centos2

nodemanager  centos1 centos2 centos3

seconderynamenode centos3

7.修改配置文件

修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JAVA_HOME


export JAVA_HOME=/opt/modules/jdk1.7.0_67

mkdir -p data/tmp

9.修改core-site.xml配置文件

<configuration>
    <property>
         <name>fs.defaultFS</name>
         <value>hdfs://centos1:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/modules/hadoop-2.5.0/data/tmp</value>
    </property>
</configuration>

10.修改hdfs-site.xml文件，配置secondarynamenode的相关信息

<configuration>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>centos3:50090</value>
    </property>
</configuration>

11.修改slaves配置文件内容，将centos1 centos2 centos3主机名称都加入到slaves配置文件中

centos1

centos2

centos3

12.配置centos2为resourcemanager资源管理器，修改yarn-site.xml

<configuration>
	<property>
      <name>yarn.resourcemanager.hostname</name>
      <value>centos2</value>
	</property>
	<property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
	</property>
	<property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
	</property>
	<property>
      <name>yarn.log-aggregation.retain-seconds</name>
      <value>640800</value>
	</property>
</configuration>

13.修改mapreduce-site的模板文件为mapreduce-site.xml，并修改为

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
  <property>
      <name>mapreduce.jobhistory.address</name>
      <value>centos1:10020</value>
  </property>
  <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>centos1:19888</value>
  </property>
</configuration>

到这里hadoop的配置文件算是修改结束

14.分发hadoop目录到各个节点下，由于hadoop集群之间使用ssh秘钥登录，因此hadoop主机之间需要能够进行相互之间的访问，这里为了省事就直接配置为相互都可以访问

在每台机器上都执行ssh-keygen的命令如下,过程中一路回车：

ssh-keygen -t  rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/mac/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/mac/.ssh/id_rsa.
Your public key has been saved in /home/mac/.ssh/id_rsa.pub.
The key fingerprint is:
e1:1d:e6:2c:ac:98:00:54:70:c0:9c:7e:7f:bb:1b:8c mac@centos1
The key's randomart image is:
+--[ RSA 2048]----+
|o+=o             |
|.+.              |
|o       . o      |
| o .   o * .     |
|  o .   S +      |
|   . +oo .       |
|    oEoo.        |
|       ..        |
|       oo        |
+-----------------+

然后使用ssh-copy-id将公钥复制到另外两台机器上,本身也需要操作

ssh-copy-id centos1

ssh-copy-id centos2

ssh-copy-id centos3

执行结束后分别用ssh链接其他机器测试下是否还需要登陆密码，如果不需要则证明成功

然后复制分发hadoop到centos2和centos3相同目录下

因此在centos1上执行下面命令

scp -r /opt/modules/hadoop-2.5.0/ centos2:/opt/modules/

scp -r /opt/modules/hadoop-2.5.0/ centos3:/opt/modules/

验证下centos2和centos3下是否存在hadoop目录了

[mac@centos2 ~]$ ls -al /opt/modules/
总用量 16
drwxr-xr-x.  4 mac  mac  4096 3月   1 04:44 .
drwxr-xr-x.  5 root root 4096 2月  28 21:56 ..
drwxr-xr-x. 10 mac  mac  4096 3月   1 04:44 hadoop-2.5.0
drwxr-xr-x.  8 mac  mac  4096 7月  26 2014 jdk1.7.0_67

[mac@centos3 ~]$ ls -al /opt/modules/
总用量 16
drwxr-xr-x.  4 mac  mac  4096 2月  28 20:44 .
drwxr-xr-x.  5 root root 4096 2月  28 13:56 ..
drwxr-xr-x. 10 mac  mac  4096 2月  28 20:44 hadoop-2.5.0
drwxr-xr-x.  8 mac  mac  4096 7月  26 2014 jdk1.7.0_67

15.在centos1中格式化hdfs文件系统

./bin/hdfs namenode -format
17/03/01 04:53:24 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = centos1.com/192.168.4.131
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.5.0

17/03/01 04:53:26 INFO common.Storage: Storage directory /opt/modules/hadoop-2.5.0/data/tmp/dfs/name has been successfully formatted.
17/03/01 04:53:27 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/03/01 04:53:27 INFO util.ExitUtil: Exiting with status 0
17/03/01 04:53:27 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at centos1.com/192.168.4.131
************************************************************/
...

16.启动hdfs文件系统

./sbin/start-dfs.sh 

17/03/01 04:58:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [centos1]
centos1: starting namenode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-mac-namenode-centos1.out
centos2: starting datanode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-mac-datanode-centos2.out
centos1: starting datanode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-mac-datanode-centos1.out
centos3: starting datanode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-mac-datanode-centos3.out
Starting secondary namenodes [centos3]
centos3: starting secondarynamenode, logging to /opt/modules/hadoop-2.5.0/logs/hadoop-mac-secondarynamenode-centos3.out
17/03/01 04:59:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

分别检测进程是否启动成功

centos1上

jps
Jps
NameNode
DataNode

centos2上

[mac@centos2 ~]$ jps
11096 DataNode
11157 Jps

centos3上

jps
DataNode
SecondaryNameNode
Jps

17.在centos2上启动yarn资源管理器

./sbin/start-yarn.sh 
starting yarn daemons
resourcemanager running as process 11402. Stop it first.
centos2: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-mac-nodemanager-centos2.out
centos3: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-mac-nodemanager-centos3.out
centos1: starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-mac-nodemanager-centos1.out

查看进程是否启动成功

centos2

[mac@centos2 hadoop-2.5.0]$ jps
DataNode
ResourceManager
Jps
NodeManager

centos1上

[mac@centos1 hadoop-2.5.0]$ jps
NodeManager
NameNode
DataNode
Jps

centos3上

 jps
DataNode
NodeManager
SecondaryNameNode
Jps

这里试了很多次每次都是nodemanager启动后关闭原因是防火墙的问题，需要将三台虚拟机的防火墙全部关闭


sudo service iptables stop

并将开启启动也关闭

sudo chkconfig iptables off

验证是否关闭成功

sudo chkconfig --list | grep iptables
iptables        0:关闭  1:关闭  2:关闭  3:关闭  4:关闭  5:关闭  6:关闭

18.启动日志服务器

./sbin/mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /opt/modules/hadoop-2.5.0/logs/mapred-mac-historyserver-centos3.out
[mac@centos3 hadoop-2.5.0]$ jps
2492 Jps
2461 JobHistoryServer
2226 SecondaryNameNode
2126 DataNode
2313 NodeManager

19.启动wordcount程序测试下是否有问题

首先上传一个文件到hdfs系统上

./bin/hdfs dfs -mkdir -p /usr/mac

./bin/hdfs dfs -copyFromLocal etc/hadoop/log4j.properties /usr/mac

执行wordcount程序

./bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /usr/mac /usr/mac/output

执行过程

17/03/01 14:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/03/01 14:01:21 INFO client.RMProxy: Connecting to ResourceManager at centos2/192.168.4.132:8032
17/03/01 14:01:22 INFO input.FileInputFormat: Total input paths to process : 1
17/03/01 14:01:22 INFO mapreduce.JobSubmitter: number of splits:1
17/03/01 14:01:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1488347393190_0002
17/03/01 14:01:23 INFO impl.YarnClientImpl: Submitted application application_1488347393190_0002
17/03/01 14:01:23 INFO mapreduce.Job: The url to track the job: http://centos2:8088/proxy/application_1488347393190_0002/
17/03/01 14:01:23 INFO mapreduce.Job: Running job: job_1488347393190_0002
17/03/01 14:01:31 INFO mapreduce.Job: Job job_1488347393190_0002 running in uber mode : false
17/03/01 14:01:31 INFO mapreduce.Job:  map 0% reduce 0%
17/03/01 14:01:38 INFO mapreduce.Job:  map 100% reduce 0%
17/03/01 14:01:44 INFO mapreduce.Job:  map 100% reduce 100%
17/03/01 14:01:44 INFO mapreduce.Job: Job job_1488347393190_0002 completed successfully
17/03/01 14:01:45 INFO mapreduce.Job: Counters: 49
  File System Counters
    FILE: Number of bytes read=11936
    FILE: Number of bytes written=217707
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=11227
    HDFS: Number of bytes written=10525
    HDFS: Number of read operations=6
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=2
  Job Counters 
    Launched map tasks=1
    Launched reduce tasks=1
    Data-local map tasks=1
    Total time spent by all maps in occupied slots (ms)=4779
    Total time spent by all reduces in occupied slots (ms)=4448
    Total time spent by all map tasks (ms)=4779
    Total time spent by all reduce tasks (ms)=4448
    Total vcore-seconds taken by all map tasks=4779
    Total vcore-seconds taken by all reduce tasks=4448
    Total megabyte-seconds taken by all map tasks=4893696
    Total megabyte-seconds taken by all reduce tasks=4554752
  Map-Reduce Framework
    Map input records=265
    Map output records=614
    Map output bytes=13507
    Map output materialized bytes=11936
    Input split bytes=109
    Combine input records=614
    Combine output records=352
    Reduce input groups=352
    Reduce shuffle bytes=11936
    Reduce input records=352
    Reduce output records=352
    Spilled Records=704
    Shuffled Maps =1
    Failed Shuffles=0
    Merged Map outputs=1
    GC time elapsed (ms)=117
    CPU time spent (ms)=1250
    Physical memory (bytes) snapshot=313421824
    Virtual memory (bytes) snapshot=1680121856
    Total committed heap usage (bytes)=136122368
  Shuffle Errors
    BAD_ID=0
    CONNECTION=0
    IO_ERROR=0
    WRONG_LENGTH=0
    WRONG_MAP=0
    WRONG_REDUCE=0
  File Input Format Counters 
    Bytes Read=11118
  File Output Format Counters 
    Bytes Written=10525

最后查看hdfs上的文件输出结果

./bin/hdfs dfs -cat /usr/mac/output/*
17/03/01 14:02:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
"AS 1
"License"); 1
"console" 1
"hadoop.root.logger". 1
# 87
#Default  1
#Security 1

......

至此 hadoop集群环境搭建完毕.

hadoop分布式环境搭建一(集群准备)

2017-02-28T08:52:00+08:00

安装准备

首先按照教程安装vmware虚拟机

链接

分布式环境搭建

1.复制安装好的centos虚拟机两份，并分别命名为centos2，centos3，这里搭建有三台主机的集群

安装后启动任意一台虚拟机如下所以，这里的用户名为mac是应为自己的mac电脑的用户名就是mac，这里为了方便自己学习方便ssh链接三台虚拟机而不用带用户名即可而使用mac用户名

2.添加自己用户的sudo权限

进入root用户

su

修改/etc/sudoers文件


vi /etc/sudoers

添加如下一行内容

mac     ALL=(ALL)       ALL

修改内容如下

## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL
mac     ALL=(ALL)       ALL

当然，同样需要分别修改另外两台

3.开启centos虚拟机的ssh服务

首先查询电脑是否安装了ssh服务，默认情况下centos都有安装

rpm -qa | grep ssh

openssh-5.3p1-84.1.el6.x86_64
openssh-server-5.3p1-84.1.el6.x86_64
libssh2-1.4.2-1.el6.x86_64
ksshaskpass-0.5.1-4.1.el6.x86_64
openssh-askpass-5.3p1-84.1.el6.x86_64
openssh-clients-5.3p1-84.1.el6.x86_64

验证服务是否开启

sudo service sshd status
openssh-daemon is stopped

开启服务

sudo service sshd start
Starting sshd:                                             [  OK  ]

设置sshd开机自启动

sudo chkconfig --list | grep sshd
sshd           	0:关闭	1:关闭	2:关闭	3:关闭	4:关闭	5:关闭	6:关闭

检查下是否开机自启动，发现没有设置

sudo chkconfig sshd on

再次检查

sudo chkconfig --list | grep sshd
sshd           	0:关闭	1:关闭	2:启用	3:启用	4:启用	5:启用	6:关闭

也可以重启电脑测试是否启动

同样设置其他两台电脑

4.直接在mac电脑上进行远程连接即可了

ssh centos1 centos2 centos 的ip地址即可

ssh1 192.168.4.131

ssh1 192.168.4.132

ssh1 192.168.4.133

5.分别修改三台虚拟机的主机名称为centos1，centos2，centos3

sudo vi /etc/sysconfig/network

[root@localhost mac]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=centos1

修改完成后没有立即生效，这里需要重启，如果不想重启可以使用下面命令

sudo hostname centos1

可以通过下面命令进行验证是否修改成功

[root@localhost mac]# hostname
centos1

同样操作另外两台电脑

6.由于拷贝完成后的虚拟机会多出来一个网卡，因此这里需要将centos2和centos3中多余的网卡删掉

首先查看文件/etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:ba:52:6d", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:35:92:1e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:36:21:ee", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

查看自己的ip地址中的mac地址是哪个

ifconfig
eth2      Link encap:Ethernet  HWaddr 00:50:56:36:21:EE  
          inet addr:192.168.4.143  Bcast:192.168.4.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe36:21ee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:148 errors:0 dropped:0 overruns:0 frame:0
          TX packets:111 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:16913 (16.5 KiB)  TX bytes:15711 (15.3 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:480 (480.0 b)  TX bytes:480 (480.0 b)

发现是最后一个，因此需要将前面的两个删掉，同事修改最后一个的eth2位eth0，修改后的配置如下

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:36:21:ee", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

最后修改/etc/sysconfig/network-scripts/目录下ifcfg-带有eh0的文件中的mac地址为上面的mac地址00:50:56:36:21:EE

sudo vi /etc/sysconfig/network-scripts/ifcfg-Auto_eth0

修改后的文件如下

TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Auto eth0"
UUID=bf045fa6-3ff3-4b53-9f9c-534985123cb1
ONBOOT=yes
HWADDR=00:50:56:36:21:EE
PEERDNS=yes
PEERROUTES=yes
LAST_CONNECT=1488272496

重启电脑就会发现网卡剩余一个了

同样修改另外一台电脑并重启

6.修改三台虚拟机的ip地址

由于安装完虚拟机后的ip是随机分配ip，可能在重启电脑后，ip地址会换掉，这样集群就肯定会出问题，所以这里直接绑定固定的ip地址就可以了这里的三台虚拟机的ip地址分配为如下 centos1 192.168.4.131 centos2 192.168.4.132 centos3 192.168.4.133

这样就进入虚拟机进行设置

修改完成后需要重启网卡

sudo service network restart
[sudo] password for mac: 
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]

将ip映射关系加入到centos1 centos2 和centos3中三台电脑的host文件中

sudo vi /etc/hosts

加入

168.4.131   centos1.com     centos1
168.4.132   centos2.com     centos2
168.4.133   centos3.com     centos3

当然可以加入到自己的mac电脑上这样就可以直接ssh主机了

按照上面步骤执行完成后，虚拟机是不能上网的，因为我们改成了固定ip地址，为了能同时上网，我们这里需要再添加一个网卡

首先关闭虚拟机，然后添加网卡

添加完成后启动系统测试下是否联网

sudo service network restart
[sudo] password for mac: 
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface Auto_eth0:  Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/4
                                                           [  OK  ]

[mac@centos1 ~]$ ping www.baidu.com
PING www.a.shifen.com (115.239.211.112) 56(84) bytes of data.
64 bytes from 115.239.211.112: icmp_seq=1 ttl=128 time=8.84 ms
64 bytes from 115.239.211.112: icmp_seq=2 ttl=128 time=8.66 ms

到这里，三台虚拟机的环境准备已经ok，下一步就是安装分布式系统了

ofbiz16.11环境搭建

2017-02-19T08:52:00+08:00

ofbiz下载安装

1.下载

进入ofbiz官方页面http://ofbiz.apache.org/download.html页面下载最新版本16.11.01

2.下载gradle

进入https://gradle.org/install#download页面下载gradle最新版本，这里是gradle-3.2.1-bin.zip

3.解压ofbiz后，将gradle-3.2.1-bin.zip拷贝到ofbiz目录下的gradle/wrapper/目录中

4.修改gradle/wrapper/中的文件gradle-wrapper.properties中的distributionUrl属性，修改后如下所示

#Sat Feb 18 13:07:49 CST 2017
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
#distributionUrl=https\://services.gradle.org/distributions/gradle-2.13-all.zip
distributionUrl=gradle-3.2.1-bin.zip

5.执行命令前修改gradle下载镜像地址为阿里云地址，只需要在用户主目录下的gradle相关目录下创建文件，并填写如下内容即可

这里的主目录为/Users/mac/.gradle，创建文件init.gradle，并填写如下内容

allprojects{
    repositories {
        def ALIYUN_REPOSITORY_URL = 'http://maven.aliyun.com/nexus/content/groups/public'
        def ALIYUN_JCENTER_URL = 'http://maven.aliyun.com/nexus/content/repositories/jcenter'
        all { ArtifactRepository repo ->
            if(repo instanceof MavenArtifactRepository){
                def url = repo.url.toString()
                if (url.startsWith('https://repo1.maven.org/maven2')) {
                    project.logger.lifecycle "Repository ${repo.url} replaced by $ALIYUN_REPOSITORY_URL."
                    remove repo
                }
                if (url.startsWith('https://jcenter.bintray.com/')) {
                    project.logger.lifecycle "Repository ${repo.url} replaced by $ALIYUN_JCENTER_URL."
                    remove repo
                }
            }
        }
        maven {
        	url ALIYUN_REPOSITORY_URL
            url ALIYUN_JCENTER_URL
        }
    }
}

6.执行启动命令,执行过程中会下载相关依赖包，并进行启动


gradlew loadDefault ofbiz

启动完成后会出现下面提示：

 [[[main(Domain:https://localhost)] Request Done- total:0.595,since last([main(Domain:http...):0.595]]
> Building 92% > :ofbiz

卡在这里不动的时候可以忽略

7.登录页面http://localhost:8080/catalog 后会自动跳转到登录页面https://localhost:8443/catalog/control/main，输入用户名及密码admin/ofbiz

修改默认数据库为mysql数据库

1.新建ofbiz相关数据库

mysql -uroot -p123456

mysql> create database ofbiz;
Query OK, 1 row affected (0.00 sec)

mysql> create database ofbizolap;
Query OK, 1 row affected (0.00 sec)

mysql> create database ofbiztenant;
Query OK, 1 row affected (0.00 sec)

mysql> create database ofbiz_odbc;
Query OK, 1 row affected (0.00 sec)

2.修改配置文件apache-ofbiz-16.11.01/framework/entity/config/entityengine.xml

修改localmysql，localmysqlolap，localmysqltenant,odbcmysql节点中的jdbc-username,jdbc-password为自己mysql的用户名及密码

<datasource name="localmysql"
        helper-class="org.apache.ofbiz.entity.datasource.GenericHelperDAO"
        field-type-name="mysql"
        check-on-start="true"
        add-missing-on-start="true"
        check-pks-on-start="false"
        use-foreign-keys="true"
        join-style="ansi-no-parenthesis"
        alias-view-columns="false"
        drop-fk-use-foreign-key-keyword="true"
        table-type="InnoDB"
        character-set="latin1"
        collate="latin1_general_cs">
    <read-data reader-name="tenant"/>
    <read-data reader-name="seed"/>
    <read-data reader-name="seed-initial"/>
    <read-data reader-name="demo"/>
    <read-data reader-name="ext"/>
    <read-data reader-name="ext-test"/>
    <read-data reader-name="ext-demo"/>
    <inline-jdbc
            jdbc-driver="com.mysql.jdbc.Driver"
            jdbc-uri="jdbc:mysql://127.0.0.1/ofbiz?autoReconnect=true"
            jdbc-username="root"
            jdbc-password="123456"
            isolation-level="ReadCommitted"
            pool-minsize="2"
            pool-maxsize="250"
            time-between-eviction-runs-millis="600000"/><!-- Please note that at least one person has experienced a problem with this value with MySQL
            and had to set it to -1 in order to avoid this issue.
            For more look at http://markmail.org/thread/5sivpykv7xkl66px and http://commons.apache.org/dbcp/configuration.html-->
    <!-- <jndi-jdbc jndi-server-name="localjndi" jndi-name="java:/MySqlDataSource" isolation-level="Serializable"/> -->
</datasource>
<datasource name="localmysqlolap"
        helper-class="org.apache.ofbiz.entity.datasource.GenericHelperDAO"
        field-type-name="mysql"
        check-on-start="true"
        add-missing-on-start="true"
        check-pks-on-start="false"
        use-foreign-keys="true"
        join-style="ansi-no-parenthesis"
        alias-view-columns="false"
        drop-fk-use-foreign-key-keyword="true"
        table-type="InnoDB"
        character-set="latin1"
        collate="latin1_general_cs">
    <read-data reader-name="tenant"/>
    <read-data reader-name="seed"/>
    <read-data reader-name="seed-initial"/>
    <read-data reader-name="demo"/>
    <read-data reader-name="ext"/>
    <read-data reader-name="ext-test"/>
    <read-data reader-name="ext-demo"/>
    <inline-jdbc
            jdbc-driver="com.mysql.jdbc.Driver"
            jdbc-uri="jdbc:mysql://127.0.0.1/ofbizolap?autoReconnect=true"
            jdbc-username="root"
            jdbc-password="123456"
            isolation-level="ReadCommitted"
            pool-minsize="2"
            pool-maxsize="250"
            time-between-eviction-runs-millis="600000"/><!-- Please note that at least one person has experienced a problem with this value with MySQL
            and had to set it to -1 in order to avoid this issue.
            For more look at http://markmail.org/thread/5sivpykv7xkl66px and http://commons.apache.org/dbcp/configuration.html-->
    <!-- <jndi-jdbc jndi-server-name="localjndi" jndi-name="java:/MySqlDataSource" isolation-level="Serializable"/> -->
</datasource>
<datasource name="localmysqltenant"
        helper-class="org.apache.ofbiz.entity.datasource.GenericHelperDAO"
        field-type-name="mysql"
        check-on-start="true"
        add-missing-on-start="true"
        check-pks-on-start="false"
        use-foreign-keys="true"
        join-style="ansi-no-parenthesis"
        alias-view-columns="false"
        drop-fk-use-foreign-key-keyword="true"
        table-type="InnoDB"
        character-set="latin1"
        collate="latin1_general_cs">
    <read-data reader-name="tenant"/>
    <read-data reader-name="seed"/>
    <read-data reader-name="seed-initial"/>
    <read-data reader-name="demo"/>
    <read-data reader-name="ext"/>
    <read-data reader-name="ext-test"/>
    <read-data reader-name="ext-demo"/>
    <inline-jdbc
            jdbc-driver="com.mysql.jdbc.Driver"
            jdbc-uri="jdbc:mysql://127.0.0.1/ofbiztenant?autoReconnect=true"
            jdbc-username="root"
            jdbc-password="123456"
            isolation-level="ReadCommitted"
            pool-minsize="2"
            pool-maxsize="250"
            time-between-eviction-runs-millis="600000"/><!-- Please note that at least one person has experienced a problem with this value with MySQL
            and had to set it to -1 in order to avoid this issue.
            For more look at http://markmail.org/thread/5sivpykv7xkl66px and http://commons.apache.org/dbcp/configuration.html-->
    <!-- <jndi-jdbc jndi-server-name="localjndi" jndi-name="java:/MySqlDataSource" isolation-level="Serializable"/> -->
</datasource>
<datasource name="odbcmysql"
        helper-class="org.apache.ofbiz.entity.datasource.GenericHelperDAO"
        field-type-name="mysql"
        check-on-start="true"
        add-missing-on-start="true"
        check-pks-on-start="false"
        use-foreign-keys="true"
        join-style="ansi-no-parenthesis"
        alias-view-columns="false"
        drop-fk-use-foreign-key-keyword="true"
        table-type="InnoDB"
        character-set="latin1"
        collate="latin1_general_cs">
    <read-data reader-name="tenant"/>
    <read-data reader-name="seed"/>
    <inline-jdbc
            jdbc-driver="com.mysql.jdbc.Driver"
            jdbc-uri="jdbc:mysql://127.0.0.1/ofbiz_odbc?autoReconnect=true"
            jdbc-username="root"
            jdbc-password="123456"
            isolation-level="ReadCommitted"
            pool-minsize="2"
            pool-maxsize="250"
            time-between-eviction-runs-millis="600000"/>
    <!-- <jndi-jdbc jndi-server-name="localjndi" jndi-name="java:/MySqlDataSource" isolation-level="Serializable"/> -->
</datasource>

3.仍然修改上面的配置文件中的开始的代理配置为mysql的配置

<delegator name="default" entity-model-reader="main" entity-group-reader="main" entity-eca-reader="main" distributed-cache-clear-enabled="false">
    <group-map group-name="org.apache.ofbiz" datasource-name="localmysql"/>
    <group-map group-name="org.apache.ofbiz.olap" datasource-name="localmysqlolap"/>
    <group-map group-name="org.apache.ofbiz.tenant" datasource-name="localmysqltenant"/>
</delegator>
<delegator name="default-no-eca" entity-model-reader="main" entity-group-reader="main" entity-eca-reader="main" entity-eca-enabled="false" distributed-cache-clear-enabled="false">
    <group-map group-name="org.apache.ofbiz" datasource-name="localmysql"/>
    <group-map group-name="org.apache.ofbiz.olap" datasource-name="localmysqlolap"/>
    <group-map group-name="org.apache.ofbiz.tenant" datasource-name="localmysqltenant"/>
</delegator>

<!-- be sure that your default delegator (or the one you use) uses the same datasource for test. You must run "gradlew loadDefault" before running "gradlew testIntegration" -->
<delegator name="test" entity-model-reader="main" entity-group-reader="main" entity-eca-reader="main">
    <group-map group-name="org.apache.ofbiz" datasource-name="localmysql"/>
    <group-map group-name="org.apache.ofbiz.olap" datasource-name="localmysqlolap"/>
    <group-map group-name="org.apache.ofbiz.tenant" datasource-name="localmysqltenant"/>
</delegator>

4.重新执行命令


./gradlew loadDefault ofbiz

spark

2017-02-07T08:52:00+08:00

spark下载安装

1.下载

进入spark官方页面http://spark.apache.org/downloads.html

选择Spark版本为1.6.1
打包类型为Pre-built for hadoop 2.6版本
然后选择下载

2.安装下载完成后直接解压，进入目录，并执行命令即可：

./bin/spark-shell 

 sc.version
res3: String = 1.6.1      

scala> sc.textFile("README.md")
res4: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at <console>:28

scala> res4.count()
res5: Long = 95                                                                 

scala> res4.first()
res6: String = # Apache Spark

查看页面可以看到相关信息 
http://localhost:4040/jobs/

rabbitmq环境搭建安装及官方例子学习使用

2017-01-07T08:52:00+08:00

rabbitmq软件安装与启动

1.rabbitmq安装

这里使用docker的方式，方便

docker pull rabbitmq
Using default tag: latest
latest: Pulling from library/rabbitmq
Digest: sha256:53c67c41323977870ad077c8a0e5ef52b2cc460206e6a965ced96ea130de1e12
Status: Image is up to date for rabbitmq:latest

也可以参考官方文档来下载安装,地址

2.rabbitmq启动

docker run -d --name rabbitmq -p 5672:5672 rabbitmq
413f469182190d05235f75337077125497eefe57abec3f37936897014c881f5a

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                   PORTS                                                   NAMES
413f46918219        rabbitmq            "docker-entrypoint.sh"   6 seconds ago       Up 5 seconds             4369/tcp, 5671/tcp, 25672/tcp, 0.0.0.0:5672->5672/tcp   rabbitmq

开启web端口访问管理控制页面

上面的docker启动rabbitmq的时候并没有释放rabbitmq的管理控制台的端口，下面进行重新操作

为了方便，这里我们直接将原先启动的容器stop掉，然后删掉，重新运行释放端口,rabbitmq的管理控制台端口为15672

docker stop 2d5e5be34be7
2d5e5be34be7

docker rm 2d5e5be34be7
2d5e5be34be7

docker run -d --name rabbitmq -p 5672:5672  -p 15672:15672 rabbitmq
8c015f9b6beb4e14e7726a7367c0e9571ae4987dca8bfb64343fd8f3eeaade03

启动容器后，执行docker命令，找到container的id

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS                                                                             NAMES
8c015f9b6beb        rabbitmq            "docker-entrypoint.sh"   10 hours ago        Up About a minute             4369/tcp, 0.0.0.0:5672->5672/tcp, 5671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp   rabbitmq

找到id为8c015f9b6beb 执行下面命令进入docker的命令控制台下

docker exec -it 8c015f9b6beb /bin/bash 
root@8c015f9b6beb:/# 

然后在docker的控制台下执行下面命令来启动rabbitmq的管理控制台的插件

rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
  mochiweb
  webmachine
  rabbitmq_web_dispatch
  amqp_client
  rabbitmq_management_agent
  rabbitmq_management

Applying plugin configuration to rabbit@8c015f9b6beb... started 6 plugins.

刷新页面http://localhost:15672/#/会看到rabbitmq的登录页面,通过guest/guest进行登录

参考链接:http://www.rabbitmq.com/management.html

编写客户端例子

rabbitmq中的helloworld(一个生产者一个消费者，一条消息只能发送给一个消费者)

简介

rabbitmq是一个消息代理。主要的目的很简单：接受和转发消息。你可以把他当做一个邮局：当你往邮箱放入你的信件的时候，你非常确定邮递员最终会将信件放到接收者手中。这个比喻把rabbitmq比作一个邮箱，邮局和邮递员。

在rabbitmq和邮局之间最主要的区别在于，他不是处理纸质的东西，而是接收，存储和转发二进制的blob数据消息。

rabbitmq或者一般的消息中存在一些专业术语

1 生产者意味着消息发送者。一个发送消息的程序叫做生产者，我们叫做P，图示如下：

2 队列是一个邮箱的名字。存在于rabbitmq中。尽管消息在rabbitmq和你的应用程序之间进行流通，他们只能存储在队列内部。一个队列不受任何限制，你可以存放任意多的消息，本质上来说他就是一个无穷的缓冲区。许多生产者可以往一个队列中发送消息，许多消费者也可以尝试从一个队列中接受数据。一个队列可以按照下图所示，上面是队列的名字。

3 消费者跟接受的意思类似，一个消费者就是一个等待接受消息的程序。我们可以如下图所画，简称为C

注意，生产者，消费者和代理不需要部署在同一台机器上。当然在大多数的应用中他们也确实不在同一台机器上。

使用java客户端

这部分我们将用java语言编写两个应用程序：一个发送单个消息的生产者和一个接受消息并打印他们的消费者。我们会忽视java的api的细节，专注于仅仅将例子跑通开始。一个hello world的消息例子。在下图中，P使我们的生产者，C是我们的消费者，中间的箱子是队列-一个rabbitmq为消费者存放消息的缓存。

配置

这里使用rabbitmq的官方客户端

在pom.xml文件中添加下面依赖

<dependency>
  <groupId>com.rabbitmq</groupId>
  <artifactId>amqp-client</artifactId>
  <version>4.0.1</version>
</dependency>

最新的版本可以通过连接查看

Sending

我们称我们的消息发送者为send，我们的消息接收者为Recv。发送者会连接rabbitmq，发送消息，然后停止。

在Send.java中我们需要导入下面包

import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;

设置类并命名队列

public class Send {
  private final static String QUEUE_NAME = "hello";

  public static void main(String[] argv)
      throws java.io.IOException {
  }
}

然后我们可以创建一个连接服务端的链接

ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();

链接是对socket链接的抽象，并且负责版本协议的协商和认证。这里我们链接了本地服务器的代理，也就是localhost。如果我们想链接一个不同机器的代理，我们仅需要修改这里的名称或者ip地址即可。

接下来，我们创建一个channel通道，我们的大部分操作的api都在这里。

为了发送，我们必须声明一个我们要发送信息的目标队列，然后我们往队列中发送消息。

channel.queueDeclare(QUEUE_NAME,false,false,false,null);
String message = "Hello world!";
channel.basicPublish("",QUEUE_NAME,null,message.getBytes("UTF-8"));
System.out.println(" [x] Sent '"+message+"'");

声明一个队列是等幂的，如果队列不存在会创建一个，消息内容是一个字节数组，所以你可以想随心所欲的编码你的内容。

最后我们关闭channel通道和链接。

channel.close();
connection.close();

下面是Send.java的所有内容

package io.github.codetosurvive1.helloworld;

/**
 * Created by mac on 2017/1/7.
 */

import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;

public class Send {

    private final static String QUEUE_NAME = "hello";


    public static void main(String args[]) throws Exception{

        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.queueDeclare(QUEUE_NAME,false,false,false,null);
        String message = "Hello world!";
        channel.basicPublish("",QUEUE_NAME,null,message.getBytes("UTF-8"));
        System.out.println(" [x] Sent '"+message+"'");

        channel.close();
        connection.close();

    }

}

发送消息不生效

如果你第一次使用rabbitmq并且没有看到你发送的消息，然后你可能会疑惑什么地方做错了。很有可能是代理没有提供足够的剩余空间启动的，默认是1g，因此拒绝了所有的消息。检查代理的日志文件来确认是否是这一点，如果可能可以减少限制。配置文件可以参考链接中的disk_free_limit

运行Server.java例子成功后，查看webui页面中是否有发送的消息

发现新建队列hello成功

点击hello队列进入查看详细信息

Receiving

上面是我们的消息发送者。不像发送者仅仅发送一条消息一样，我们的接收者是从rabbitmq中接受到推送过来的消息，所以我们需要不停的监听消息并进行输出。

Recv.java几乎跟Send一样

import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Consumer;
import com.rabbitmq.client.DefaultConsumer;

额外的DefaultConsumer是一个实现Consumer消费者接口的类，我们用它来缓存从server推送过来的消息。

跟sender一样新建类，创建一个连接和一个通道，声明一个我们用来消费的队列，注意，这应该跟我们把消息推送给的相应的队列对应。

public class Recv {

    private static final String QUEUE_NAME="hello";

    public static void main(String args[]) throws Exception{
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();


        channel.queueDeclare(QUEUE_NAME,false,false,false,null);

        System.out.println("waiting for messages.To exit press CTRL+C");
    }

}

注意我们这里也声明了队列，因为我们可能在sender之前启动了receiver，在我么尝试消费消息之前应该确保队列已经存在。

我们然后需要告诉服务器给我们从队列中传输消息。然后他会异步的给我们推送消息，我们以一个对象的形式提供一个回调函数，来给我们缓存消息知道我们使用它们。这就是DefaultConsumer父类做的。

Consumer consumer = new DefaultConsumer(channel){
    @Override
    public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        String message = new String(body,"UTF-8");

        System.out.println("received:'"+message+"'");
    }
};

channel.basicConsume(QUEUE_NAME,consumer);

下面是所有的Recv.java代码

package io.github.codetosurvive1.helloworld;

import com.rabbitmq.client.*;

import java.io.IOException;

/**
 * Created by mac on 2017/1/8.
 */
public class Recv {

    private static final String QUEUE_NAME="hello";

    public static void main(String args[]) throws Exception{
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();


        channel.queueDeclare(QUEUE_NAME,false,false,false,null);

        System.out.println("waiting for messages.To exit press CTRL+C");


        Consumer consumer = new DefaultConsumer(channel){
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
                String message = new String(body,"UTF-8");

                System.out.println("received:'"+message+"'");
            }
        };

        channel.basicConsume(QUEUE_NAME,consumer);

    }

}

运行Recv.java后

waiting for messages.To exit press CTRL+C
received:'Hello world!'

通过页面查看，已经没有上次的消息了

再运行一次server.java发现控制台中又多了一条打印信息

waiting for messages.To exit press CTRL+C
received:'Hello world!'
received:'Hello world!'

使用amqp.node客户端

配置

首选安装客户端


npm install amqplib

send.js中需要首先引入库文件


var amqp = require('amqplib/callback_api');

然后连接rabbitmq服务器


amqp.connect('amqp://localhost', function(err, conn) {});

接下来创建一个通道，


amqp.connect('amqp://localhost', function(err, conn) {
    conn.createChannel(function(err, ch) {});
});

为了发送消息，我们必须声明我们要发送的目的队列，然后向队列中发送消息


amqp.connect('amqp://localhost', function(err, conn) {
    conn.createChannel(function(err, ch) {
        var q = "hello";
        ch.assertQueue(q, { durable: false });

        ch.sendToQueue(q, new Buffer('Hello world!'));
        console.log(" Sent 'hello world!");
    });
});

最后关闭连接并退出


setTimeout(function() {
        conn.close();
        process.exit(0);
    }, 500);

Sender.js的全部代码如下：


#!/usr/bin/env node

var amqp = require('amqplib/callback_api');

amqp.connect('amqp://localhost', function(err, conn) {
    conn.createChannel(function(err, ch) {
        var q = "hello";
        ch.assertQueue(q, { durable: false });

        ch.sendToQueue(q, new Buffer('Hello world!'));
        console.log(" Sent 'hello world!");

    });

    setTimeout(function() {
        conn.close();
        process.exit(0);
    }, 500);
});

Receiving


#!/usr/bin/env node

var amqp = require('amqplib/callback_api');

amqp.connect('amqp://localhost', function(err, conn) {
    conn.createChannel(function(err, ch) {
        var q = "hello";
        ch.assertQueue(q, { durable: false });
        console.log("waiting for messages");

        ch.consume(q, function(msg) {
            console.log(" received message %s", msg.content.toString());
        }, { noAck: true });
    })
})

./send.js

./receive.js

scala

Send.scala

package codetosurvive1.helloworld

import com.rabbitmq.client.ConnectionFactory;

object Send {

  private val QUEUE_NAME="hello";

  def main(argv:Array[String]){

    val factory = new ConnectionFactory();
    factory.setHost("localhost");
    val connection = factory.newConnection();
    val channel = connection.createChannel();
    channel.queueDeclare(QUEUE_NAME,false,false,false,null);

    val message = "hello world!";
    channel.basicPublish("",QUEUE_NAME,null,message.getBytes("UTF-8"));
    println("sent message :"+message);

    channel.close();
    connection.close();
  }
}

Receive.scala

package codetosurvive1.helloworld;

import com.rabbitmq.client.AMQP.BasicProperties
import com.rabbitmq.client.{ConnectionFactory, DefaultConsumer, Envelope};

/**
  * Created by mac on 2017/1/8.
  */
object Receive {

  private val QUEUE_NAME="hello";

  def main(args:Array[String]){

    val factory = new ConnectionFactory();
    factory.setHost("localhost");

    val connection = factory.newConnection();
    val channel = connection.createChannel();

    channel.queueDeclare(QUEUE_NAME,false,false,false,null);

    println("waiting for messages.");

    val consumer = new DefaultConsumer(channel){
      override def handleDelivery(consumerTag: String,
                                  envelope: Envelope,
                                  properties: BasicProperties,
                                  body: Array[Byte]): Unit = {
        var message = new String(body,"UTF-8");
        println("received message :"+message);
      }
    }

    channel.basicConsume(QUEUE_NAME,true,consumer);
  }

}

rabbitmq中的work queues工作队列（一个生产者多个消费者，一条消息只能发送给一个消费者）

HelloWorld例子中我们编写程序来往一个已经命名的队列中发送和接受消息。下面我们将要创建一个工作队列用来分发耗时的任务到不同的工人中。工作队列或者任务队列的主要思想是为了避免立刻去做一些资源密集性的任务并不必等待他完成。相反我们调度任务来后面去完成。一个后台运行的工作进程会弹出任务斌给最终执行任务。当你运行多个工作进程，任务将在他们之间共享。

这一部分在web应用中尤其有用，因为在一个短的http请求中不可能处理非常复杂的任务。

准备：

在helloworld例子中，我们发送了一个包含“hello world”内容的消息。现在我们将发送代表复杂任务的字符串。我们不需要真是的工作任务，比如修改图片尺寸或者需要渲染的pdf文件，我们就仅仅通过Thread.sleep()方法来假设人任务的复杂程度。我们将.的个数作为任务的复杂程度。每一个点代表一秒的工作，比如假设任务描述为Hello…将会消耗三秒钟。

我们仅需要简单的修改我们之前的Send.java文件来允许任意的消息被发送。这个程序会调度任务到我们的工作队列中，我们命名为NewTask.java

package io.github.codetosurvive1.workqueues;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.MessageProperties;

public class NewTask {

    private static final String TASK_QUEUE_NAME = "task_queue";

    public static void main(String[] argv) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);

        String message = getMessage(argv);

        channel.basicPublish("", TASK_QUEUE_NAME,
                MessageProperties.PERSISTENT_TEXT_PLAIN,
                message.getBytes("UTF-8"));
        System.out.println(" [x] Sent '" + message + "'");

        channel.close();
        connection.close();
    }

    private static String getMessage(String[] strings) {
        if (strings.length < 1)
            return "Hello World!";
        return joinStrings(strings, " ");
    }

    private static String joinStrings(String[] strings, String delimiter) {
        int length = strings.length;
        if (length == 0) return "";
        StringBuilder words = new StringBuilder(strings[0]);
        for (int i = 1; i < length; i++) {
            words.append(delimiter).append(strings[i]);
        }
        return words.toString();
    }
}

NewTask所做的工作便是根据命令行传递的参数来往task_queue队列中发送消息，并且队列是可以序列化到硬盘的，而worker所做的工作便是接受队列发送来的消息，并根据消息的.的个数来执行时间复杂度不同的处理操作。

分发策略：轮询分发策略（默认）和公平策略

上面的任务队列或者工作队列的一个优点就是可以并行的工作。一旦工作有积压，我们仅仅通过增加工人就可以很好的进行拓展。

默认的情况下，rabbitmq将会按顺序分发消息给下一个消费者，平均而言，每个消费者会得到相同数量的消息，这种分发消息的策略成为轮询分发策略。

消息应答

执行一个任务可能花费几秒。如果一个消费者开始了一个很长的任务并且只执行了部分任务就死掉了，你可能会疑问到底发生了什么。用现在的代码来说，每次rabbitmq讲消息传输给消费者后，就立即将消息从内存中删除。这样的话，如果你杀掉了一个worker，你将会丢失掉他正在处理的消息。我们也会丢失掉所有的我们分发到这个特殊worker上但是没有处理的消息。

但是我们不想丢失掉任何的任务。如果一个worker死掉了，我们期望的是这个任务会被分发给另外一个worker。

为了确保一条消息永远不会被丢失，rabbitmq支持消息应答。当消息被消费者接受，处理之后，消费者会给rabbitmq发送应答，rabbitmq在接受到应答之后才会删除掉这条消息。

如果一个消费者死掉了，包括channel通道关闭了，链接关闭了或者tcp链接丢失了，这样就没有应答，rabbitmq会理解为这个消息没有被处理完毕，他将会重新将消息仍回到队列中。如果这时候有其他的消费者在线，那么他将会分发消息到其他的消费者中。这样的话，你就能确保没有消息会丢失，及时worker工作者偶然会死亡掉。

这里没有任何的消息超时，如果消费者死掉了，rabbitmq会重新传输消息。尽管处理一个消息耗费了非常非常长久的时间，那么也是ok的。

消息应答默认的情况下是开着的。可以通过autoAck=true标志来设置将其关闭。

消息耐久性

上面讲解了如何确保一个消费者在死掉以后，任务不会丢失。但是我们的任务在rabbitmq服务停止掉后仍然会丢失。

当rabbitmq退出或者挂掉之后，他会遗忘掉队列和队列中的消息，除非你告诉他不要忘记。两件事情能够确保消息不会丢失：我们需要确队列和消息都是持久的。

首先我们需要确保rabbitmq永远不会丢失掉我们的队列，为了达到这个目的，我们需要声明队列为durable：

boolean durable = true;
channel.queueDeclare("hello", durable, false, false, null);

尽管这两条语句是对的，但是当前情况下是不生效的。因为我们之前已经声明hello队列是不持久化的。rabbitmq是不允许你用不同的参数重新定义一个已经存在的队列的，并且他会返回一个错误信息给任何想尝试这么做的程序。但是这里有一个快速生效的方法，就是声明一个不同的名字，比如task_queue

boolean durable = true;
channel.queueDeclare("task_queue", durable, false, false, null);

这个task_queue需要在producer和consumer的代码中都需要修改

这时候，我们确保task_queue队列在rabbitmq重启后不会丢失。现在我们需要将我们的消息进行持久化，通过设置MessageProperties(他实现了BasicProperties)为PERSISTENT_TEXT_PLAIN。

import com.rabbitmq.client.MessageProperties;

channel.basicPublish("", "task_queue",
            MessageProperties.PERSISTENT_TEXT_PLAIN,
            message.getBytes());

注意消息持久化

将消息标记为持久化并不能完全保证消息不会丢失。尽管告诉rabbitmq将消息保存到磁盘中，但是这里仍然会有一个短时间的错失窗口-当rabbitmq已经接收到消息但是还没有对他进行保存。而且rabbitmq不会为每条消息都做fsync(2)-他可能仅仅保存到缓存中并不是真正的写到磁盘中。持久化保证还不强健，但是对于我们简单的任务队列已经足够了。如果你想要一个更强健的保证，你可以使用 publisher confirms

公平调度

你可能已经注意到，分发消息并没有像我们期望的那样工作，比如，有两个worker工人，当所有的奇怪的信息很重量级，一些即时消息会很轻量级，一个worker将会一直忙碌，而另外一个将会基本不工作。好吧，rabbitmq并不知道这个事情，因此就会一直这样分配任务消息。

这样的事情会发生的原因在于rabbitmq在一个消息进入到队列中之后仅仅调度消息。他并没有查看每个消费者没有做出应答的消息的数字，他仅仅盲目的每n个消息轮回给每n个消费者。

为了检测到这个，我们可以使用basicQos方法来设置为1.这会告诉rabbitmq每次不要给消费者超过一个的消息。换句话说，在消费者处理并应答上条消息之前不要为其分发新的任务消息。这样，他就会分配给下一个不在忙碌的worker。

int prefetchCount = 1;
channel.basicQos(prefetchCount);

注意queue size

如果所有的workers都在忙碌中，你的队列可能会增长。如果你想注意下他，你可能增加更多的workers或者你使用其他的策略。

rabbitmq中的publish/subscribe（一条消息可以发送给多个消费者）

上面我们讲解了如何创建一个工作队列或者说是任务队列，一个生产者，一个队列，有多个消费者来消费。这里存在一个假设条件，就是每个任务仅仅被传送给一个消费者。接下来我们将会做一些彻底不同的东西-我们将阐述消息给多个消费者。这个模式成为主题的发布/订阅模式。

为了阐述这个模式，我们将会构造一个简单的日志系统。他包含两个程序-第一个是提交日志消息，第二个是接受消息并打印他们。

在我们的日志系统中，每个运行的receiver程序都会接受到消息。这样的话，我们就可以启动一个receiver来接受消息，然后序列化到磁盘；同事我们可以继续运行一个receiver，然后将日志显示到屏幕当中。

重要的是，发布的日志消息会广播给所有的receivers

Exchanges

在之前的章节中，我们从一个队列中发送和接受消息。现在，是时候介绍rabbit中的全部的消息模型了。

让我们快速的复习之前的章节中提到的内容：

一个producer是一个发送消息的应用程序。一个队列是一个存储消息的缓存。一个消费者是一个接受消息的应用程序。

在rabitmq中的消息模型的核心思想是：一个producer生产者绝不直接给队列发送消息。实际上，生产者也确实经常不知道一个消息是否被传输到一个队列。

相反，一个生产者只能将消息传递给交换区。一个交换区是一个非常简单的东西。一边，他从生产者那里接受消息，另一边他将消息推送到队列。exchange交换区必须准确的知道当接受到一个消息后该如何做。他应该继续追加到某个特定的队列？还是应该追加到很多队列中？或者是应该丢弃？这里的规则是通过exchange 类型定义的。

这里有几个可用的exchange类型：direct，topic，headers和fanout。我们这里会关注最后一个，fanout。让我们创建一个这种类型的exchange，并称作logs。

channel.exchangeDeclare("logs","fanout");

fanout类型的exchange非常简单，你可能仅通过名字就能猜测出，他仅仅广播他所接受到的消息到所有的他知道的队列中。而且这正好是我们的日志所需要的功能。

列出交换区
为了列出服务器中的交换区，你可以运行rabbitmqctl：

docker exec -it 8c015f9b6beb /bin/bash
root@8c015f9b6beb:/# rabbitmqctl list_exchanges     
Listing exchanges ...
amq.topic   topic
amq.headers headers
amq.direct  direct
amq.rabbitmq.trace  topic
amq.rabbitmq.log    topic
amq.match   headers
    direct
amq.fanout  fanout

在上面的列表中，这里有amq.*的交换区和默认的（没命名）的交换区。他们是默认创建的，但是并不是说你可以使用他们。

无名交换区

在之前的章节中我们并不知道任何关于交换区的概念，但是我们仍然能够给队列发送消息。这可能是因为我们使用了默认的交换区，也就是我们用“”标示的交换区。

回忆一下我们是怎样发送消息的：

channel.basicPublish("", "hello", null, message.getBytes());

第一个参数就是exchange的名字。空字符串标示默认的或者无名的exchange：消息被路由到routingKey所指定的队列中。

现在我们可以发布我们制定的交换区中

channel.basicPublish( "logs", "", null, message.getBytes());

临时队列

你可能记得之前我们用队列hello和task_queue两个队列了，能够指定一个队列对我们来说是很重要的，我们需要为消费者指定相同的队列。当你需要在生产者和消费者之间共享队列的时候，给队列命名就很重要了。

但是这并不是我们的日志例子的情况。我们想监听到所有的日志消息，你仅仅是一个子集。我们也只对当前的流动的信息感兴趣，对以前的老的没兴趣。为了解决这个问题，我们需要两件事情。

首先，无论我们什么时候链接rabbit，我们需要一个新鲜的的，空的队列。为了达到这个目的，我们可能用一个随机的名字来创建一个队列，更好的情况下是让server为我们选择一个随机的队列。

其次，一旦我们不再链接消费者之后，队列应该立刻被删除。

在java的client中，我们使用无参数的queueDeclare()，我们创建一个不持久化的，单独的，自动删除的队列。

String queueName = channel.queueDeclare().getQueue();

这样返回的队列名称是一个随机的名称，可能名字像这样：amq.gen-JzTY20BRgKO-HjmUJj0wLg

Bindings

我们已经创建了一个fanout的交换区和一个队列，现在我们需要告诉交换区发送消息到我们的队列。在交换区和队列之间的关系叫做绑定

channel.queueBind(queueName, "logs", "");

这样之后，logs交换区会将消息发送到我们的队列

列出bindings

你可以通过下面的指定列出存在的bingdings

rabbitmqctl list_bindings
Listing bindings ...
    exchange    task_queue  queue   task_queue  []
root@8c015f9b6beb:/# 

把他们合并在一起之后

产生日志消息的生产者程序，跟之前的看上去改动不大。最终要的改动是我们现在想发送消息到我们的日志交换区中而不是没有名字的默认的交换区。我们需要发送消息时候我们需要指定一个routingKey队列名称，但是对于fanout交换区而言，这会被忽略掉。下面是EmitLog.java的内容

package io.github.codetosurvive1.pubandsub;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
/**
 * Created by mac on 2017/1/9.
 */
public class EmitLog {

    private static final String EXCHANGE_NAME = "logs";

    public static void main(String[] argv) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.exchangeDeclare(EXCHANGE_NAME, "fanout");

        String message = getMessage(argv);

        channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes("UTF-8"));
        System.out.println(" [x] Sent '" + message + "'");

        channel.close();
        connection.close();
    }

    private static String getMessage(String[] strings){
        if (strings.length < 1)
            return "info: Hello World!";
        return joinStrings(strings, " ");
    }

    private static String joinStrings(String[] strings, String delimiter) {
        int length = strings.length;
        if (length == 0) return "";
        StringBuilder words = new StringBuilder(strings[0]);
        for (int i = 1; i < length; i++) {
            words.append(delimiter).append(strings[i]);
        }
        return words.toString();
    }
}

如你所见，当建立连接之后，我们声明了交换区。这一步是必须的，因为发布到不存在的交换区是不允许的。

如果没有队列被绑定到exchange上，消息就会丢失，但是对于我们而言还算ok，如果没有消费者在监听，我们完全可以丢弃消息。

ReceiveLogs.java代码：

package io.github.codetosurvive1.pubandsub;
import com.rabbitmq.client.*;
import java.io.IOException;

/**
 * Created by mac on 2017/1/9.
 */
public class ReceiveLogs {
    private static final String EXCHANGE_NAME = "logs";

    public static void main(String[] argv) throws Exception {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.exchangeDeclare(EXCHANGE_NAME, "fanout");
        String queueName = channel.queueDeclare().getQueue();
        channel.queueBind(queueName, EXCHANGE_NAME, "");

        System.out.println(" [*] Waiting for messages. To exit press CTRL+C");

        Consumer consumer = new DefaultConsumer(channel) {
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope,
                                       AMQP.BasicProperties properties, byte[] body) throws IOException {
                String message = new String(body, "UTF-8");
                System.out.println(" [x] Received '" + message + "'");
            }
        };
        channel.basicConsume(queueName, true, consumer);
    }
}

webdriveragent调试及usb测试

2017-01-06T08:52:00+08:00

操作步骤

1.首先启动手机上的facebook的webdriveragent，这里直接使用macaca改造后的webdriveragent测试

npm install app-inspector -g

WEBDRIVERAGENT_PORT=8900 xcodebuild test -project /usr/local/lib/node_modules/app-inspector/node_modules/webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj -scheme WebDriverAgentRunner -destination id=7286ffb19da8cb913b629211fad1f6c8b547ba1

解释：

WEBDRIVERAGENT_PORT=8900 指定手机端启动webdriver agent服务时候绑定手机端的端口，默认8100 -project /usr/local/lib/node_modules/app-inspector/node_modules/webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj 指定facebook的webdriver agent工程代码目录

-destination id=7286ffb19da8cb913b629211fad1f6c8b547ba1 id后面的数字是真机的device id，这里使用自己的真机

运行上面代码启动服务之后，控制台打印日志如下，最后成功信息为


writeDictToFile:1278 ==== Successfully wrote Manifest cache to /var/folders/cc/tqd_99wx1wb1sgpr8pjv4w140000gn/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/WebDriverAgentRunner-Runner.app.BgF3K6/WebDriverAgentRunner-Runner.app_sparse.ipa/ManifestCache.plist



Test Suite 'All tests' started at 2017-01-06 09:30:31.934
Test Suite 'WebDriverAgentRunner.xctest' started at 2017-01-06 09:30:31.936
Test Suite 'UITestingUITests' started at 2017-01-06 09:30:31.937
Test Case '-[UITestingUITests testRunner]' started.
    t =     0.00s     Start Test at 2017-01-06 09:30:31.941
    t =     0.01s     Set Up

2.映射手机端的端口到电脑上，使用的工具是iproxy

首先需要下载iproxy

brew install libimobiledevice

brew install usbmuxd

通过下面指令映射端口


/usr/local/bin/iproxy 8900 8900 7286ffb19da8cb913b629211fad1f6c8b547ba15

第一个8900：为电脑端的端口第二个8900：为手机端webdriveragent服务的端口最后的数字：为自己手机的序列号

3.测试

直接post url：http://127.0.0.1:8900/session 查看返回数据

{
  "value": "'bundleId' desired capability not provided",
  "sessionId": null,
  "status": 13
}

直接get url:http://127.0.0.1:8900/status 查看返回数据

{
  "value": {
    "state": "success",
    "os": {
      "name": "iOS",
      "version": "10.2"
    },
    "ios": {
      "simulatorVersion": "10.2"
    },
    "build": {
      "time": "Jan  6 2017 09:44:15"
    }
  },
  "sessionId": null,
  "status": 0
}

调试源代码

1.打开xcode项目

open /usr/local/lib/node_modules/app-inspector/node_modules/webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj

2.修改schema中的默认端口，并启动

手机端会安装webdriver agent，并最小化

安装成功后控制台会显示如下信息

2017-01-06 10:17:19.907330 XCTRunner[1773:647989] Running tests...
2017-01-06 10:17:20.969984 XCTRunner[1773:647989] Continuing to run tests in the background with task ID 1
Test Suite 'All tests' started at 2017-01-06 10:17:21.651
Test Suite 'WebDriverAgentRunner.xctest' started at 2017-01-06 10:17:21.653
Test Suite 'UITestingUITests' started at 2017-01-06 10:17:21.654
Test Case '-[UITestingUITests testRunner]' started.
    t =     0.00s     Start Test at 2017-01-06 10:17:21.657
    t =     0.00s     Set Up
2017-01-06 10:17:21.670553 XCTRunner[1773:648013] [User Defaults] Failed to write value for key AutomationDisableFauxCollectionCells in CFPrefsPlistSource<0x174111820> (Domain: com.apple.Accessibility, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null)): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access, switching to read-only
2017-01-06 10:17:21.672376 XCTRunner[1773:647989] Built at Jan  6 2017 10:17:06
2017-01-06 10:17:21.705725 XCTRunner[1773:647989] ServerURLHere->http://(null):8900<-ServerURLHere
2017-01-06 10:17:21.707139 XCTRunner[1773:647989] Listening on USB

看到ServerURLHere->http://(null):8900<-ServerURLHere证明启动成功

如果不进行上面的修改默认端口的话，直接test会显示如下信息，并且进行下面的3操作后是无法成功进行访问http://localhost:8100/status的
ServerURLHere->http://(null):null<-ServerURLHere

3.启动iproxy映射端口，通过usb通信


/usr/local/bin/iproxy 8900 8900 7286ffb19da8cb913b629211fad1f6c8b547ba15

4.访问http://127.0.0.1:8900/status，查看返回信息

{
  "value": {
    "state": "success",
    "os": {
      "name": "iOS",
      "version": "10.2"
    },
    "ios": {
      "simulatorVersion": "10.2"
    },
    "build": {
      "time": "Jan  6 2017 10:17:08"
    }
  },
  "sessionId": null,
  "status": 0
}

5.在打开的xcode的项目中搜索status接口，并打断点进行调试

继续访问http://127.0.0.1:8900/status，查看是否进入断点

成功进入断点

docker快速发布nodejs应用

2016-12-25T13:52:00+08:00

安装redis

1.docker安装redis

docker pull redis
Using default tag: latest
latest: Pulling from library/redis
75a822cd7888: Pull complete
e40c2fafe648: Pull complete
ce384d4aea4f: Pull complete
5e29dd684b84: Pull complete
29a3c975c335: Pull complete
a405554540f9: Pull complete
4b2454731fda: Pull complete
Digest: sha256:be03cb3b2e0d290c3d6670d1843b6036c1914663574bb624ffb62535344de5b4
Status: Downloaded newer image for redis:latest

mac:~ mac$ docker  images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              latest              67591570dd29        9 days ago          191.8 MB
redis               latest              d59dc9e6d0bf        10 days ago         182.9 MB
registry            latest              c9bd19d022f6        9 weeks ago         33.3 MB

2.启动redis

docker run --name redis-server -d redis redis-server --appendonly yes
e2c2f5eb97b3ad14a58dda238a8e3c19623e36555777576e57dc1da9bcb05d11

解释一下：

–name：是指定容器的名字 -d：是后台运行 redis-server –appendonly yes：是运行容器中的redis-server，并打开redis数据序列化功能

验证是否启动

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                    NAMES
e2c2f5eb97b3        redis               "docker-entrypoint.sh"   About a minute ago   Up About a minute   6379/tcp                 redis-server

安装nodejs

1.安装node镜像

docker pull node
Using default tag: latest
latest: Pulling from library/node
75a822cd7888: Already exists
57de64c72267: Pull complete
4306be1e8943: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
d8daeae79eeb: Pull complete
Digest: sha256:356097f9b8059a29476134b58249a123211153faf823a5c3d5370e3518219a89
Status: Downloaded newer image for node:latest

2.启动node镜像，并安装cnpm ,然后打包产生新的镜像，命名为nodecnpm

docker run -it node /bin/bash
root@f04b5e564560:/#

npm install cnpm -g --registry=https://registry.npm.taobao.org

docker commit f04b5e564560 nodecnpm
sha256:82d2fb5e840f82177446ab6fc9df8f5c361662d95718292023f14714b48fb326

exit

3.创建本地目录,并编写nodejs应用


mkdir -p ~/software/docker/node/docker_node

touch package.json
mac:docker_node mac$ vi package.json

在~/software/docker/node/docker_node目录下创建package.json文件，并如下内容


{

    "name": "docker_node",
    "version": "0.0.1",
    "dependencies": {
        "express":"4.14.0",
        "redis":"2.6.3"
    },
    "engines": {
        "node":">=0.10.0"
    }
}

新建app.js，并添加下面内容


var express = require('express');

var redis = require('redis');

var app = express();

var redisHost = process.env['REDIS_PORT_6379_TCP_ADDR'];

var redisPort = process.env['REDIS_PORT_6379_TCP_PORT'];

var redisClient = redis.createClient(redisPort,redisHost);

app.get('/',function(req,res){

    console.log('get request');

    redisClient.get('access_count',function(err,countNum){
        if(err){
            return res.send('get access count error');
        }

        if(!countNum){
            countNum=1;
        }else{
            countNum = parseInt(countNum)+1;
        }

        redisClient.set('access_count',countNum,function(err){
            if(err){
                return res.send('set access count error');
            }
            res.send(countNum.toString())
        });

    });
});


app.listen(8000);

4.运行npm容器，并将上面编写的代码映射到容器中，然后安装一下相关依赖


docker run --rm -it -v /Users/mac/software/docker/node/docker_node/:/var/node/docker_node -w /var/node/docker_node/ nodecnpm cnpm install

-w：表示命令的当前目录

5.运行容器中的app.js

docker run -v /Users/mac/software/docker/node/docker_node/:/var/node/docker_node -p 8000:8000 -d  --link redis-server:redis -w /var/node/docker_node/ nodecnpm node app.js
3f8e4b4531a8901ab1ead9e53f364a45957c4bf38d9a326d493c65907d5e7470

–link：链接redis容器

6.刷新页面http://localhost:8000可以看到数字的变换

代码地址

https://github.com/CodeToSurvive1/docker_node

docker搭建jenkins

2016-12-25T13:52:00+08:00

1.安装jenkins

docker pull jenkins
Using default tag: latest
latest: Pulling from library/jenkins

75a822cd7888: Already exists
57de64c72267: Already exists
4306be1e8943: Already exists
1e6944bfb718: Pull complete
3521f2f45ed2: Pull complete
9c2f0d9b5f90: Pull complete
8cedcf6d2527: Pull complete
31f99da7583d: Pull complete
2b1a4e887f7e: Pull complete
ec60c180ba47: Pull complete
b8c5058508d7: Pull complete
5f6956ea8053: Pull complete
11322c12182f: Pull complete
012931252722: Pull complete
1414621e5a2e: Pull complete
bcff974135a1: Pull complete
27a8f00c6f2a: Pull complete
3b2fdd5946f5: Pull complete
e5af99d78c8f: Pull complete
Digest: sha256:e635156257ec0e64f03607a89aac7383f192f0aebc7f70cba1e35461b76c1fad
Status: Downloaded newer image for jenkins:latest
mac:~ mac$

2.创建本地目录，映射到容器中来保存jenkins的数据

mkdir -p ~/software/docker/jenkins/jenkins_home

docker run -d --name jenkins -p 49001:8080 -v /Users/mac/software/docker/jenkins/jenkins_home:/var/jenkins_home jenkins
2553a256e0d4817ee4328718aeeaa0370fedd64cfbd8d7205120217f730be316

3.稍等一会后打开本地的http://localhost:49001，会看到下面的页面

4.进入到docker容器中查看文件中的密码

docker exec -it jenkins /bin/bash

jenkins@2553a256e0d4:/$ cat /var/jenkins_home/secrets/initialAdminPassword
da702c468f9642b3a85ac25220f6165e

5.下一步安装插件

经过漫长的等待下载后

输入用户名及密码后并设置为管理员

6.进入到系统管理-管理插件-已安装，确保下面两款插件已经安装，如果没有安装，则进行安装并重启jenkins

7.进入系统管理-系统设置来对刚刚安装的插件进行相关配置

这里设置自己的mac电脑远程ssh

test测试成功

8.点击新建->输入项目名称->选择自由软件风格

这样，到这一步之后其实就可以保存进行构建测试了，只是需要模拟真实的发布版本的情况，一般我们的手工操作如下：

a.从svn或者git上下载源代码 		
b.将本地代码打包，并排除不需要的文件，比如.git文件  		
c.将打包好的程序包上传到远程服务器中  	
d.停止远程服务器的服务，删除服务器上的代码，解压新的代码 	
e.通过新的package来安装依赖，然后重启服务  

针对上面的步骤，我们需要做的指令如下

rm -rf /var/jenkins_home/workspace/docker_node.tar.gz
tar -zcvf /var/jenkins_home/workspace/docker_node/docker_node.tar.gz -C /var/jenkins_home/workspace/docker_node . --exclude="*.git"

9.选择构建->execute shell

将上面编写的删除打包好的压缩包及重新打包的代码粘贴到上面的地方

这样在每次构建的时候，都会在github下载新代码之后，重新压缩打包

10.继续选择添加构建步骤->send files or execute commands over ssh

在execute command 中粘贴如下代码

rm -rf /Users/mac/software/docker/node/docker_node/README.md

rm -rf /Users/mac/software/docker/node/docker_node/package.json

rm -rf /Users/mac/software/docker/node/docker_node/app.js

tar -zxvf /Users/mac/software/docker/node/docker_node/docker_node.tar.gz -C /Users/mac/software/docker/node/docker_node

docker run --rm -v /Users/mac/software/docker/node/docker_node/:/var/node/docker_node -w /var/node/docker_node/ nodecnpm cnpm install

docker run -d --name 'dockercount' -p 8000:8000 -v /Users/mac/software/docker/node/docker_node:/var/node/docker_node --link redis-server:redis -w /var/node/docker_node cnpm node app.js

rm -rf /Users/mac/software/docker/node/docker_node/docker_node.tar.gz

ubuntu安装docker私有服务器

2016-12-25T13:52:00+08:00

安装registry

1.安装之前请先安装docker

ubuntu下安装docker

2.安装并启动registry


docker run -d -p 5000:5000 --restart=always --name registry registry:2

3.下载自己的镜像并进行相关修改

sudo docker pull node
Using default tag: latest
latest: Pulling from library/node
75a822cd7888: Pull complete 
57de64c72267: Pull complete 
4306be1e8943: Pull complete 
871436ab7225: Pull complete 
0110c26a367a: Pull complete 
1f04fe713f1b: Pull complete 
d8daeae79eeb: Pull complete 
Digest: sha256:356097f9b8059a29476134b58249a123211153faf823a5c3d5370e3518219a89
Status: Downloaded newer image for node:latest

sudo docker run -it node /bin/bash
root@6a75f7e56cfe:/# 


npm install pm2 -g

npm install cnpm -g --registry=https://registry.npm.taobao.org


root@6a75f7e56cfe:/# exit
exit

4.找到刚才的容器

ubuntu@ubuntu:~$ sudo docker ps -al
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
6a75f7e56cfe        node                "/bin/bash"         3 minutes ago       Exited (0) 47 seconds ago                       dreamy_cori

5.提交容器

sudo docker commit 6a75f7e56cfe nodepm2
sha256:11031297762fccbb39181f237e5d355c70e1630d064990b027ee5f7cc218a848

检查镜像是否生成  
ubuntu@ubuntu:~$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nodepm2             latest              11031297762f        4 seconds ago       713.7 MB
node                latest              d1699fb7d2bf        3 days ago          659.8 MB
registry            2                   c9bd19d022f6        9 weeks ago         33.3 MB
hello-world         latest              c54a2cc56cbb        5 months ago        1.848 kB

6.将nodepm2镜像打标签，并提交到registry中

sudo docker tag nodepm2 192.168.4.138:5000/nodepm2
ubuntu@ubuntu:~$ sudo docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
192.168.4.138:5000/nodepm2   latest              b26536a28863        32 seconds ago      713.7 MB
nodepm2                      latest              b26536a28863        32 seconds ago      713.7 MB
node                         latest              d1699fb7d2bf        3 days ago          659.8 MB
registry                     2                   c9bd19d022f6        9 weeks ago         33.3 MB
hello-world                  latest              c54a2cc56cbb        5 months ago        1.848 kB


ubuntu@ubuntu:~$ sudo docker push 192.168.4.138:5000/nodepm2
The push refers to a repository [192.168.4.138:5000/nodepm2]
Get https://192.168.4.138:5000/v1/_ping: http: server gave HTTP response to HTTPS client

7.上面push的时候报错，修改配置文件,并重新启动docker

sudo vi /etc/default/docker

添加下面一行    

DOCKER_OPTS="$DOCKER_OPTS --insecure-registry=192.168.4.138:5000"

sudo service docker restart

这里使用的Ubuntu系统，但是配置后没起作用，需要再进行下面配置

sudo vi /etc/docker/daemon.json

添加下面内容  

{"registry-mirrors": ["http://6f89f03a.m.daocloud.io"],"insecure-registries":["192.168.4.138:5000"] }

然后重启docker    

sudo service docker stop

sudo service docker start

可以在链接https://github.com/docker/distribution/issues/1874#issuecomment-237194314中找到相关信息

8.重新执行命令

sudo docker push 192.168.4.138:5000/nodepm2

The push refers to a repository [192.168.4.138:5000/nodepm2]
ac790c708fef: Pushed 
a05ad5eac50b: Pushed 
604c78617f34: Pushed 
fa18e5ffd316: Pushed 
0a5e2b2ddeaa: Pushed 
53c779688d06: Pushed 
60a0858edcd5: Pushed 
b6ca02dfe5e6: Pushed 
latest: digest: sha256:002b64efd563616feb27f9ec41494a2329a3415af04f82536d3172f60cde4815 size: 2007

9.直接在自己的mac机器上测试下在192.168.4.138:5000搭建的是否成功

docker pull 192.168.4.138:5000/nodepm2
Using default tag: latest
Error response from daemon: Get https://192.168.4.138:5000/v1/_ping: http: server gave HTTP response to HTTPS client

看来跟上面的错误是一样的，由于mac的配置跟ubuntu配置不太一样，这里直接截图

配置完成后重启docker

docker pull 192.168.4.138:5000/nodepm2
Using default tag: latest
latest: Pulling from nodepm2
75a822cd7888: Already exists
57de64c72267: Already exists
4306be1e8943: Already exists
871436ab7225: Already exists
0110c26a367a: Already exists
1f04fe713f1b: Already exists
d8daeae79eeb: Already exists
e74e30bbf941: Pull complete
Digest: sha256:002b64efd563616feb27f9ec41494a2329a3415af04f82536d3172f60cde4815
Status: Downloaded newer image for 192.168.4.138:5000/nodepm2:latest

注意：

想要从私有服务器中下载的话，私有服务器跟下载节点都需要配置上面的insecure-registries

参考文档

https://github.com/docker/docker.github.io/blob/master/registry/deploying.md

https://github.com/docker/distribution/issues/1874#issuecomment-237194314

ubuntu安装docker

2016-12-25T12:52:00+08:00

ubuntu安装docker

1.更新系统源

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates

2.添加GPG key

sudo apt-key adv \
               --keyserver hkp://ha.pool.sks-keyservers.net:80 \
               --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

执行结果为

Executing: /tmp/tmp.Jek7P0R86h/gpg.1.sh --keyserver
hkp://ha.pool.sks-keyservers.net:80
--recv-keys
58118E89F3A912897C070ADBF76221572C52609D
gpg: requesting key 2C52609D from hkp server ha.pool.sks-keyservers.net
gpg: key 2C52609D: public key "Docker Release Tool (releasedocker) <docker@docker.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

3.根据系统版本找到下面对应的链接,这里使用的是Ubuntu 16.04版本，也就是最后一个

Ubuntu version	Repository
Precise 12.04 (LTS)	deb https://apt.dockerproject.org/repo ubuntu-precise main
Trusty 14.04 (LTS)	deb https://apt.dockerproject.org/repo ubuntu-trusty main
Wily 15.10	deb https://apt.dockerproject.org/repo ubuntu-wily main
Xenial 16.04 (LTS)	deb https://apt.dockerproject.org/repo ubuntu-xenial main

4.将下面命令的repo替换为3中的链接

echo "<REPO>" | sudo tee /etc/apt/sources.list.d/docker.list

修改为

echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list

5.更新源并进行安装


sudo apt-get update

如果这一步骤很慢，则可以修改第4步骤中的文件内容为阿里镜像的地址


deb http://mirrors.aliyun.com/docker-engine/apt/repo/  ubuntu-xenial main

6.执行下面命令，查看可用版本


apt-cache policy docker-engine

7.安装docker


sudo apt-get install docker-engine

8.启动docker服务


sudo service docker start

9.查看运行状态

sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: e
   Active: active (running) since Sat 2016-12-24 19:02:21 PST; 42s ago
     Docs: https://docs.docker.com
 Main PID: 10630 (dockerd)
    Tasks: 17
   Memory: 34.5M
      CPU: 274ms
   CGroup: /system.slice/docker.service
           ├─10630 /usr/bin/dockerd -H fd://
           └─10645 docker-containerd -l unix:///var/run/docker/libcontainerd/doc

Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.578076079-08:00
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.578862967-08:00
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.632339397-08:00
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.722353068-08:00
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.865416140-08:00
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.865672519-08:00
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.865818794-08:00
Dec 24 19:02:21 ubuntu systemd[1]: Started Docker Application Container Engine.
Dec 24 19:02:21 ubuntu dockerd[10630]: time="2016-12-24T19:02:21.988106862-08:00
Dec 24 19:02:56 ubuntu systemd[1]: Started Docker Application Container Engine.

10.运行hello-world例子


sudo docker run hello-world

这一步的操作首先会去docker官方下载hello-world镜像，然后运行

如果这一步也下载缓慢，则执行下面命令，使用daocloud加速器,然后重启docker

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://6f89f03a.m.daocloud.io

sudo systemctl restart docker.service

11.验证是否成功

ubuntu@ubuntu:/etc/apt$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
hello-world         latest              c54a2cc56cbb        5 months ago        1.848 kB


ubuntu@ubuntu:/etc/apt$ sudo docker ps -al
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
c1e66fe1d26a        hello-world         "/hello"            2 minutes ago       Exited (0) 2 minutes ago                       pensive_shirley

参考文档

https://docs.docker.com/engine/installation/linux/ubuntulinux/

cdh环境搭建(四)安装cdh的相关服务

2016-12-08T21:52:00+08:00

cdh环境搭建(四)安装cdh的相关服务

1.添加zookeeper服务

这里选择三台主机作为zookeeper的集群

启动命令测试

 cd /opt/cloudera/parcels/CDH/bin/
[cdh@centos3 bin]$ ./zookeeper-client 

Connecting to localhost:2181
2016-12-18 23:24:13,680 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.3.6--1, built on 07/28/2015 22:12 GMT
2016-12-18 23:24:13,686 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=centos3.com
2016-12-18 23:24:13,686 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_67
2016-12-18 23:24:13,688 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2016-12-18 23:24:13,688 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre
2016-12-18 23:24:13,688 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../build/classes:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../build/lib/*.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.3.6.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/zookeeper.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/zookeeper-3.4.5-cdh5.3.6.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin/../lib/zookeeper/lib/jline-0.9.94.jar
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.32-358.el6.x86_64
2016-12-18 23:24:13,689 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=cdh
2016-12-18 23:24:13,690 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/cdh
2016-12-18 23:24:13,690 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/cloudera/parcels/CDH-5.3.6-1.cdh5.3.6.p0.11/bin
2016-12-18 23:24:13,691 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@52dce479
Welcome to ZooKeeper!
2016-12-18 23:24:13,747 [myid:] - INFO  [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost.localdomain/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-12-18 23:24:13,759 [myid:] - INFO  [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /127.0.0.1:46301, server: localhost.localdomain/127.0.0.1:2181
2016-12-18 23:24:13,777 [myid:] - INFO  [main-SendThread(localhost.localdomain:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost.localdomain/127.0.0.1:2181, sessionid = 0x1591283a7e30005, negotiated timeout = 30000
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

create /test 'test-data'
Created /test
[zk: localhost:2181(CONNECTED) 4] ls /
[test, zookeeper]
[zk: localhost:2181(CONNECTED) 5] ls /test/

2.安装hdfs

3.安装yarn

4.安装hive

cdh环境搭建(三)yum离线安装cdh

2016-12-08T12:52:00+08:00

cdh环境搭建(三)yum离线安装cdh

1.下载cdh软件包,链接

http://archive.cloudera.com/cdh5/parcels/5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel
http://archive.cloudera.com/cdh5/parcels/5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1

2.选择管理->设置

3.将1中下载的parcel文件放到目录下/opt/cloudera/parcel-repo

sudo mv * /opt/cloudera/parcel-repo/

 cd /opt/cloudera/parcel-repo/
[cdh@centos2 parcel-repo]$ ll
总用量 1473856
-rwxrwxr-x 1 cdh cdh 1509217191 12月  8 20:54 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel
-rwxrwxr-x 1 cdh cdh         41 12月  8 20:54 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1
[cdh@centos2 parcel-repo]$ 

4.将CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1该名为CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha

sudo mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha
[cdh@centos2 parcel-repo]$ ll
总用量 1473856
-rwxrwxr-x 1 cdh cdh 1509217191 12月  8 20:54 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel
-rwxrwxr-x 1 cdh cdh         41 12月  8 20:54 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha

5.将CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel重命名，因为多了一个parcel后缀

sudo mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel

6.重启cm

sudo service cloudera-scm-server status
cloudera-scm-server (pid  5623) is running...
[cdh@centos2 parcel-repo]$ sudo service cloudera-scm-server restart
Stopping cloudera-scm-server:                              [  OK  ]
Starting cloudera-scm-server:                              [  OK  ]

7.修改安装cm主机上配置的yum源为本地的服务器地址,上篇文章中配置的centos2

 sudo vi cloudera-manager.repo

[cloudera-manager]
name=Cloudera Manager
baseurl=http://centos2.com/cm5/redhat/6/x86_64/cm/5/
gpgkey = http://centos2.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck=1

8.复制该文件到集群的其他节点centos1和centos3

$ sudo rm -rf *
[sudo] password for cdh: 
[cdh@centos3 yum.repos.d]$ sudo scp cdh@centos2.com:/etc/yum.repos.d/cloudera-manager.repo ./
The authenticity of host 'centos2.com (192.168.4.132)' can't be established.
RSA key fingerprint is 0a:28:51:a8:23:f3:7b:84:8a:c6:82:de:84:a6:9c:bb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'centos2.com,192.168.4.132' (RSA) to the list of known hosts.
cdh@centos2.com's password: 
cloudera-manager.repo                                                                                                                              100%  177     0.2KB/s   00:00    
[cdh@centos3 yum.repos.d]$ 

9.配置cm中的相关配置

9.1添加主机

点击搜索

9.2修改cm agent的yum的安装地址，也就是yum源中配置的地址

9.3选择sdk安装

继续

9.4修改用户名及密码

9.5进入安装页面

报错

修改三台主机的sudoers文件


cdh     ALL=(ALL)       NOPASSWD: ALL

9.6重新安装

9.7修改相关三个错误信息

9.7.1

检查 /etc/hosts 时发现以下错误... 
主机名称 centos2.com 不是 centos2.com 上的 /etc/hosts 中的地址 192.168.4.132 的第一个匹配。相反，archive.cloudera.com 是第一个匹配。FQDN 必须是相应 IP 的 /etc/hosts 中的第一个条目。
在 centos2.com 上的 /etc/hosts 中，IP 192.168.4.132 出现多次。给定 IP 应只列出一次。


[cdh@centos2 ~]$ sudo vi /etc/hosts

删除下面信息

192.168.4.132   archive.cloudera.com

9.7.2

Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 60。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf 以在重启后保存该设置。您可以继续进行安装，但可能会遇到问题，Cloudera Manager 报告您的主机由于交换运行状况不佳。以下主机受到影响： 
centos[1-3].com

在三台主机上都执行下面信息

sudo sysctl -w vm.swappiness=0
vm.swappiness = 0

su

echo "vm.swappiness=0" >> /etc/sysctl.conf

exit

9.7.3

已启用“透明大页面”，它可能会导致重大的性能问题。版本为“CentOS release 6.4 (Final)”且发行版为“2.6.32-358.el6.x86_64”的 Kernel 已将 enabled 设置为“[always] never”，并将 defrag 设置为“[always] never”。请运行“echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag”以禁用此设置，然后将同一命令添加到一个 init 脚本中，如 /etc/rc.local，这样当系统重启时就会予以设置。或者，升级到 RHEL 6.5 或更新版本，它们不存在此错误。将会影响到以下主机： 
centos[1-3].com

分别在三台机器上切换到root用户执行下面命令

su

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

exit

并分别将命令添加到/etc/rc.local文件中

sudo vi /etc/rc.local 

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

重新检测后全部通过，点击完成

10.添加cloudera management service

这里都设置centos2作为角色管理者

设置数据库

注意这里的数据库主机名称端口前面要添加上主机地址，不然测试连接不通过记住密码gCn0ZFl3JD

点击继续

点击详细信息可以查看启动相关进程

安装成功

11.查看主页面

cdh环境搭建(二)yum离线安装cm

2016-12-08T08:52:00+08:00

cdh环境搭建(二)软件安装

CM软件安装,这里使用5.3.6版本

1.下载bin文件，http://archive.cloudera.com/cm5/installer/5.3.6/ ,一个文件

2.下载rpm相关依赖包，http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/,总共7个文件

下载完成后目录结构如下所示

3.下载依赖包，数据库postgresq及repo-libs

4.搭建本地yum源

4.1安装repo-libs软件包

sudo rpm -ivh ./*.rpm
warning: ./createrepo-0.9.9-18.el6.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing...                ########################################### [100%]
   1:deltarpm               ########################################### [ 33%]
   2:python-deltarpm        ########################################### [ 67%]
   3:createrepo             ########################################### [100%]

4.2启动Apache服务器

[cdh@centos Desktop]$ rpm -qa | grep httpd
httpd-tools-2.2.15-26.el6.centos.x86_64
httpd-2.2.15-26.el6.centos.x86_64
[cdh@centos Desktop]$ service httpd status
httpd is stopped
[cdh@centos Desktop]$ sudo service httpd start
[sudo] password for cdh: 
Starting httpd:                                            [  OK  ]
[cdh@centos Desktop]$ sudo chkconfig httpd on
[cdh@centos Desktop]$ sudo chkconfig --list | grep httpd
httpd          	0:off	1:off	2:on	3:on	4:on	5:on	6:off

4.3在apache网站目录下创建新的目录，用来存放软件

cd /var/www/html/

[cdh@centos html]$ pwd
/var/www/html
[cdh@centos html]$ sudo mkdir -p cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/

访问centos中的ip地址可以看到开启的apache服务器,http://192.168.4.129/cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/

4.4上传文件到上面的新目录下,这里使用scp拷贝，也可以使用上传工具

[cdh@centos html]$ sudo service sshd start
Generating SSH1 RSA host key:                              [  OK  ]
Generating SSH2 RSA host key:                              [  OK  ]
Generating SSH2 DSA host key:                              [  OK  ]
Starting sshd:                                             [  OK  ]
[cdh@centos html]$ sudo chkconfig sshd on
[cdh@centos html]$ sudo chkconfig --list | grep sshd
sshd           	0:off	1:off	2:on	3:on	4:on	5:on	6:off

将cm软件先上传到centos根目录下

 scp software/bigdata/cloudera/cm-5.3.6/* cdh@192.168.4.129:~/Desktop/
cdh@192.168.4.129's password: 
cloudera-manager-agent-5.3.6-1.cm536.p0.244.e 100% 3896KB   3.8MB/s   00:00    
cloudera-manager-daemons-5.3.6-1.cm536.p0.244 100%  476MB  36.6MB/s   00:13    
cloudera-manager-installer.bin                100%  502KB 502.2KB/s   00:00    
cloudera-manager-server-5.3.6-1.cm536.p0.244. 100% 7852     7.7KB/s   00:00    
cloudera-manager-server-db-2-5.3.6-1.cm536.p0 100% 9884     9.7KB/s   00:00    
enterprise-debuginfo-5.3.6-1.cm536.p0.244.el6 100%  677KB 676.8KB/s   00:00    
jdk-6u31-linux-amd64.rpm                      100%   68MB  34.0MB/s   00:02    
oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm   100%  135MB 135.5MB/s   00:01  

将上传到cenos桌面上的rpm软件包全部放到apache新建的目录下

cd cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/
[cdh@centos x86_64]$ ls
[cdh@centos x86_64]$ sudo mv ~/Desktop/*.rpm ./
[cdh@centos x86_64]$ ls -al
total 700576
drwxr-xr-x 2 root root      4096 Dec  7 22:54 .
drwxr-xr-x 3 root root      4096 Dec  7 22:44 ..
-rwxrwxr-x 1 cdh  cdh    3989520 Dec  7 22:53 cloudera-manager-agent-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rwxrwxr-x 1 cdh  cdh  499418684 Dec  7 22:53 cloudera-manager-daemons-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rwxrwxr-x 1 cdh  cdh       7852 Dec  7 22:53 cloudera-manager-server-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rwxrwxr-x 1 cdh  cdh       9884 Dec  7 22:53 cloudera-manager-server-db-2-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rwxrwxr-x 1 cdh  cdh     693024 Dec  7 22:53 enterprise-debuginfo-5.3.6-1.cm536.p0.244.el6.x86_64.rpm
-rwxrwxr-x 1 cdh  cdh   71204325 Dec  7 22:53 jdk-6u31-linux-amd64.rpm
-rwxrwxr-x 1 cdh  cdh  142039186 Dec  7 22:53 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

继续访问页面查看是否成功

在/var/www/html/cm5/redhat/6/x86_64/cm/5目录下执行命令

cd /var/www/html/cm5/redhat/6/x86_64/cm/5

[cdh@centos2 5]$ sudo createrepo .
Spawning worker 0 with 6 pkgs
Workers Finished
Gathering worker results

Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

4.5新建yum源文件cloudera-manager.repo

cd /etc/yum.repos.d/
[cdh@centos yum.repos.d]$ pwd
/etc/yum.repos.d   		

这里将所有的repo源先移走或者直接删掉

sudo mv CentOS-* ~/Desktop/

新建repo源

sudo vi cloudera-manager.repo

添加下面内容

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64                  
name=Cloudera Manager
baseurl=http://192.168.4.129/cm5/redhat/6/x86_64/cm/5.3.6/
gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera    
gpgcheck = 0          

4.6安装postgresq

上传postgresq到centos中的cdh用户目录下的software下

mac:cloudera mac$ scp -r postgresq-libs/ cdh@192.168.4.129:/home/cdh/software
postgresql-8.4.18-1.el6_4.x86_64.rpm          100% 2838KB   2.8MB/s   00:00    
postgresql-libs-8.4.18-1.el6_4.x86_64.rpm     100%  201KB 200.9KB/s   00:00    
postgresql-server-8.4.18-1.el6_4.x86_64.rpm   100% 3518KB   3.4MB/s   00:00  

安装postgresql

sudo rpm -ivh *.rpm
warning: postgresql-8.4.18-1.el6_4.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Preparing...                ########################################### [100%]
   1:postgresql-libs        ########################################### [ 33%]
   2:postgresql             ########################################### [ 67%]
   3:postgresql-server      ########################################### [100%]

4.7修改本机的ip地址与域名绑定,将archive.cloudera.com域名绑定本地ip，这样就能通过本地搭建的80服务器httpd服务器访问下载了，如果不添加该步骤，则总是联网下载

sudo vi /etc/hosts

192.168.4.130   archive.cloudera.com

4.8执行cloudera-manager-installer.bin文件

必须要用root权限


sudo ./cloudera-manager-installer.bin

4.9执行完毕后会看到提示信息，访问7180端口看到下面界面

用admin/admin登录后看到页面

选择免费版本后，继续

cdh环境搭建(一)环境准备

2016-12-07T08:52:00+08:00

cdh环境搭建(一)环境准备

环境准备

1.卸载系统自带的jdk

查询系统自带jdk

rpm -qa | grep java
java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
tzdata-java-2012j-1.el6.noarch
java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
libreoffice-javafilter-3.4.5.2-16.1.el6_3.x86_64
java_cup-0.10k-5.el6.x86_64

卸载所有自带的jdk

sudo rpm -e --nodeps java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64 java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64 tzdata-java-2012j-1.el6.noarch java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 libreoffice-javafilter-3.4.5.2-16.1.el6_3.x86_64 java_cup-0.10k-5.el6.x86_64
[sudo] password for centos: 

2.关闭防火墙,并设置重启后也是关闭

$ sudo service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[centos@centos Desktop]$ sudo service iptables status
iptables: Firewall is not running.
[centos@centos Desktop]$ sudo chkconfig  iptables off


chkconfig --list | grep iptables
iptables       	0:off	1:off	2:off	3:off	4:off	5:off	6:off

3.添加用户及密码

切换到root用户，添加用户cdh，密码123456

[root@centos Desktop]# useradd cdh
[root@centos Desktop]# passwd cdh
Changing password for user cdh.
New password: 
BAD PASSWORD: it is too simplistic/systematic
BAD PASSWORD: is too simple
Retype new password: 
passwd: all authentication tokens updated successfully.

4.禁用ipv6

[root@centos Desktop]# echo "alia net-pf-10 off" >> /etc/modprobe.d/dist.conf 
[root@centos Desktop]# echo "alia ipv6 off" >> /etc/modprobe.d/dist.conf 
[root@centos Desktop]# tail -f /etc/modprobe.d/dist.conf 
alias rpc_svc_gss_pipefs sunrpc

install eth1394 /bin/true

install snd-emu10k1 /sbin/modprobe --ignore-install snd-emu10k1 && /sbin/modprobe snd-emu10k1-synth

alias gre0 ip_gre
alias char-major-89-* i2c-dev
alia net-pf-10 off
alia ipv6 off

5.关闭selinux


vi /etc/sysconfig/selinux

修改为


SELINUX=disabled

6.修改用户打开最大进程数及最大文件句柄数

首先查看系统的默认配置

[root@centos Desktop]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7689
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7689
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

ulimit -u
7689

修改


vi /etc/security/limits.conf

*       soft    nofile  65535
*       hard    nofile  65535
*       soft    nproc   32000
*       hard    nproc   32000

7.修改sudoer(非必须)

8.修改主机名

9.修改主机名与ip映射关系

复制虚拟机后进行相关配置

1.复制之前的centos虚拟机

2.将centos复制两次后如下图所示

3.分别修改主机名称为centos2.com及centos3.com

4.集群时间同步，这里使用centos作为主服务器，其他两台的时间根据centos时间进行同步，软件使用ntp进行同步时间修改第一台centos的ntp配置文件


sudo vi /etc/ntp.conf

总共需要修改三处

将下面一行取消注释, 并将restrict后的网段修改为自己的ip网段



restrict 192.168.4.0 mask 255.255.255.0 nomodify notrap

注释掉下面三行

#server 0.centos.pool.ntp.org
#server 1.centos.pool.ntp.org
#server 2.centos.pool.ntp.org

将下面两行释放开

server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10

修改centos的ntp服务器配置


vi /etc/sysconfig/ntpd

在该文件开始添加下面一行配置


SYNC_HWCLOCK=yes

启动服务端

[cdh@centos Desktop]$ sudo service ntpd status
ntpd is stopped
[cdh@centos Desktop]$ sudo service ntpd start
Starting ntpd:                                             [  OK  ]
[cdh@centos Desktop]$ sudo service ntpd status
ntpd (pid  3470) is running...
[cdh@centos Desktop]$ sudo chkconfig ntpd on
[cdh@centos Desktop]$ sudo chkconfig --list | grep ntp
ntpd           	0:off	1:off	2:on	3:on	4:on	5:on	6:off
ntpdate        	0:off	1:off	2:off	3:off	4:off	5:off	6:off

在centos2和centos3客户端中添加linux的contable定时任务来同步机器时间

 su
Password: 
[root@centos3 Desktop]# crontab -e

添加如下内容进行时间同步,192.168.4.129为centos时间服务器的ip地址


0-59/10 * * * * /usr/sbin/ntpdate 192.168.4.129

直接测试下，同步时间效果,会报错，这是由于server启动后需要过几分钟才能在从节点上进行同步时间

[root@centos3 Desktop]# /usr/sbin/ntpdate 192.168.4.129

 7 Dec 04:51:41 ntpdate[3580]: no server suitable for synchronization found

过几分钟后重新尝试,同步成功

[root@centos3 Desktop]# /usr/sbin/ntpdate 192.168.4.129
 6 Dec 22:56:59 ntpdate[3658]: adjust time server 192.168.4.129 offset 0.001352 sec
[root@centos3 Desktop]# date
Tue Dec  6 22:58:25 CST 2016

vmware fusion安装centos虚拟机

2016-12-04T08:52:00+08:00

1.下载vmware fusion并安装,密码: qifu

2.新建虚拟机

3.虚拟机设置完毕后，进入安装页面

4.安装完毕后重启

5.修改主机名称

首先切换到root用户

su


vi /etc/sysconfig/network

修改hostname

然后重启reboot，重启后查看主机名称是否修改成功

hostname

centos.com

6.添加sudo操作

默认是没有sudo操作的，如下图所示

同样先切换到root用户

su


vi /etc/sudoers

添加该用户相关配置


centos      ALL=(ALL)   ALL

保存后退出root用户再进行测试

tomcat性能调优

2016-11-19T22:18:00+08:00

1.修改connector的protocol模式

对于不同版本的tomcat，该值也不同，比如

tomcat6，启动tomcat，测试下来发现默认是Http11Protocol：

org.apache.coyote.http11.Http11Protocol - same as HTTP/1.1，阻塞式IO		
org.apache.coyote.http11.Http11NioProtocol - non blocking Java connector，非阻塞IO 		
org.apache.coyote.http11.Http11AprProtocol - the APR connector.		

启动tomcat6后查看后台日志，

信息: Deploying web application directory ROOT
十一月 19, 2016 10:41:28 下午 org.apache.coyote.http11.Http11Protocol start
信息: Starting Coyote HTTP/1.1 on http-8080
十一月 19, 2016 10:41:28 下午 org.apache.jk.common.ChannelSocket init
信息: JK: ajp13 listening on /0.0.0.0:8009
十一月 19, 2016 10:41:28 下午 org.apache.jk.server.JkMain start
信息: Jk running ID=0 time=0/19  config=null
十一月 19, 2016 10:41:28 下午 org.apache.catalina.startup.Catalina start
信息: Server startup in 716 ms

发现默认是Http11Protocol，即为阻塞式io，因此这里是可以进行优化的

修改conf/server.xml文件中的配置

<Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />

为

 <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol" 
               connectionTimeout="20000" 
               redirectPort="8443" />

重新启动tomcat，查看日志

信息: Deploying web application directory ROOT
十一月 19, 2016 10:44:31 下午 org.apache.coyote.http11.Http11NioProtocol start
信息: Starting Coyote HTTP/1.1 on http-8080
十一月 19, 2016 10:44:31 下午 org.apache.jk.common.ChannelSocket init
信息: JK: ajp13 listening on /0.0.0.0:8009
十一月 19, 2016 10:44:31 下午 org.apache.jk.server.JkMain start
信息: Jk running ID=0 time=0/28  config=null
十一月 19, 2016 10:44:31 下午 org.apache.catalina.startup.Catalina start
信息: Server startup in 710 ms

这时候发现配置ok，使用了非阻塞式io

注意：不同版本的tomcat可能不同

tomcat7也是使用了bio的代理

信息: Deployment of web application directory /Users/mac/software/tomcat/apache-tomcat-7.0.73/webapps/ROOT has finished in 42 ms
十一月 19, 2016 10:45:50 下午 org.apache.coyote.AbstractProtocol start
信息: Starting ProtocolHandler ["http-bio-8080"]
十一月 19, 2016 10:45:50 下午 org.apache.coyote.AbstractProtocol start
信息: Starting ProtocolHandler ["ajp-bio-8009"]
十一月 19, 2016 10:45:50 下午 org.apache.catalina.startup.Catalina start
信息: Server startup in 874 ms

因此tomcat7也可以跟上面的tomcat6一样进行配置优化

tomcat8就不同了

19-Nov-2016 22:50:03.085 信息 [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [http-nio-8080]
19-Nov-2016 22:50:03.094 信息 [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [ajp-nio-8009]
19-Nov-2016 22:50:03.095 信息 [main] org.apache.catalina.startup.Catalina.start Server startup in 861 ms

默认就使用了nio的代理

查看官方文档发现，tomcat8中只有下面三中，而且默认就是第一种

org.apache.coyote.http11.Http11NioProtocol - non blocking Java NIO connector
org.apache.coyote.http11.Http11Nio2Protocol - non blocking Java NIO2 connector
org.apache.coyote.http11.Http11AprProtocol - the APR/native connector. 

2.修改线程池,默认是不用配置的，tomcat中有自己的内部设置，也可以通过server.xml手动指定

首先通过配置tomcat的管理员界面，查看默认值

tomcat6,直接配置tomcat-users.xml文件

<role rolename="tomcat"/>
<role rolename="role1"/>
<role rolename="manager-gui"/>

<user username="tomcat" password="tomcat" roles="tomcat,role1,manager-gui"/>

输入tomat/tomcat进入管理页面看到,tomcat的默认线程数量，可以从连接看到相关默认配置，http://tomcat.apache.org/tomcat-8.0-doc/config/executor.html

修改server.xml文件

放开注释

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>

并修改

<Connector port="8080" executor="tomcatThreadPool" protocol="org.apache.coyote.http11.Http11NioProtocol" 
               connectionTimeout="20000" 
               redirectPort="8443" />

重启tomcat

但是不知道为何，web页面上没有显示修改后的线程数量，该问题待以后研究

3.连接器优化

enableLookups   是否查询远程访问ip的dns，默认为false		
maxParameterCount	最大参数数量，get或者post的键值对个数，默认最大10000，多余的会被忽略.手动改为小于0则不限制 	
maxPostSize		post请求时候FORM表单的最大字节数，默认2M，配置为小于0，则不限制 	
protocol    	上面说过，默认是BIO		
redirectport  	安全转发端口 		
URIEncoding		解决乱码问题，默认是ISO-8859-1 				
acceptCount		最大队列数量,默认100，超过之后直接拒绝 		
compression 	开启压缩，GZIP，off/on/force来设置,默认是off，一般不用，我们直接在nginx上进行配置 	
connectionUploadTimeout 	指定上传超时时间，disableUploadTimeout设置为false时候有用
disableUploadTimeout 	是否禁用上传超时		
executor 		指定线程池	
maxConnections 		最大连接数 	
maxThreads 		tomcat能够创建的最大线程数，默认200 	
minSpareThreads 	保证最小空余线程数量 		

4.禁用ajp,因为一般都用nginx服务器直接返回静态文件

直接将server.xml中的下面配置注释掉即可

 <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

重启后日志中没有ajp相关

信息: Deployment of web application directory /Users/mac/software/tomcat/apache-tomcat-7.0.73/webapps/ROOT has finished in 39 ms
十一月 20, 2016 1:16:30 下午 org.apache.coyote.AbstractProtocol start
信息: Starting ProtocolHandler ["http-bio-8080"]
十一月 20, 2016 1:16:30 下午 org.apache.catalina.startup.Catalina start
信息: Server startup in 776 ms

5.java相关内存优化

springside4自定义项目配置

2016-11-13T15:18:00+08:00

springside4自定义项目配置

从上一篇文章springside4环境搭建中可以看出，通过./generate-project.sh生成项目后，还需要手工改page的代码，如果想配置mysql数据库，还需要自己改造pom文件及其相关的配置文件，很是麻烦，那么如何直接改springside4的模版引擎呢。这里springside4是使用了maven-archetype来作为模版引擎来生成代码的。

1.进入到support目录下会看到maven的archetype，如下图所示

2.首先修改pom文件

解注掉mysql的驱动包程序

<jdbc.driver.groupId>mysql</jdbc.driver.groupId>
<jdbc.driver.artifactId>mysql-connector-java</jdbc.driver.artifactId>
<jdbc.driver.version>5.1.22</jdbc.driver.version>

修改mysql初始化数据的脚本目录

由

<profile>
	<id>refresh-db</id>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-antrun-plugin</artifactId>
				<configuration>
					<target>
						<property file="src/main/resources/application.development.properties" />
						<property file="src/main/resources/application.properties" />
						<sql driver="${jdbc.driver}" url="${jdbc.url}" userid="${jdbc.username}" password="${jdbc.password}" onerror="continue" encoding="${project.build.sourceEncoding}">
							<classpath refid="maven.test.classpath" />
							<transaction src="src/main/resources/sql/h2/schema.sql" />
							<transaction src="src/test/resources/data/h2/import-data.sql" />
						</sql>
					</target>
				</configuration>
			</plugin>
		</plugins>
	</build>
</profile>

修改为

<profile>
	<id>refresh-db</id>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-antrun-plugin</artifactId>
				<configuration>
					<target>
						<property file="src/main/resources/application.development.properties" />
						<property file="src/main/resources/application.properties" />
						<sql driver="${jdbc.driver}" url="${jdbc.url}" userid="${jdbc.username}" password="${jdbc.password}" onerror="continue" encoding="${project.build.sourceEncoding}">
							<classpath refid="maven.test.classpath" />
							<transaction src="src/main/resources/sql/mysql/schema.sql" />
							<transaction src="src/test/resources/data/h2/import-data.sql" />
						</sql>
					</target>
				</configuration>
			</plugin>
		</plugins>
	</build>
</profile>

3.修改翻页模版代码，进入到/Users/mac/software/github/springside4/support/maven-archetype/src/main/resources/archetype-resources/src/main/webapp/WEB-INF/tags/pagination.tag文件进行修改page.hasPreviousPage()修改为page.hasPrevious()及page.hasNextPage()修改为page.hasNext()

4.修改/Users/mac/software/github/springside4/support/maven-archetype/src/main/resources/archetype-resources/src/main/resources/application.properties文件

修改为

#set( $symbol_pound = '#' )
#set( $symbol_dollar = '$' )
#set( $symbol_escape = '\' )
${symbol_pound}h2 database settings
${symbol_pound}jdbc.driver=org.h2.Driver
${symbol_pound}jdbc.url=jdbc:h2:file:~/.h2/${artifactId};AUTO_SERVER=TRUE;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
${symbol_pound}jdbc.username=sa
${symbol_pound}jdbc.password=

${symbol_pound}log4jdbc driver
${symbol_pound}jdbc.driver=net.sf.log4jdbc.DriverSpy
${symbol_pound}jdbc.url=jdbc:log4jdbc:h2:file:~/.h2/${artifactId};AUTO_SERVER=TRUE;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE

${symbol_pound}oracle database settings
${symbol_pound}jdbc.driver=oracle.jdbc.driver.OracleDriver
${symbol_pound}jdbc.url=jdbc:oracle:thin:@127.0.0.1:1521:XE
${symbol_pound}jdbc.username=${artifactId}
${symbol_pound}jdbc.password=${artifactId}

${symbol_pound}mysql database setting
jdbc.driver=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/${artifactId}?useUnicode=true&characterEncoding=utf-8
jdbc.username=root
jdbc.password=123456

${symbol_pound}connection pool settings
jdbc.pool.maxIdle=10
jdbc.pool.maxActive=50

5.修改mysql的脚本，否则报错/Users/mac/software/github/springside4/support/maven-archetype/src/main/resources/archetype-resources/src/main/resources/sql/mysql/schema.sql,将register_date的default 0 删除修改后

#set( $symbol_pound = '#' )
#set( $symbol_dollar = '$' )
#set( $symbol_escape = '\' )
drop table if exists ${tablePrefix}task;
drop table if exists ${tablePrefix}user;

create table ${tablePrefix}task (
	id bigint auto_increment,
	title varchar(128) not null,
	description varchar(255),
	user_id bigint not null,
    primary key (id)
) engine=InnoDB;

create table ${tablePrefix}user (
	id bigint auto_increment,
	login_name varchar(64) not null unique,
	name varchar(64) not null,
	password varchar(255) not null,
	salt varchar(64) not null,
	roles varchar(255) not null,
	register_date timestamp not null ,
	primary key (id)
) engine=InnoDB;

6.进入到/Users/mac/software/github/springside4/support/maven-archetype目录下重新安装

./install.sh

./install.sh 

[INFO] Install archetype to local repository.
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building SpringSide :: Archetype :: QuickStart 4.3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ quickstart-archetype ---
[INFO] Deleting /Users/mac/software/github/springside4/support/maven-archetype/target
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) @ quickstart-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 87 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) @ quickstart-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/mac/software/github/springside4/support/maven-archetype/src/test/resources
[INFO] 
[INFO] --- maven-archetype-plugin:2.2:jar (default-jar) @ quickstart-archetype ---
[INFO] Building archetype jar: /Users/mac/software/github/springside4/support/maven-archetype/target/quickstart-archetype-4.3.0-SNAPSHOT
[INFO] 
[INFO] --- maven-archetype-plugin:2.2:integration-test (default-integration-test) @ quickstart-archetype ---
[WARNING] No Archetype IT projects: root 'projects' directory not found.
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ quickstart-archetype ---
[INFO] Installing /Users/mac/software/github/springside4/support/maven-archetype/target/quickstart-archetype-4.3.0-SNAPSHOT.jar to /Users/mac/software/repository/io/springside/examples/quickstart-archetype/4.3.0-SNAPSHOT/quickstart-archetype-4.3.0-SNAPSHOT.jar
[INFO] Installing /Users/mac/software/github/springside4/support/maven-archetype/pom.xml to /Users/mac/software/repository/io/springside/examples/quickstart-archetype/4.3.0-SNAPSHOT/quickstart-archetype-4.3.0-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-archetype-plugin:2.2:update-local-catalog (default-update-local-catalog) @ quickstart-archetype ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.318 s
[INFO] Finished at: 2016-11-13T11:23:03+08:00
[INFO] Final Memory: 13M/155M
[INFO] ------------------------------------------------------------------------

7.重新切换回/Users/mac/software/github/springside4目录，并重新新建项目

./generate-project.sh 
[INFO] Generating project in ./generated-projects
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] >>> maven-archetype-plugin:2.4:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO] 
[INFO] <<< maven-archetype-plugin:2.4:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO] 
[INFO] --- maven-archetype-plugin:2.4:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[WARNING] Archetype not found in any catalog. Falling back to central repository (http://repo.maven.apache.org/maven2).
[WARNING] Use -DarchetypeRepository=<your repository> if archetype's repository is elsewhere.
Downloading: http://repo.maven.apache.org/maven2/io/springside/examples/quickstart-archetype/4.3.0-SNAPSHOT/maven-metadata.xml
Define value for property 'groupId': : com.hitwh
Define value for property 'artifactId': : dataapi
Define value for property 'version':  1.0-SNAPSHOT: : 1.0
Define value for property 'package':  com.hitwh: : dataapi
Define value for property 'tablePrefix': : hadoop_
Confirm properties configuration:
groupId: com.hitwh
artifactId: dataapi
version: 1.0
package: dataapi
tablePrefix: hadoop_
 Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: quickstart-archetype:4.3.0-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.hitwh
[INFO] Parameter: artifactId, Value: dataapi
[INFO] Parameter: version, Value: 1.0
[INFO] Parameter: package, Value: dataapi
[INFO] Parameter: packageInPathFormat, Value: dataapi
[INFO] Parameter: tablePrefix, Value: hadoop_
[INFO] Parameter: package, Value: dataapi
[INFO] Parameter: version, Value: 1.0
[INFO] Parameter: groupId, Value: com.hitwh
[INFO] Parameter: artifactId, Value: dataapi
[INFO] project created from Archetype in dir: /Users/mac/software/github/springside4/generated-projects/dataapi
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39.115 s
[INFO] Finished at: 2016-11-13T11:24:54+08:00
[INFO] Final Memory: 14M/177M
[INFO] ------------------------------------------------------------------------

8.进入到生成的dataapi项目下,并给bin目录下的脚本添加可执行权限

cd /Users/mac/software/github/springside4/generated-projects/dataapi

chmod a+x ./bin/*

然后在mysql数据库中添加项目名称的数据库,否则报错数据库找不到  	

新建数据库后执行bin目录下的	

cd bin

./refresh-db.sh

9.直接用idea打开dataapi工程，启动tomcat，查看首页

当然，上面修改的模版引擎已经提交到https://github.com/CodeToSurvive1/springside4/releases/tag/v1.1的默认分支dev_4.0上，直接fork就不用修改了

springside4环境搭建

2016-11-12T08:18:00+08:00

springside4环境搭建

官方地址(https://github.com/springside/springside4)
由于官方的https://github.com/springside/springside4目前的主分支是master，直接下载会报错，因此需要下载4.0分支，但是启动后查看quickstart仍然会报错，这里直接fork一个版本，在官方基础上进行改造。因此可以直接下载本人的分支,(https://github.com/CodeToSurvive1/springside4.git)，注意，本人已经将主分支切换到修复错误后的分支，可以直接clone即可		

下面会从官方中直接fork进行操作，如果直接使用本人的	[https://github.com/CodeToSurvive1/springside4.git](https://github.com/CodeToSurvive1/springside4.git)分支，请跳过下面(一，fork改造	)，直接进入（二，环境搭建）				 	

一，fork改造

1.首先从https://github.com/springside/springside4直接fork

2.进入到自己fork的分支上，复制地址

3.通过git clone 直接下载,注意，这样加了参数-b意思为下载分支branch，默认不加参数是下载的主分支

git clone -b dev_4.0 https://github.com/CodeToSurvive1/springside4.git
Cloning into 'springside4'...
remote: Counting objects: 24321, done.
remote: Total 24321 (delta 0), reused 0 (delta 0), pack-reused 24321
Receiving objects: 100% (24321/24321), 3.82 MiB | 520.00 KiB/s, done.
Resolving deltas: 100% (8658/8658), done.
Checking connectivity... done.

二，环境搭建

1.下载后，直接进入到springside4中,运行启动命令即可

如果使用本人的分支直接下载
git clone https://github.com/CodeToSurvive1/springside4.git

cd springside4

pwd
/Users/mac/software/github/springside4
mac:springside4 mac$ ./quick-start.sh 

2.经过漫长的maven下载编译过程过后,会在后台启动服务，这时候直接访问url即可(如果maven下载缓慢，可以直接下载本人共享的maven的repository(https://pan.baidu.com/s/1c2mktoO),不过百度云盘也够慢的，自己选择吧

http://localhost:8081/showcase/ 案例集合

http://localhost:8080/quickstart/ 快速演示

用admin，admin登录报错，是由于代码问题，如果直接fork的我的版本，是没有问题的,会直接看到下面的步骤6的成功页面

3.直接用idea打开后,看到下面目录结构

4.打开路径/Users/mac/software/github/springside4/examples/quickstart/src/main/webapp/WEB-INF/tags/pagination.tag文件，直接将下面内容替换掉

直接修改为

5.再次启动命令，进入页面查看

首先干掉上次启动服务后的两个后台进程，否则报端口冲突

ps -ef | grep jetty
  501  3466     1   0 10:51上午 ttys005    1:00.60 /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/bin/java -classpath /Users/mac/software/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar -Dclassworlds.conf=/Users/mac/software/apache-maven-3.3.9/bin/m2.conf -Dmaven.home=/Users/mac/software/apache-maven-3.3.9 -Dmaven.multiModuleProjectDirectory=/Users/mac/software/github/springside4/examples/quickstart org.codehaus.plexus.classworlds.launcher.Launcher clean jetty:run
  501  3467     1   0 10:51上午 ttys005    1:29.03 /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/bin/java -classpath /Users/mac/software/apache-maven-3.3.9/boot/plexus-classworlds-2.5.2.jar -Dclassworlds.conf=/Users/mac/software/apache-maven-3.3.9/bin/m2.conf -Dmaven.home=/Users/mac/software/apache-maven-3.3.9 -Dmaven.multiModuleProjectDirectory=/Users/mac/software/github/springside4/examples/showcase org.codehaus.plexus.classworlds.launcher.Launcher clean jetty:run -Djetty.port=8081
  501  4162  2854   0 11:04上午 ttys005    0:00.00 grep jetty
mac:springside4 mac$ kill -9 3466
mac:springside4 mac$ kill -9 3467

./quick-start.sh 

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.296 s
[INFO] Finished at: 2016-11-12T11:06:39+08:00
[INFO] Final Memory: 32M/277M
[INFO] ------------------------------------------------------------------------
[Step 3] Start all example projects.
[INFO] Please wait a moment then access below demo sites:
[INFO] http://localhost:8080/quickstart
[INFO] http://localhost:8081/showcase

6.继续查看刚才报错的页面http://localhost:8080/quickstart,登录后

三，使用springside4创建自己的maven工程,并配置为mysql数据库

1.直接执行命令generate-project.sh

 pwd
/Users/mac/software/github/springside4
mac:springside4 mac$ ./generate-project.sh 

2.通过命令行输入maven相关配置后

[INFO] Generating project in ./generated-projects
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] >>> maven-archetype-plugin:2.4:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO] 
[INFO] <<< maven-archetype-plugin:2.4:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO] 
[INFO] --- maven-archetype-plugin:2.4:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[WARNING] Archetype not found in any catalog. Falling back to central repository (http://repo.maven.apache.org/maven2).
[WARNING] Use -DarchetypeRepository=<your repository> if archetype's repository is elsewhere.
Downloading: http://repo.maven.apache.org/maven2/io/springside/examples/quickstart-archetype/4.3.0-SNAPSHOT/maven-metadata.xml
Define value for property 'groupId': : com.hitwh
Define value for property 'artifactId': : hadoop_project
Define value for property 'version':  1.0-SNAPSHOT: : 1.0
Define value for property 'package':  com.hitwh: : hadoop
Define value for property 'tablePrefix': : hadoop_
Confirm properties configuration:
groupId: com.hitwh
artifactId: hadoop_project
version: 1.0
package: hadoop
tablePrefix: hadoop_
 Y: : Y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: quickstart-archetype:4.3.0-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.hitwh
[INFO] Parameter: artifactId, Value: hadoop_project
[INFO] Parameter: version, Value: 1.0
[INFO] Parameter: package, Value: hadoop
[INFO] Parameter: packageInPathFormat, Value: hadoop
[INFO] Parameter: tablePrefix, Value: hadoop_
[INFO] Parameter: package, Value: hadoop
[INFO] Parameter: version, Value: 1.0
[INFO] Parameter: groupId, Value: com.hitwh
[INFO] Parameter: artifactId, Value: hadoop_project
[INFO] project created from Archetype in dir: /Users/mac/software/github/springside4/generated-projects/hadoop_project
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:32 min
[INFO] Finished at: 2016-11-12T11:13:19+08:00
[INFO] Final Memory: 15M/205M
[INFO] ------------------------------------------------------------------------

3.这个时候会在generated-projects文件夹下出现刚才的项目

4.修改pom文件

解注掉mysql驱动包配置

<jdbc.driver.groupId>mysql</jdbc.driver.groupId>
<jdbc.driver.artifactId>mysql-connector-java</jdbc.driver.artifactId>
<jdbc.driver.version>5.1.22</jdbc.driver.version>

并将最下面的配置文件

<!-- 刷新开发环境数据库 -->
<profile>
	<id>refresh-db</id>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-antrun-plugin</artifactId>
				<configuration>
					<target>
						<property file="src/main/resources/application.development.properties" />
						<property file="src/main/resources/application.properties" />
						<sql driver="${jdbc.driver}" url="${jdbc.url}" userid="${jdbc.username}" password="${jdbc.password}" onerror="continue" encoding="${project.build.sourceEncoding}">
							<classpath refid="maven.test.classpath" />
							<transaction src="src/main/resources/sql/h2/schema.sql" />
							<transaction src="src/test/resources/data/h2/import-data.sql" />
						</sql>
					</target>
				</configuration>
			</plugin>
		</plugins>
	</build>
</profile>

修改为

<!-- 刷新开发环境数据库 -->
<profile>
	<id>refresh-db</id>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-antrun-plugin</artifactId>
				<configuration>
					<target>
						<property file="src/main/resources/application.development.properties" />
						<property file="src/main/resources/application.properties" />
						<sql driver="${jdbc.driver}" url="${jdbc.url}" userid="${jdbc.username}" password="${jdbc.password}" onerror="continue" encoding="${project.build.sourceEncoding}">
							<classpath refid="maven.test.classpath" />
							<transaction src="src/main/resources/sql/mysql/schema.sql" />
							<transaction src="src/test/resources/data/h2/import-data.sql" />
						</sql>
					</target>
				</configuration>
			</plugin>
		</plugins>
	</build>
</profile>

5.修改/Users/mac/software/github/springside4/generated-projects/hadoop_project/src/main/resources/application.properties中的配置，改为mysql配置

#h2 database settings
#jdbc.driver=org.h2.Driver
#jdbc.url=jdbc:h2:file:~/.h2/hadoop_project;AUTO_SERVER=TRUE;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
#jdbc.username=sa
#jdbc.password=

#log4jdbc driver
#jdbc.driver=net.sf.log4jdbc.DriverSpy
#jdbc.url=jdbc:log4jdbc:h2:file:~/.h2/hadoop_project;AUTO_SERVER=TRUE;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE

#oracle database settings
#jdbc.driver=oracle.jdbc.driver.OracleDriver
#jdbc.url=jdbc:oracle:thin:@127.0.0.1:1521:XE
#jdbc.username=hadoop_project
#jdbc.password=hadoop_project

#mysql database setting
jdbc.driver=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/hadoop_project?useUnicode=true&characterEncoding=utf-8
jdbc.username=root
jdbc.password=123456

#connection pool settings
jdbc.pool.maxIdle=10
jdbc.pool.maxActive=50

6.修改路径/Users/mac/software/github/springside4/generated-projects/hadoop_project/src/main/resources/sql/mysql/schema.sql中的脚本，否则报错Invalid default value for ‘register_date’

drop table if exists hadoop_task;
drop table if exists hadoop_user;

create table hadoop_task (
	id bigint auto_increment,
	title varchar(128) not null,
	description varchar(255),
	user_id bigint not null,
    primary key (id)
) engine=InnoDB;

create table hadoop_user (
	id bigint auto_increment,
	login_name varchar(64) not null unique,
	name varchar(64) not null,
	password varchar(255) not null,
	salt varchar(64) not null,
	roles varchar(255) not null,
	register_date timestamp not null ,
	primary key (id)
) engine=InnoDB;

7.在mysql数据库中建立hadoop_project数据库

8.执行切换到目录下执行建表语句

mac:bin mac$ pwd
/Users/mac/software/github/springside4/generated-projects/hadoop_project/bin
mac:bin mac$ ./refresh-db.sh 
[INFO] Re-create the schema and provision the sample data.
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Springside :: Example :: QuickStart 1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (default-cli) @ hadoop_project ---
[INFO] Executing tasks

main:
      [sql] Executing resource: /Users/mac/software/github/springside4/generated-projects/hadoop_project/src/main/resources/sql/mysql/schema.sql
      [sql] Executing resource: /Users/mac/software/github/springside4/generated-projects/hadoop_project/src/test/resources/data/h2/import-data.sql
      [sql] 11 of 11 SQL statements executed successfully
[INFO] Executed tasks
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.270 s
[INFO] Finished at: 2016-11-12T11:37:35+08:00
[INFO] Final Memory: 12M/227M
[INFO] ------------------------------------------------------------------------
mac:bin mac$ 

9.同样需要修改/Users/mac/software/github/springside4/generated-projects/hadoop_project/src/main/webapp/WEB-INF/tags/pagination.tag文件中的两处错误修改后为

10.单独打开新建的hadoop_project项目

重新导入maven相关包，

11.新建tomcat服务器，并进行相关配置

12.启动tomcat，不要忘记先干掉之前的jetty进程，访问8080端口

http://localhost:8080/login

生成的hadoop_project地址可以通过https://github.com/CodeToSurvive1/hadoop_project/tree/v1.0下载

mac系统搭建手工测试openstf

2016-11-02T08:18:00+08:00

openstf环境搭建

官方地址https://github.com/openstf/stf

1.安装依赖环境


brew install rethinkdb graphicsmagick zeromq protobuf yasm pkg-config

2.安装stf


npm install -g stf

3.启动rethinkdb,注意，启动后会生成数据目录，因此最好放在你想存放数据的地方，这里新建目录

cd /Users/mac/software/github/macaca

mkdir -p openstf/rethinkdb

cd /Users/mac/software/github/macaca/openstf/rethinkdb

rethinkdb

Recursively removing directory /Users/mac/software/github/macaca/openstf/rethinkdb/rethinkdb_data/tmp
Initializing directory /Users/mac/software/github/macaca/openstf/rethinkdb/rethinkdb_data
Running rethinkdb 2.3.5 (CLANG 8.0.0 (clang-800.0.38))...
Running on Darwin 16.1.0 x86_64
Loading data from directory /Users/mac/software/github/macaca/openstf/rethinkdb/rethinkdb_data
Listening for intracluster connections on port 29015
Listening for client driver connections on port 28015
Listening for administrative HTTP connections on port 8080
Listening on cluster addresses: 127.0.0.1, ::1
Listening on driver addresses: 127.0.0.1, ::1
Listening on http addresses: 127.0.0.1, ::1
To fully expose RethinkDB on the network, bind to all addresses by running rethinkdb with the `--bind all` command line option.
Server ready, "mac_local_x2o" 01cdb2a8-0138-4245-9c28-02b3d95a9a6c

4.启动stf

新建窗口，启动stf

stf local

stf local
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js migrate"
INF/db 3337 [*] Connecting to 127.0.0.1:28015
INF/db:setup 3337 [*] Database "stf" created
INF/db:setup 3337 [*] Table "users" created
INF/db:setup 3337 [*] Table "vncauth" created
INF/db:setup 3337 [*] Table "devices" created
INF/db:setup 3337 [*] Table "logs" created
INF/db:setup 3337 [*] Index "users"."adbKeys" created
INF/db:setup 3337 [*] Waiting for index "users"."adbKeys"
INF/db:setup 3337 [*] Index "vncauth"."response" created
INF/db:setup 3337 [*] Waiting for index "vncauth"."response"
INF/db:setup 3337 [*] Index "devices"."owner" created
INF/db:setup 3337 [*] Waiting for index "devices"."owner"
INF/db:setup 3337 [*] Index "vncauth"."responsePerDevice" created
INF/db:setup 3337 [*] Waiting for index "vncauth"."responsePerDevice"
INF/db:setup 3337 [*] Index "devices"."present" created
INF/db:setup 3337 [*] Waiting for index "devices"."present"
INF/db:setup 3337 [*] Table "accessTokens" created
INF/db:setup 3337 [*] Index "devices"."providerChannel" created
INF/db:setup 3337 [*] Waiting for index "devices"."providerChannel"
INF/db:setup 3337 [*] Index "accessTokens"."email" created
INF/db:setup 3337 [*] Waiting for index "accessTokens"."email"
INF/db:setup 3337 [*] Index "devices"."owner" is ready
INF/db:setup 3337 [*] Index "accessTokens"."email" is ready
INF/db:setup 3337 [*] Index "users"."adbKeys" is ready
INF/db:setup 3337 [*] Index "vncauth"."response" is ready
INF/db:setup 3337 [*] Index "vncauth"."responsePerDevice" is ready
INF/db:setup 3337 [*] Index "devices"."providerChannel" is ready
INF/db:setup 3337 [*] Index "devices"."present" is ready
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js triproxy app001 --bind-pub tcp://127.0.0.1:7111 --bind-dealer tcp://127.0.0.1:7112 --bind-pull tcp://127.0.0.1:7113"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js triproxy dev001 --bind-pub tcp://127.0.0.1:7114 --bind-dealer tcp://127.0.0.1:7115 --bind-pull tcp://127.0.0.1:7116"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js processor proc001 --connect-app-dealer tcp://127.0.0.1:7112 --connect-dev-dealer tcp://127.0.0.1:7115"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js processor proc002 --connect-app-dealer tcp://127.0.0.1:7112 --connect-dev-dealer tcp://127.0.0.1:7115"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js reaper reaper001 --connect-push tcp://127.0.0.1:7116 --connect-sub tcp://127.0.0.1:7111"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js provider --name mac.local --min-port 7400 --max-port 7700 --connect-sub tcp://127.0.0.1:7114 --connect-push tcp://127.0.0.1:7116 --group-timeout 900 --public-ip localhost --storage-url http://localhost:7100/ --adb-host 127.0.0.1 --adb-port 5037 --vnc-initial-size 600x800"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js auth-mock --port 7120 --secret kute kittykat --app-url http://localhost:7100/"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js app --port 7105 --secret kute kittykat --auth-url http://localhost:7100/auth/mock/ --websocket-url http://localhost:7110/"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js api --port 7106 --secret kute kittykat --connect-push tcp://127.0.0.1:7113 --connect-sub tcp://127.0.0.1:7111"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js websocket --port 7110 --secret kute kittykat --storage-url http://localhost:7100/ --connect-sub tcp://127.0.0.1:7111 --connect-push tcp://127.0.0.1:7113"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js storage-temp --port 7102"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js storage-plugin-image --port 7103 --storage-url http://localhost:7100/"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js storage-plugin-apk --port 7104 --storage-url http://localhost:7100/"
INF/util:procutil 3336 [*] Forking "/usr/local/lib/node_modules/stf/lib/cli.js poorxy --port 7100 --app-url http://localhost:7105/ --auth-url http://localhost:7120/ --api-url http://localhost:7106/ --websocket-url http://localhost:7110/ --storage-url http://localhost:7102/ --storage-plugin-image-url http://localhost:7103/ --storage-plugin-apk-url http://localhost:7104/"
INF/triproxy 3345 [app001] PUB socket bound on tcp://127.0.0.1:7111
INF/triproxy 3345 [app001] DEALER socket bound on tcp://127.0.0.1:7112
INF/triproxy 3345 [app001] PULL socket bound on tcp://127.0.0.1:7113
INF/triproxy 3346 [dev001] PUB socket bound on tcp://127.0.0.1:7114
INF/triproxy 3346 [dev001] DEALER socket bound on tcp://127.0.0.1:7115
INF/triproxy 3346 [dev001] PULL socket bound on tcp://127.0.0.1:7116
INF/reaper 3349 [reaper001] Subscribing to permanent channel "*ALL"
INF/reaper 3349 [reaper001] Reaping devices with no heartbeat
INF/db 3347 [proc001] Connecting to 127.0.0.1:28015
INF/db 3348 [proc002] Connecting to 127.0.0.1:28015
INF/db 3349 [reaper001] Connecting to 127.0.0.1:28015
INF/processor 3347 [proc001] App dealer connected to "tcp://127.0.0.1:7112"
INF/processor 3347 [proc001] Device dealer connected to "tcp://127.0.0.1:7115"
INF/processor 3348 [proc002] App dealer connected to "tcp://127.0.0.1:7112"
INF/processor 3348 [proc002] Device dealer connected to "tcp://127.0.0.1:7115"
INF/reaper 3349 [reaper001] Receiving input from "tcp://127.0.0.1:7111"
INF/reaper 3349 [reaper001] Sending output to "tcp://127.0.0.1:7116"
INF/poorxy 3358 [*] Listening on port 7100
INF/storage:plugins:image 3356 [*] Listening on port 7103
INF/auth-mock 3351 [*] Listening on port 7120
INF/provider 3350 [*] Subscribing to permanent channel "tNTchuTuQWeSBGur9r8xsg=="
INF/storage:plugins:apk 3357 [*] Listening on port 7104
INF/provider 3350 [*] Sending output to "tcp://127.0.0.1:7116"
INF/provider 3350 [*] Receiving input from "tcp://127.0.0.1:7114"
INF/storage:temp 3355 [*] Listening on port 7102
INF/app 3352 [*] Using pre-built resources
INF/app 3352 [*] Listening on port 7105
INF/db 3352 [*] Connecting to 127.0.0.1:28015
INF/websocket 3354 [*] Subscribing to permanent channel "*ALL"
INF/websocket 3354 [*] Listening on port 7110
INF/db 3354 [*] Connecting to 127.0.0.1:28015
INF/websocket 3354 [*] Sending output to "tcp://127.0.0.1:7113"
INF/websocket 3354 [*] Receiving input from "tcp://127.0.0.1:7111"
INF/api 3353 [*] Subscribing to permanent channel "*ALL"
INF/api 3353 [*] Listening on port 7106
INF/api 3353 [*] Sending output to "tcp://127.0.0.1:7113"
INF/api 3353 [*] Receiving input from "tcp://127.0.0.1:7111"
INF/db 3353 [*] Connecting to 127.0.0.1:28015
INF/provider 3350 [*] Tracking devices

5.访问地址，http://localhost:7105/

随便输入用户名及邮箱

这时候随便插入一台android设备后，会出现页面

一会后识别成功

点击mi5图标进入手机操控页面

reliable-master源码安装及调试及插件配置

2016-11-01T08:52:00+08:00

源码环境下载搭建

1.从github上下载最新的源码

git clone https://github.com/reliablejs/reliable-master.git
Cloning into 'reliable-master'...
remote: Counting objects: 992, done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 992 (delta 7), reused 0 (delta 0), pack-reused 942
Receiving objects: 100% (992/992), 301.72 KiB | 66.00 KiB/s, done.
Resolving deltas: 100% (517/517), done.
Checking connectivity... done.

2.编译源码

cd reliable-master/
make build 

3.启动服务

make server
./bin/reliable-master server -p 3333 -w 1 --verbose
>> index.js:44:14 [master] pid:3521 worker fork success
>> index.js:62:14 [master] pid:3521 worker online
>> index.js:21:10 [worker:1] pid:3522 workder init with config:
 { server: { worker: '1', port: '3333', protocol: 'http', verbose: true },
  database: 'mongodb://localhost/reliable',
  site: 
   { title: 'Reliable',
     baseurl: 'http://reliable-test.com',
     locale: 'en-US',
     docurl: 'https://macacajs.github.io/macaca',
     issueurl: 'https://github.com/reliablejs/reliable-master',
     login: true },
  auth: 
   { github: 
      { client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' },
     gitlab: 
      { protocol: 'http',
        server_url: '127.0.0.1:3000',
        client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' } },
  mail: 
   { name: 'reliable test',
     port: 465,
     host: 'smtp.reliable-test.com',
     secure: true,
     ignoreTLS: true,
     auth: { user: 'test@reliable-test.com', pass: 'reliable' },
     sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200' },
  plugins: {},
  pluginTextPrefix: 'reliable-plugin',
  pkg: 
   { name: 'reliable-master',
     version: '1.1.0',
     description: 'Distributed reliable continuous integration service.',
     keywords: [ 'ci', 'automation', [length]: 2 ],
     bin: { 'reliable-master': './bin/reliable-master' },
     repository: 
      { type: 'git',
        url: 'git@github.com:reliablejs/reliable-master.git' },
     dependencies: 
      { 'ansi-to-html': '~0.3.0',
        babel: '~5.6.14',
        badgeboard: '~0.1.1',
        bcryptjs: '~2.2.1',
        chalk: '~1.1.1',
        co: '~4.6',
        'co-body': '~1.1.0',
        'co-request': '^0.2.1',
        commander: '~2.0.0',
        'detect-port': '~0.1.3',
        koa: '~0.21.0',
        'koa-csrf': '2.1.3',
        'koa-favicon': '~1.2.0',
        'koa-generic-session': '~1.10.0',
        'koa-redis': '~1.0.1',
        'koa-router': '~3.7.0',
        'koa-static': '~1.4.7',
        mongoose: '=4.1.5',
        nodemailer: '~1.4.0',
        'npm-update': '1.0.2',
        react: '~0.14.0',
        'react-dom': '~0.14.2',
        redis: '~2.3.1',
        'reliable-events': '^0.1.4',
        'reliable-github-oauth': '^1.0.0',
        'reliable-gitlab-oauth': '^1.0.0',
        'reliable-logger': '^1.0.7',
        'reliable-mail': '^0.2.0',
        'reliable-plugin': '^0.1.8',
        validator: '~3.37.0',
        xutil: '~1.0.0',
        yamljs: '~0.2.4',
        zmq: '2.13.0' },
     devDependencies: 
      { 'co-mocha': '~1.1.2',
        'command-line-test': '^1.0.5',
        eslint: '~1.1.0',
        'eslint-plugin-react': '~3.2.3',
        istanbul: '*',
        'json-loader': '~0.5.2',
        'jsx-loader': '~0.13.2',
        killing: '~1.0.1',
        mocha: '~2.4.5',
        'pre-commit': '~1.1.1',
        'react-d3-components': '~0.6.0',
        should: '*',
        webpack: '^1.13.1' },
     scripts: { test: 'make test', lint: 'make lint' },
     'pre-commit': [ 'lint', [length]: 1 ],
     engines: { node: '>=4.2.1' },
     homepage: 'https://reliablejs.github.io/reliable',
     license: 'MIT' } }
>> index.js:32:10 [worker:1] pid:3522 base middlewares attached
>> index.js:36:12 [worker:1] pid:3522 middleware: i18n registed
>> render.js:15:8 [worker:1] pid:3522 render view path: /Users/mac/software/github/macaca/reliable-master/web/views
>> index.js:36:12 [worker:1] pid:3522 middleware: inject registed
>> index.js:36:12 [worker:1] pid:3522 middleware: favicon registed
>> index.js:36:12 [worker:1] pid:3522 middleware: powerby registed
>> index.js:36:12 [worker:1] pid:3522 middleware: static registed
>> router.js:41:10 [worker:1] pid:3522 router set
>> index.js:67:14 [master] pid:3521 listening worker id: 1, pid: 3522, address: 172.31.218.249:3333
>> Server start at 2016-11-01 08:57:40 | http://172.31.218.249:3333

前提是需要首先启动mongodb服务及redis服务，请参考连接mac系统搭建master-slave环境搭建(非docker版本)

邮箱配置

默认情况下，reliable-master是配置了邮箱服务插件的，那么该如何使用邮箱服务呢

这里注册了一个163邮箱,下面进行单独的说明

1.申请个人邮箱**#163.com，密码qwer1234

2.修改/Users/mac/software/github/macaca/reliable-master/common目录下的config.js文件

默认值为

mail: {
    name: 'reliable test',
    port: 465,
    host: 'smtp.reliable-test.com',
    secure: true,
    ignoreTLS: true,
    auth: {
      user: 'test@reliable-test.com',
      pass: 'reliable'
    },
    sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200'
  },

修改为注册的邮箱

mail: {
    name: '李小蛟',
    port: 465,
    host: 'smtp.163.com',
    secure: true,
    ignoreTLS: true,
    auth: {
      user: '***@163.com',
      pass: 'qwer1234'
    },
    sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200'
  },

并修改语言版本locale: ‘en-US’,为locale: ‘zh-CN’,

3.重新编译reliable-master

mac:reliable-master mac$ make build
Hash: 6a4c6d908ed02be70570
Version: webpack 1.13.3
Time: 4289ms
       Asset     Size  Chunks             Chunk Names
     auth.js  3.78 kB       0  [emitted]  auth
dashboard.js  1.14 MB       1  [emitted]  dashboard
  history.js  1.14 MB       2  [emitted]  history
     home.js  1.14 MB       3  [emitted]  home
    index.js  2.42 kB       4  [emitted]  index
    login.js  3.34 kB       5  [emitted]  login
 password.js  4.49 kB       6  [emitted]  password
   signup.js  3.99 kB       7  [emitted]  signup
     task.js  1.14 MB       8  [emitted]  task
     user.js   1.4 kB       9  [emitted]  user
    + 191 hidden modules

4.启动master服务(前提启动mongodb及redis服务)

make server

./bin/reliable-master server -p 3333 -w 1 --verbose

*****************************************
*  version 1.0.0 is outdate             *
*  run: npm i -g reliable-master@1.1.0  *
*****************************************

>> index.js:44:14 [master] pid:4401 worker fork success
>> index.js:62:14 [master] pid:4401 worker online
>> index.js:21:10 [worker:1] pid:4404 workder init with config:
 { server: { worker: '1', port: '3333', protocol: 'http', verbose: true },
  database: 'mongodb://localhost/reliable',
  site: 
   { title: 'Reliable',
     baseurl: 'http://reliable-test.com',
     locale: 'en-US',
     docurl: 'https://macacajs.github.io/macaca',
     issueurl: 'https://github.com/reliablejs/reliable-master',
     login: true },
  auth: 
   { github: 
      { client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' },
     gitlab: 
      { protocol: 'http',
        server_url: '127.0.0.1:3000',
        client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' } },
  mail: 
   { name: '李小蛟',
     port: 465,
     host: 'smtp.163.com',
     secure: true,
     ignoreTLS: true,
     auth: { user: 'bankcomm_test@163.com', pass: 'qwer1234' },
     sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200' },
  plugins: {},
  pluginTextPrefix: 'reliable-plugin',
  pkg: 
   { name: 'reliable-master',
     version: '1.1.0',
     description: 'Distributed reliable continuous integration service.',
     keywords: [ 'ci', 'automation', [length]: 2 ],
     bin: { 'reliable-master': './bin/reliable-master' },
     repository: 
      { type: 'git',
        url: 'git@github.com:reliablejs/reliable-master.git' },
     dependencies: 
      { 'ansi-to-html': '~0.3.0',
        babel: '~5.6.14',
        badgeboard: '~0.1.1',
        bcryptjs: '~2.2.1',
        chalk: '~1.1.1',
        co: '~4.6',
        'co-body': '~1.1.0',
        'co-request': '^0.2.1',
        commander: '~2.0.0',
        'detect-port': '~0.1.3',
        koa: '~0.21.0',
        'koa-csrf': '2.1.3',
        'koa-favicon': '~1.2.0',
        'koa-generic-session': '~1.10.0',
        'koa-redis': '~1.0.1',
        'koa-router': '~3.7.0',
        'koa-static': '~1.4.7',
        mongoose: '=4.1.5',
        nodemailer: '~1.4.0',
        'npm-update': '1.0.2',
        react: '~0.14.0',
        'react-dom': '~0.14.2',
        redis: '~2.3.1',
        'reliable-events': '^0.1.4',
        'reliable-github-oauth': '^1.0.0',
        'reliable-gitlab-oauth': '^1.0.0',
        'reliable-logger': '^1.0.7',
        'reliable-mail': '^0.2.0',
        'reliable-plugin': '^0.1.8',
        validator: '~3.37.0',
        xutil: '~1.0.0',
        yamljs: '~0.2.4',
        zmq: '2.13.0' },
     devDependencies: 
      { 'co-mocha': '~1.1.2',
        'command-line-test': '^1.0.5',
        eslint: '~1.1.0',
        'eslint-plugin-react': '~3.2.3',
        istanbul: '*',
        'json-loader': '~0.5.2',
        'jsx-loader': '~0.13.2',
        killing: '~1.0.1',
        mocha: '~2.4.5',
        'pre-commit': '~1.1.1',
        'react-d3-components': '~0.6.0',
        should: '*',
        webpack: '^1.13.1' },
     scripts: { test: 'make test', lint: 'make lint' },
     'pre-commit': [ 'lint', [length]: 1 ],
     engines: { node: '>=4.2.1' },
     homepage: 'https://reliablejs.github.io/reliable',
     license: 'MIT' } }
>> index.js:32:10 [worker:1] pid:4404 base middlewares attached
>> index.js:36:12 [worker:1] pid:4404 middleware: i18n registed
>> render.js:15:8 [worker:1] pid:4404 render view path: /Users/mac/software/github/macaca/reliable-master/web/views
>> index.js:36:12 [worker:1] pid:4404 middleware: inject registed
>> index.js:36:12 [worker:1] pid:4404 middleware: favicon registed
>> index.js:36:12 [worker:1] pid:4404 middleware: powerby registed
>> index.js:36:12 [worker:1] pid:4404 middleware: static registed
>> router.js:41:10 [worker:1] pid:4404 router set
>> index.js:67:14 [master] pid:4401 listening worker id: 1, pid: 4404, address: 172.31.218.249:3333
>> Server start at 2016-11-01 09:11:20 | http://172.31.218.249:3333

5.访问http://172.31.218.249:3333地址，

登录邮箱查看是否收到发送的邮件。

在发送邮件之前，请确定自己的163邮箱已经开启stmp及pop服务，才可以发送邮件。

需要注意，我这里使用新注册的邮箱，开启stmp服务是需要设置授权码的，但是reliable-master中的配置文件是不支持这种授权码形式的，所以新注册的163邮箱是无法使用的，但是以前注册的163邮箱是可以的，我这里使用的是以前的163邮箱。qq邮箱现在也是需要设置授权码才能开通stmp服务的。如果163邮箱和qq邮箱都无法使用，那还是找一家邮箱注册用户不太多的来测试邮件插件吧，小公司用户量少的应该基本不会有授权码这一单独设置的。

6.可以订阅任务跑脚本尝试一下了.

插件安装

对于插件的安装还是很简单的，只需要在config文件中添加插件名称设置为true即可，目前官方提供的插件有两个，dingdingtalk跟slack

1.配置

下面是配置方式

  plugins: {
      'reliable-plugin-dingtalk': true,
      'reliable-plugin-slack':true
  },  

2.安装插件

然后进入reliable-master目录下安装reliable-plugin-dingtalk及reliable-plugin-slack插件

npm install --save reliable-plugin-slack 
reliable-plugin-slack@0.1.3 node_modules/reliable-plugin-slack
├── co-body@4.2.0 (inflation@2.0.0, qs@4.0.0, raw-body@2.1.7, type-is@1.6.13)
├── koa-static@2.0.0 (debug@2.2.0, koa-send@3.1.1)
├── request@2.76.0 (aws-sign2@0.6.0, tunnel-agent@0.4.3, forever-agent@0.6.1, oauth-sign@0.8.2, caseless@0.11.0, is-typedarray@1.0.0, stringstream@0.0.5, isstream@0.1.2, aws4@1.5.0, json-stringify-safe@5.0.1, extend@3.0.0, node-uuid@1.4.7, qs@6.3.0, combined-stream@1.0.5, mime-types@2.1.12, tough-cookie@2.3.2, form-data@2.1.1, hawk@3.1.3, http-signature@1.1.1, har-validator@2.0.6)
└── react@15.3.2 (object-assign@4.1.0, loose-envify@1.3.0, fbjs@0.8.5)


npm install --save reliable-plugin-dingtalk
reliable-plugin-dingtalk@0.1.4 node_modules/reliable-plugin-dingtalk
├── only@0.0.2
├── koa-static@2.0.0 (debug@2.2.0, koa-send@3.1.1)
├── co-body@4.2.0 (inflation@2.0.0, qs@4.0.0, type-is@1.6.13, raw-body@2.1.7)
├── request@2.76.0 (aws-sign2@0.6.0, tunnel-agent@0.4.3, oauth-sign@0.8.2, forever-agent@0.6.1, is-typedarray@1.0.0, caseless@0.11.0, stringstream@0.0.5, aws4@1.5.0, isstream@0.1.2, json-stringify-safe@5.0.1, extend@3.0.0, node-uuid@1.4.7, qs@6.3.0, combined-stream@1.0.5, mime-types@2.1.12, tough-cookie@2.3.2, form-data@2.1.1, hawk@3.1.3, http-signature@1.1.1, har-validator@2.0.6)
├── react@15.3.2 (object-assign@4.1.0, loose-envify@1.3.0, fbjs@0.8.5)
└── babel-preset-react@6.16.0 (babel-plugin-syntax-jsx@6.18.0, babel-plugin-syntax-flow@6.18.0, babel-plugin-transform-react-jsx@6.8.0, babel-plugin-transform-flow-strip-types@6.18.0, babel-plugin-transform-react-display-name@6.8.0, babel-plugin-transform-react-jsx-source@6.9.0, babel-plugin-transform-react-jsx-self@6.11.0)

3.编译

make build
Hash: 4430e4f8d53acc4b7889
Version: webpack 1.13.3
Time: 3733ms
       Asset     Size  Chunks             Chunk Names
     auth.js  3.78 kB       0  [emitted]  auth
dashboard.js  1.14 MB       1  [emitted]  dashboard
  history.js  1.14 MB       2  [emitted]  history
     home.js  1.14 MB       3  [emitted]  home
    index.js  2.42 kB       4  [emitted]  index
    login.js  3.34 kB       5  [emitted]  login
 password.js  4.49 kB       6  [emitted]  password
   signup.js  3.99 kB       7  [emitted]  signup
     task.js  1.14 MB       8  [emitted]  task
     user.js   1.4 kB       9  [emitted]  user
    + 191 hidden modules

4.运行

make server
./bin/reliable-master server -p 3333 -w 1 --verbose
>> index.js:44:14 [master] pid:15102 worker fork success
>> index.js:62:14 [master] pid:15102 worker online
>> index.js:21:10 [worker:1] pid:15103 workder init with config:
 { server: { worker: '1', port: '3333', protocol: 'http', verbose: true },
  database: 'mongodb://localhost/reliable',
  site: 
   { title: 'Reliable',
     baseurl: 'http://reliable-test.com',
     locale: 'zh-CN',
     docurl: 'https://macacajs.github.io/macaca',
     issueurl: 'https://github.com/reliablejs/reliable-master',
     login: true },
  auth: 
   { github: 
      { client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' },
     gitlab: 
      { protocol: 'http',
        server_url: '127.0.0.1:3000',
        client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' } },
  mail: 
   { name: '李小蛟',
     port: 465,
     host: 'smtp.163.com',
     secure: true,
     ignoreTLS: true,
     auth: { user: 'lixiaojiao_hit@163.com', pass: 'loveTing1314' },
     sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200' },
  plugins: 
   { 'reliable-plugin-dingtalk': true,
     'reliable-plugin-slack': true },
  pluginTextPrefix: 'reliable-plugin',
  pkg: 
   { name: 'reliable-master',
     version: '1.1.0',
     description: 'Distributed reliable continuous integration service.',
     keywords: [ 'ci', 'automation', [length]: 2 ],
     bin: { 'reliable-master': './bin/reliable-master' },
     repository: 
      { type: 'git',
        url: 'git@github.com:reliablejs/reliable-master.git' },
     dependencies: 
      { 'ansi-to-html': '~0.3.0',
        babel: '~5.6.14',
        badgeboard: '~0.1.1',
        bcryptjs: '~2.2.1',
        chalk: '~1.1.1',
        co: '~4.6',
        'co-body': '~1.1.0',
        'co-request': '^0.2.1',
        commander: '~2.0.0',
        'detect-port': '~0.1.3',
        koa: '~0.21.0',
        'koa-csrf': '2.1.3',
        'koa-favicon': '~1.2.0',
        'koa-generic-session': '~1.10.0',
        'koa-redis': '~1.0.1',
        'koa-router': '~3.7.0',
        'koa-static': '~1.4.7',
        mongoose: '=4.1.5',
        nodemailer: '~1.4.0',
        'npm-update': '1.0.2',
        react: '~0.14.0',
        'react-dom': '~0.14.2',
        redis: '~2.3.1',
        'reliable-events': '^0.1.4',
        'reliable-github-oauth': '^1.0.0',
        'reliable-gitlab-oauth': '^1.0.0',
        'reliable-logger': '^1.0.7',
        'reliable-mail': '^0.2.0',
        'reliable-plugin': '^0.1.8',
        'reliable-plugin-dingtalk': '^0.1.4',
        'reliable-plugin-slack': '^0.1.3',
        validator: '~3.37.0',
        xutil: '~1.0.0',
        yamljs: '~0.2.4',
        zmq: '2.13.0' },
     devDependencies: 
      { 'co-mocha': '~1.1.2',
        'command-line-test': '^1.0.5',
        eslint: '~1.1.0',
        'eslint-plugin-react': '~3.2.3',
        istanbul: '*',
        'json-loader': '~0.5.2',
        'jsx-loader': '~0.13.2',
        killing: '~1.0.1',
        mocha: '~2.4.5',
        'pre-commit': '~1.1.1',
        'react-d3-components': '~0.6.0',
        should: '*',
        webpack: '^1.13.1' },
     scripts: { test: 'make test', lint: 'make lint' },
     'pre-commit': [ 'lint', [length]: 1 ],
     engines: { node: '>=4.2.1' },
     homepage: 'https://reliablejs.github.io/reliable',
     license: 'MIT' } }
>> index.js:32:10 [worker:1] pid:15103 base middlewares attached
>> index.js:36:12 [worker:1] pid:15103 middleware: i18n registed
>> render.js:15:8 [worker:1] pid:15103 render view path: /Users/mac/software/github/macaca/reliable-master/web/views
>> index.js:36:12 [worker:1] pid:15103 middleware: inject registed
>> index.js:36:12 [worker:1] pid:15103 middleware: favicon registed
>> index.js:36:12 [worker:1] pid:15103 middleware: powerby registed
>> index.js:36:12 [worker:1] pid:15103 middleware: static registed
plugin: reliable-plugin-dingtalk registered
plugin: reliable-plugin-slack registered
>> router.js:41:10 [worker:1] pid:15103 router set
>> index.js:67:14 [master] pid:15102 listening worker id: 1, pid: 15103, address: 172.31.218.249:3333
>> Server start at 2016-11-01 16:40:13 | http://172.31.218.249:3333

5.查看页面

调试开发reliable-master

这里使用vscode进行调试

由于reliable-master使用cluster模式，在启动服务的时候会启动多进程，这样导致无法调试，因此这里将服务启动后的核心服务端程序单独摘出来，编写js来调试

1.编写js脚本

在/Users/mac/software/github/macaca/reliable-master/init/目录下新建runner.js，并编写下面内容

'use strict';

const cluster = require('cluster');

const Web = require('../web');
const Task = require('../core').Task;
const _ = require('../common/utils/helper');

const defaultOpts = require('../common/config').get();
const program = require('commander');

let options = _.clone(defaultOpts);
_.merge(options.server, _.getConfig(program));
options.pkg = require('../package');

/**
 * Created by mac on 2016/10/31.
 */
Web.init(options, () => {
    Task.bind();
});

2.打开debug标签

修改路径为刚才的runner.js

3.在runner.js中打断点，继续重新点击debug按钮

在/Users/mac/software/github/macaca/reliable-master/web/controllers/dashboard.js中打入断点

刷新http://172.31.218.249:8080/dashboard页面看到截图信息

hive安装，远程模式

2016-10-30T09:04:00+08:00

hive远程模式安装步骤

远程模式，特点是：hive服务和metastore在不同的进程内，可能是不同的机器。该模式需要将hive.metastore.local设置为false，并将hive.metastore.uris设置为metastore服务器URI，如有多个metastore服务器，URI之间用逗号分隔。metastore服务器URI的格式为thrift://hostort

下面配置方式为，将mac电脑作为hive的服务端程序，centos作为远程客户端程序来搭建

1.配置hive服务端程序配置

下面还是使用hive安装，内嵌模式及本地模式中的配置，

配置修改为mysql数据库，这样可以支持多会话连接,在conf目录下新建hive-site.xml文件并添加如下内容

cd /Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf

touch hive-hive-site.xml

vi hive-hive-site.xml


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

	<property>
	  <name>javax.jdo.option.ConnectionURL</name>
	  <value>jdbc:mysql://localhost:3306/hive_metastore?createDatabaseIfNotExist=true</value>
	  <description>mysql连接地址</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionDriverName</name>
	  <value>com.mysql.jdbc.Driver</value>
	  <description>mysql驱动类</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionUserName</name>
	  <value>hive</value>
	  <description>刚刚创建的用户名hive</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionPassword</name>
	  <value>123456</value>
	  <description>hive用户对应的mysql密码</description>
	</property>

</configuration>

同时需要讲mysql驱动程序拷贝到lib目录下，否则会报驱动程序找不到的错误 cp ../../cdh/hive-0.13.1-cdh5.3.6/lib/mysql-connector-java-5.1.27-bin.jar ./lib/

2.配置客户端程序

由于本人有使用centos虚拟机，因此这里直接使用centos虚拟机作为客户端来访问上面的mac配置的hive服务端程序

2.1将hive安装包拷贝到centos虚拟机中

在mac中拷贝hive安装程序到centos中，下面的192.168.36.129为centos的ip地址，hadoop为centos虚拟机用户名，拷贝到centos中的/home/hadoop/bigdata/software目录中

scp hive-0.13.1-cdh5.3.6.tar.gz hadoop@192.168.36.129:/home/hadoop/bigdata/software
hive-0.13.1-cdh5.3.6.tar.gz   100%   61MB  61.1MB/s   00:01 

2.2进入centos系统中进行相关配置

解压hive安装程序到cdh目录下
tar -zxvf hive-0.13.1-cdh5.3.6.tar.gz -C ../cdh/

查看centos中的java环境变量
echo $JAVA_HOME
/home/hadoop/bigdata/cdh/jdk1.7.0_67

cd /home/hadoop/bigdata/cdh/hive-0.13.1-cdh5.3.6

cp conf/hive-env.sh.template conf/hive-env.sh

cp conf/hive-default.xml.template conf/hive-site.xml

修改hive-env.sh,配置为centos中的hadoop安装目录

HADOOP_HOME=/home/hadoop/bigdata/cdh/hadoop-2.5.0-cdh5.3.6
export HIVE_CONF_DIR=/home/hadoop/bigdata/cdh/hive-0.13.1-cdh5.3.6/conf

修改hive-site.xml，注意这里的ip是mac的ip

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>
  
  <property>  
    <name>hive.metastore.warehouse.dir</name>  
    <value>/user/hive/warehouse</value>  
  </property>  
     
  <property>  
    <name>hive.metastore.local</name>  
    <value>false</value>  
  </property>  
    
  <property>  
    <name>hive.metastore.uris</name>  
    <value>thrift://172.31.218.249:9083</value>  
  </property>  

</configuration>

3.启动mac电脑的hdfs文件系统及hive的服务端程序

mac电脑上

启动hdfs服务		

```xml

cd software/cdh/hadoop-2.5.0-cdh5.3.6/

./sbin/start-dfs.sh

启动hive服务端metastore程序

./bin/hive --service metastore
Starting Hive Metastore Server

进入centos，启动hive客户端程序

[hadoop@bigdata1 hive-0.13.1-cdh5.3.6]$ ./bin/hive
16/10/29 19:06:37 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.

Logging initialized using configuration in jar:file:/home/hadoop/bigdata/cdh/hive-0.13.1-cdh5.3.6/lib/hive-common-0.13.1-cdh5.3.6.jar!/hive-log4j.properties
hive> show databases;
OK
default
Time taken: 0.59 seconds, Fetched: 1 row(s)
hive> use default;
OK
Time taken: 0.039 seconds
hive> show tables;
OK
sample_07
sample_08
xxx
Time taken: 0.067 seconds, Fetched: 3 row(s)
hive> 

在mac上启动hive客户端查看是否与centos看到的结果一致

mac:hive-0.13.1-cdh5.3.6 mac$ ./bin/hive

Logging initialized using configuration in file:/Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf/hive-log4j.properties
hive (default)> show databases;
OK
database_name
default
Time taken: 0.746 seconds, Fetched: 1 row(s)
hive (default)> use default;
OK
Time taken: 0.015 seconds
hive (default)> show tables;
OK
tab_name
sample_07
sample_08
xxx

到此搭建完毕

总结：

其实仅连接远程的mysql并不能称之为“远程模式”，是否远程指的是metastore和hive服务是否在同一进程内。

参考连接：

http://bi.dataguru.cn/thread-32711-1-1.html

zookeeper安装-单机版

2016-10-29T15:46:00+08:00

zookeeper安装-单机版

1.首先去cdh官网下载zookeeper

2.解压zookeeper,这里安装单机版

解压缩
tar -zxvf zookeeper-3.4.5-cdh5.3.6.tar.gz -C newcdh/
cd /Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/

赋值配置文件
cp conf/zoo_sample.cfg conf/zoo.cfg


创建数据目录
mkdir -p data/zookeeper

3.配置zoo.cfg文件的数据目录,修改后如下所示

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/data/zookeeper
# the port at which the clients will connect
clientPort=2181
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

4.启动zookeeper即可

./bin/zkServer.sh start

JMX enabled by default
Using config: /Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

验证是否启动成功

./bin/zkServer.sh status
JMX enabled by default
Using config: /Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Mode: standalone

或者通过zkCli.sh客户端验证

./bin/zkCli.sh
Connecting to localhost:2181
2016-10-29 15:57:47,450 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.3.6--1, built on 07/28/2015 22:11 GMT
2016-10-29 15:57:47,453 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=mac.cn
2016-10-29 15:57:47,453 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_91
2016-10-29 15:57:47,455 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2016-10-29 15:57:47,455 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre
2016-10-29 15:57:47,455 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../build/classes:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../build/lib/*.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../share/zookeeper/zookeeper-3.4.5-cdh5.3.6.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../share/zookeeper/slf4j-log4j12-1.7.5.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../share/zookeeper/slf4j-api-1.7.5.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../share/zookeeper/netty-3.2.2.Final.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../share/zookeeper/log4j-1.2.16.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../share/zookeeper/jline-0.9.94.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../src/java/lib/*.jar:/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf:
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/Users/mac/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/var/folders/0s/bccg6xrx6f31mql9grggj1mr0000gn/T/
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Mac OS X
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=x86_64
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=10.12.1
2016-10-29 15:57:47,456 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=mac
2016-10-29 15:57:47,457 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/Users/mac
2016-10-29 15:57:47,457 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6
2016-10-29 15:57:47,458 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@67424e82
Welcome to ZooKeeper!
2016-10-29 15:57:47,487 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2016-10-29 15:57:47,577 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:57346, server: localhost/0:0:0:0:0:0:0:1:2181
[zk: localhost:2181(CONNECTING) 0] 2016-10-29 15:57:47,620 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1580f7083be0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /zookeeper
[quota]
[zk: localhost:2181(CONNECTED) 2]

可视化工具zkdash

1.下载可视化工具zkdash

git clone https://github.com/ireaderlab/zkdash.git
cd zkdash
pip install -r requirements.txt

2.链接mysql数据库，并新建数据库zkdash

mysql -uroot -p123456

create database zkdash ;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| manage_test        |
| mysql              |
| performance_schema |
| sys                |
| zkdash             |
+--------------------+
6 rows in set (0.00 sec)

3.修改/Users/mac/software/bigdata/zkdash/conf下的conf.yml文件

修改数据库及用户名和密码

# database
DATABASE:
 db: 'zkdash'
 host: 'localhost'
 port: 3306
 user: 'root'
 passwd: '123456'

# 是否通过QConf获取zookeeper数据（使用该项需要在本地先运行QConf客户端agent）
USE_QCONF: False

# log conf
LOG_ITEMS:
  - file: /Users/mac/software/bigdata/zkdash/data/logs/zkdash/zkdash.log
    log_levels:
      - DEBUG
      - INFO
      - WARNING
      - ERROR
      - CRITICAL
    format: '[%(levelname)s %(asctime)s %(filename)s %(lineno)d] %(message)s'
    when: 'midnight'
    interval: 1
    backup_count: 10
    backup_suffix: '%Y%m%d'
    level: 'DEBUG'
  - file: /Users/mac/software/bigdata/zkdash/data/logs/zkdash/zkdash.error.log
    log_levels:
      - WARNING
      - ERROR
      - CRITICAL
    format: '[%(levelname)s %(asctime)s %(filename)s %(lineno)d] %(message)s'
    when: 'midnight'
    interval: 1
    backup_count: 30
    backup_suffix: '%Y%m%d'
    level: 'DEBUG'

4.初始化数据库

cd zkdash
sudo python bin/syncdb.py      

created table: zd_qconf_agent
created table: zd_qconf_feedback
created table: zd_snapshot
created table: zd_snapshot_tree
created table: zd_znode
created table: zd_zookeeper

5.运行

如果运行报错，可以加上sudo

python init.py -port=8888

访问页面http://localhost:8888通过zookeeper管理

hbase安装

2016-10-29T15:31:00+08:00

1.首先去cdh官网下载hbase

2.解压下载后的hbase-0.98.6-cdh5.3.6.tar.gz


tar -zxvf hbase-0.98.6-cdh5.3.6.tar.gz -C newcdh/

3.切换到conf目录下修改配置文件信息

tar -zxvf hbase-0.98.6-cdh5.3.6.tar.gz -C newcdh/

cd /Users/mac/software/newcdh/hbase-0.98.6-cdh5.3.6/conf

4.修改环境文件hbase-env.sh

修改下面几项信息

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home

默认在tmp目录下，避免因为重启导致问题
export HBASE_PID_DIR=/Users/mac/software/newcdh/hbase-0.98.6-cdh5.3.6/pids

默认为true，也就是说让hbase使用子集的zookeeper，我们这里不使用自己管理，我们交给zookeeper集群管理

export HBASE_MANAGES_ZK=false

5.修改regionservers文件，将所有的regionservers节点添加进来


mac.cn

6.创建本地数据目录

mkdir -p data/tmp

7.修改配置文件hbase-site.xml文件,配置如下所示

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>

  <property >
    <name>hbase.tmp.dir</name>
    <value>/Users/mac/software/newcdh/hbase-0.98.6-cdh5.3.6/data/tmp</value>
    <description>本地文件系统的临时目录.</description>
  </property>

  <property >
    <name>hbase.rootdir</name>
    <value>hdfs://mac.cn:8020/hbase</value>
    <description>region servers共享目录，hbase持久化的hdfs文件系统路径，需要完整路径.</description>
  </property>

  <property >
    <name>hbase.cluster.distributed</name>
    <value>true</value>
    <description>是否是分布式启动，如果不是分布式，则所有的hbase和zookeeper进程会在同一个jvm进程中启动
    </description>
  </property>

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>mac.cn</value>
    <description>配置zookeeper的集群访问节点</description>

  </property>

  <property>
    <name>hbase.regionserver.codecs</name>
    <value>gz</value>
  </property>

</configuration>

8.首先启动zookeeper节点,如果没有配置zookeeper的可以参考连接

./bin/zkServer.sh start
JMX enabled by default
Using config: /Users/mac/software/newcdh/zookeeper-3.4.5-cdh5.3.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

9.启动hbase进程

首先启动hmaster进程(关闭stop)   

./bin/hbase-daemon.sh start master
starting master, logging to /Users/mac/software/cdh/hbase-0.98.6-cdh5.3.6/logs/hbase-mac-master-mac.local.out

验证是否启动      

mac:hbase-0.98.6-cdh5.3.6 mac$ jps
12656 HMaster
11314 QuorumPeerMain
3906 DataNode
4005 SecondaryNameNode
12439 Main
12686 Jps
3823 NameNode

启动hregionserver进程(关闭stop)   
mac:hbase-0.98.6-cdh5.3.6 mac$ ./bin/hbase-daemon.sh start regionserver
starting regionserver, logging to /Users/mac/software/cdh/hbase-0.98.6-cdh5.3.6/logs/hbase-mac-regionserver-mac.local.out

验证是否启动
mac:hbase-0.98.6-cdh5.3.6 mac$ jps
12656 HMaster
11314 QuorumPeerMain
3906 DataNode
12756 Jps
4005 SecondaryNameNode
12439 Main
12732 HRegionServer
3823 NameNode

或者通过下面的方式启动

非后台进程启动(关闭stop)
./bin/hbase master start

非后台进程启动(关闭stop)
./bin/hbase regionserver start

10.打开hbase shell客户端查看是否启动成功，是否能够访问hbase的相关表数据

 ./bin/hbase shell
2016-10-29 16:19:44,597 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.98.6-cdh5.3.6, rUnknown, Tue Jul 28 15:17:11 PDT 2015

list_namespace
NAMESPACE                                                                       
default                                                                         
hbase                                                                           
2 row(s) in 0.1730 seconds

hbase(main):006:0> list
TABLE                                                                           
student                                                                         
1 row(s) in 0.0280 seconds

=> ["student"]

hive安装，内嵌模式及本地模式

2016-10-29T13:04:00+08:00

hive内嵌安装步骤

内嵌模式，特点是：hive服务和metastore服务运行在同一个进程中，derby服务也运行在该进程中。该模式无需特殊配置

1.首先去cdh官网下载hive

2.解压缩,并切换到hive配置文件目录中

tar -zxvf hive-0.13.1-cdh5.3.6.tar.gz -C newcdh/

cp hive-log4j.properties.template hive-log4j.properties
cp hive-env.sh.template hive-env.sh

3.修改hive-env.sh文件，设置下面的两行配置，HADOOP_HOME和HIVE_CONF_DIR配置

HADOOP_HOME=/Users/mac/software/cdh/hadoop-2.5.0-cdh5.3.6/
export HIVE_CONF_DIR=/Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf

4.这时候可以直接执行启动hive命令了

mac:hive-0.13.1-cdh5.3.6 mac$ ./bin/hive

Logging initialized using configuration in file:/Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf/hive-log4j.properties
hive> 

5.启动成功后，执行命令验证是否安装成功。

hive> show databases;
OK
default
Time taken: 0.539 seconds, Fetched: 1 row(s)
hive> use default;
OK
Time taken: 0.021 seconds
hive> show tables;
OK
Time taken: 0.037 seconds

6.同时查看hive目录下生成了metastore_db目录结构,这是由于hive默认使用derby数据库，因此在启动过程中使用derby数据库存储表结构相关信息,注意，这里的derby数据库是单会话的，一次只能有一个连接，并且derby数据库会在当前目录下生成

hive本地安装配置及高级配置

本地模式，特点是：hive服务和metastore服务运行在同一个进程中，mysql是单独的进程，可以在同一台机器上，也可以在远程机器上。

7.下面配置修改为mysql数据库，这样可以支持多会话连接,在conf目录下新建hive-site.xml文件并添加如下内容

cd /Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf

touch hive-hive-site.xml

vi hive-hive-site.xml


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

	<property>
	  <name>javax.jdo.option.ConnectionURL</name>
	  <value>jdbc:mysql://localhost:3306/hive_metastore?createDatabaseIfNotExist=true</value>
	  <description>mysql连接地址</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionDriverName</name>
	  <value>com.mysql.jdbc.Driver</value>
	  <description>mysql驱动类</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionUserName</name>
	  <value>hive</value>
	  <description>刚刚创建的用户名hive</description>
	</property>

	<property>
	  <name>javax.jdo.option.ConnectionPassword</name>
	  <value>123456</value>
	  <description>hive用户对应的mysql密码</description>
	</property>

</configuration>

同时需要讲mysql驱动程序拷贝到lib目录下，否则会报驱动程序找不到的错误 cp ../../cdh/hive-0.13.1-cdh5.3.6/lib/mysql-connector-java-5.1.27-bin.jar ./lib/ 重启hive，验证

mac:hive-0.13.1-cdh5.3.6 mac$ ./bin/hive

Logging initialized using configuration in file:/Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf/hive-log4j.properties
hive> show databases;
OK
default
Time taken: 0.484 seconds, Fetched: 1 row(s)
hive> use default;
OK
Time taken: 0.02 seconds
hive> show tables;
OK
sample_07
sample_08
xxx

这里便是因为hive使用了我们配置的mysql数据库，mysql数据库中原来配置过一次hive表，里面有之前建立的表,由此验证配置mysql成功这时查询表数据会报错，因为没有启动hdfs服务

select * from xxx;
FAILED: SemanticException Unable to determine if hdfs://mac.cn:8020/user/hive/warehouse/xxx is encrypted: java.net.ConnectException: Call From mac.local/172.31.218.249 to mac.cn:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

启动hdfs服务

cd software/cdh/hadoop-2.5.0-cdh5.3.6/

./sbin/start-dfs.sh 

再次查询后

hive> select * from xxx;
OK

Time taken: 0.439 seconds

8.配置在查询结果中显示列名称

在hive-site.xml中配置

<property>
	<name>hive.cli.print.header</name>
	<value>true</value>
	<description>Whether to print the names of the columns in query output.</description>
</property>

重新进入hive命令行验证是否成功

hive> select * from xxx;
OK
xxx.id
Time taken: 0.524 seconds

9.配置默认显示当前所在的数据库

在hive-site.xml中添加

<property>
	<name>hive.cli.print.current.db</name>
	<value>true</value>
	<description>Whether to include the current database in the Hive prompt.</description>
</property>

重新进入hive命令行验证是否成功

mac:hive-0.13.1-cdh5.3.6 mac$ ./bin/hive

Logging initialized using configuration in file:/Users/mac/software/newcdh/hive-0.13.1-cdh5.3.6/conf/hive-log4j.properties
hive (default)> show databases;
OK
database_name
default
Time taken: 0.484 seconds, Fetched: 1 row(s)
hive (default)> 

nginx+flume+hadoop实战

2016-10-29T09:03:00+08:00

实战说明

本文通过nginx服务器收集访问日志，并通过flume实时抽取日志到hdfs文件系统中

nginx安装

1.安装nginx

brew install nginx

Docroot is: /usr/local/var/www

The default port has been set in /usr/local/etc/nginx/nginx.conf to 8080 so that
nginx can run without sudo.

nginx will load all files in /usr/local/etc/nginx/servers/.

To have launchd start nginx now and restart at login:
  brew services start nginx
Or, if you don't want/need a background service you can just run:
  nginx
==> Summary
🍺  /usr/local/Cellar/nginx/1.10.2_1

启动浏览器查看8080端口是否成功，http://localhost:8080

2.这里使用自己的github个人网站，直接将编译过的_site文件夹拷贝到nginx的html目录下(不懂的可以网上搜索)

cd /usr/local/Cellar/nginx/1.10.2_1/html

mac:html mac$ cp -r /Users/mac/software/github/macaca/CodeToSurvive1.github.io/_site/* ./
mac:html mac$ ls
404.html	about.html	atom.xml	faqs.html	index.html	macaca.html	tags.html
CNAME		archives.html	categories.html	favicon.ico	js		posts
README.md	assets		css		fonts		links.html	robots.txt

3.修改nginx配置文件，添加日志输出格式及日志输出文件

添加日志格式


log_format  lf  '$remote_addr^A$msec^A$http_host^A$request_uri';

在nginx目录下新建日志文件夹

/usr/local/Cellar/nginx/1.10.2_1
pwd

/usr/local/Cellar/nginx/1.10.2_1

mkdir logs
cd logs/
pwd
/usr/local/Cellar/nginx/1.10.2_1/logs

添加日志输出文件


access_log  /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log  lf;

修改后的nginx.conf配置文件如下所示

worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  lf  '$remote_addr^A$msec^A$http_host^A$request_uri';
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       8080;
        server_name  localhost;
        access_log  /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log  lf;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
    include servers/*;
}

4.将nginx进行平滑重启


nginx -s reload

5.访问页面并查看日志文件中的输出数据

mac:logs mac$ tail -f host.access.log
0.0.1^A1477709204.499^Alocalhost:8080^A/fonts/glyphicons-halflings-regular.ttf
0.0.1^A1477709205.588^Alocalhost:8080^A/favicon.ico
0.0.1^A1477709231.377^Alocalhost:8080^A/posts/nginx-flume-hadoop.html
0.0.1^A1477709231.392^Alocalhost:8080^A/css/bootstrap.min.css
0.0.1^A1477709231.392^Alocalhost:8080^A/css/blog.css
0.0.1^A1477709231.392^Alocalhost:8080^A/css/prettify.css
0.0.1^A1477709231.392^Alocalhost:8080^A/js/jquery-1.11.0.js
0.0.1^A1477709231.392^Alocalhost:8080^A/js/bootstrap.min.js
0.0.1^A1477709231.392^Alocalhost:8080^A/js/run_prettify.js
0.0.1^A1477709231.491^Alocalhost:8080^A/fonts/glyphicons-halflings-regular.ttf







0.0.1^A1477709735.875^Alocalhost:8080^A/
0.0.1^A1477709735.889^Alocalhost:8080^A/css/bootstrap.min.css
0.0.1^A1477709735.901^Alocalhost:8080^A/css/blog.css
0.0.1^A1477709735.901^Alocalhost:8080^A/js/bootstrap.min.js
0.0.1^A1477709735.902^Alocalhost:8080^A/js/run_prettify.js
0.0.1^A1477709735.903^Alocalhost:8080^A/js/jquery-1.11.0.js
0.0.1^A1477709735.903^Alocalhost:8080^A/css/prettify.css
0.0.1^A1477709736.785^Alocalhost:8080^A/fonts/glyphicons-halflings-regular.ttf
0.0.1^A1477709742.728^Alocalhost:8080^A/posts/nginx-flume-hadoop.html
0.0.1^A1477709748.243^Alocalhost:8080^A/posts/macaca-plugin-develop.html
0.0.1^A1477709749.219^Alocalhost:8080^A/posts/mac-redis-install.html
0.0.1^A1477709750.355^Alocalhost:8080^A/posts/mcaca-master-slave-no-docker.html
0.0.1^A1477709750.416^Alocalhost:8080^A/assets/2016-10-28_14-00-51.png

flume环境搭建

1.下载flume

由于本人使用的cdh版本的hadoop，因此这里配套下载cdh版本的flume

2.解压缩即可

tar -zxvf flume-ng-1.5.0-cdh5.3.6.tar.gz  
cd apache-flume-1.5.0-cdh5.3.6-bin/
pwd
/Users/mac/software/apache-flume-1.5.0-cdh5.3.6-bin

3.编写flume文件

#定义source，channels，sinks三大组件名称
agent.sources = tailSource
agent.channels = memoryChannel
agent.sinks = hdfsSink logger

#定义source的属性为exec执行程序
agent.sources.tailSource.type =exec
agent.sources.tailSource.channels = memoryChannel
agent.sources.tailSource.command=tail -f /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log


#定义sinks的属性为hdfs相关配置
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = hdfs://mac.cn:8020/user/mac/logs/%Y/%m/%d/
agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true


#定义sinks的属性为hdfs相关配置
agent.sinks.logger.type = logger
agent.sinks.logger.channel = memoryChannel

#定义channels的属性为内存相关配置
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

4.如果上面不加agent.sinks.hdfsSink.hdfs.useLocalTimeStamp = true这句话会在启动过程中报错，如下

2016-10-29 12:23:33,775 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:467)] process failed
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
	at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:224)
	at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:422)
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:388)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
2016-10-29 12:23:33,775 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:23:33,777 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:471)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
	at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:224)
	at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:422)
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:388)
	... 3 more

5.启动flume

不要忘记首先启动hadoop相关进程

./bin/flume-ng agent -n agent -c conf/ -Dflume.root.logger=INFO,console -f conf/flume-conf.properties

2016-10-29 12:27:44,524 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/flume-conf.xml
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:logger
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:931)] Added sinks: hdfsSink logger Agent: agent
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:27:44,532 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:logger
2016-10-29 12:27:44,549 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:141)] Post-validation flume configuration contains configuration for agents: [agent]
2016-10-29 12:27:44,549 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:145)] Creating channels
2016-10-29 12:27:44,557 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel memoryChannel type memory
2016-10-29 12:27:44,564 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel memoryChannel
2016-10-29 12:27:44,566 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source tailSource, type exec
2016-10-29 12:27:44,580 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: logger, type: logger
2016-10-29 12:27:44,582 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: hdfsSink, type: hdfs
2016-10-29 12:27:44,869 (conf-file-poller-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-10-29 12:27:45,079 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:559)] Hadoop Security enabled: false
2016-10-29 12:27:45,082 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel memoryChannel connected to [tailSource, logger, hdfsSink]
2016-10-29 12:27:45,093 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{tailSource=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:tailSource,state:IDLE} }} sinkRunners:{logger=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3cc13c74 counterGroup:{ name:null counters:{} } }, hdfsSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7e0d961c counterGroup:{ name:null counters:{} } }} channels:{memoryChannel=org.apache.flume.channel.MemoryChannel{name: memoryChannel}} }
2016-10-29 12:27:45,098 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel memoryChannel
2016-10-29 12:27:45,164 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: CHANNEL, name: memoryChannel: Successfully registered new MBean.
2016-10-29 12:27:45,165 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: memoryChannel started
2016-10-29 12:27:45,169 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink logger
2016-10-29 12:27:45,170 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink hdfsSink
2016-10-29 12:27:45,170 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source tailSource
2016-10-29 12:27:45,171 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:169)] Exec source starting with command:tail -f /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log
2016-10-29 12:27:45,172 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: hdfsSink: Successfully registered new MBean.
2016-10-29 12:27:45,172 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: hdfsSink started
2016-10-29 12:27:45,175 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SOURCE, name: tailSource: Successfully registered new MBean.
2016-10-29 12:27:45,175 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SOURCE, name: tailSource started
2016-10-29 12:27:49,179 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,180 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,180 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,180 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,180 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,181 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,181 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,181 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,181 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:27:49,189 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSSequenceFile.configure(HDFSSequenceFile.java:63)] writeFormat = Writable, UseRawLocalFileSystem = false
2016-10-29 12:27:49,231 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:261)] Creating hdfs://mac.cn:8020/user/mac/logs/2016/10/29//FlumeData.1477715269187.tmp


2016-10-29 12:28:15,180 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/flume-conf.xml
2016-10-29 12:28:15,181 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:logger
2016-10-29 12:28:15,224 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:28:15,224 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:28:15,224 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:28:15,225 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:931)] Added sinks: hdfsSink logger Agent: agent
2016-10-29 12:28:15,225 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:hdfsSink
2016-10-29 12:28:15,228 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:logger
2016-10-29 12:28:15,235 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:141)] Post-validation flume configuration contains configuration for agents: [agent]
2016-10-29 12:28:15,235 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:145)] Creating channels
2016-10-29 12:28:15,236 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel memoryChannel
2016-10-29 12:28:15,236 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source tailSource, type exec
2016-10-29 12:28:15,237 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: logger, type: logger
2016-10-29 12:28:15,237 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: hdfsSink, type: hdfs
2016-10-29 12:28:15,237 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:559)] Hadoop Security enabled: false
2016-10-29 12:28:15,237 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel memoryChannel connected to [tailSource, logger, hdfsSink]
2016-10-29 12:28:15,238 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:101)] Shutting down configuration: { sourceRunners:{tailSource=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:tailSource,state:START} }} sinkRunners:{logger=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3cc13c74 counterGroup:{ name:null counters:{runner.backoffs.consecutive=5, runner.backoffs=6} } }, hdfsSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7e0d961c counterGroup:{ name:null counters:{runner.backoffs.consecutive=4, runner.backoffs=5} } }} channels:{memoryChannel=org.apache.flume.channel.MemoryChannel{name: memoryChannel}} }
2016-10-29 12:28:15,238 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:105)] Stopping Source tailSource
2016-10-29 12:28:15,238 (conf-file-poller-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:171)] Stopping component: EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:tailSource,state:START} }
2016-10-29 12:28:15,241 (conf-file-poller-0) [INFO - org.apache.flume.source.ExecSource.stop(ExecSource.java:192)] Stopping exec source with command:tail -f /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log
2016-10-29 12:28:15,242 (pool-5-thread-1) [INFO - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:376)] Command [tail -f /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log] exited with 143
2016-10-29 12:28:15,243 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:150)] Component type: SOURCE, name: tailSource stopped
2016-10-29 12:28:15,243 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:156)] Shutdown Metric for type: SOURCE, name: tailSource. source.start.time == 1477715265175
2016-10-29 12:28:15,243 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:162)] Shutdown Metric for type: SOURCE, name: tailSource. source.stop.time == 1477715295243
2016-10-29 12:28:15,244 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.append-batch.accepted == 0
2016-10-29 12:28:15,244 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.append-batch.received == 0
2016-10-29 12:28:15,244 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.append.accepted == 0
2016-10-29 12:28:15,244 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.append.received == 0
2016-10-29 12:28:15,244 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.events.accepted == 10
2016-10-29 12:28:15,245 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.events.received == 10
2016-10-29 12:28:15,245 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SOURCE, name: tailSource. src.open-connection.count == 0
2016-10-29 12:28:15,245 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:115)] Stopping Sink logger
2016-10-29 12:28:15,245 (conf-file-poller-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:171)] Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3cc13c74 counterGroup:{ name:null counters:{runner.backoffs.consecutive=5, runner.backoffs=6} } }
2016-10-29 12:28:15,246 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:115)] Stopping Sink hdfsSink
2016-10-29 12:28:15,246 (conf-file-poller-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:171)] Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7e0d961c counterGroup:{ name:null counters:{runner.backoffs.consecutive=4, runner.backoffs=5} } }
2016-10-29 12:28:15,246 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink.stop(HDFSEventSink.java:500)] Closing hdfs://mac.cn:8020/user/mac/logs/2016/10/29//FlumeData
2016-10-29 12:28:15,247 (conf-file-poller-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:390)] Closing hdfs://mac.cn:8020/user/mac/logs/2016/10/29//FlumeData.1477715269187.tmp
2016-10-29 12:28:15,279 (hdfs-hdfsSink-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:656)] Renaming hdfs://mac.cn:8020/user/mac/logs/2016/10/29/FlumeData.1477715269187.tmp to hdfs://mac.cn:8020/user/mac/logs/2016/10/29/FlumeData.1477715269187
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:150)] Component type: SINK, name: hdfsSink stopped
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:156)] Shutdown Metric for type: SINK, name: hdfsSink. sink.start.time == 1477715265172
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:162)] Shutdown Metric for type: SINK, name: hdfsSink. sink.stop.time == 1477715295288
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.batch.complete == 0
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.batch.empty == 5
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.batch.underflow == 1
2016-10-29 12:28:15,288 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.connection.closed.count == 1
2016-10-29 12:28:15,289 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.connection.creation.count == 1
2016-10-29 12:28:15,289 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.connection.failed.count == 0
2016-10-29 12:28:15,289 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.event.drain.attempt == 1
2016-10-29 12:28:15,289 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: SINK, name: hdfsSink. sink.event.drain.sucess == 1
2016-10-29 12:28:15,289 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.stopAllComponents(Application.java:125)] Stopping Channel memoryChannel
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise(LifecycleSupervisor.java:171)] Stopping component: org.apache.flume.channel.MemoryChannel{name: memoryChannel}
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:150)] Component type: CHANNEL, name: memoryChannel stopped
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:156)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.start.time == 1477715265165
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:162)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.stop.time == 1477715295290
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.capacity == 100
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.current.size == 0
2016-10-29 12:28:15,290 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.event.put.attempt == 10
2016-10-29 12:28:15,291 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.event.put.success == 10
2016-10-29 12:28:15,291 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.event.take.attempt == 22
2016-10-29 12:28:15,291 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.stop(MonitoredCounterGroup.java:178)] Shutdown Metric for type: CHANNEL, name: memoryChannel. channel.event.take.success == 10
2016-10-29 12:28:15,291 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{tailSource=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:tailSource,state:IDLE} }} sinkRunners:{logger=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2b01ef06 counterGroup:{ name:null counters:{} } }, hdfsSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@413efc8b counterGroup:{ name:null counters:{} } }} channels:{memoryChannel=org.apache.flume.channel.MemoryChannel{name: memoryChannel}} }
2016-10-29 12:28:15,300 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel memoryChannel
2016-10-29 12:28:15,300 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:160)] Waiting for channel: memoryChannel to start. Sleeping for 500 ms
2016-10-29 12:28:15,300 (lifecycleSupervisor-1-6) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: memoryChannel started
2016-10-29 12:28:15,804 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink logger
2016-10-29 12:28:15,804 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink hdfsSink
2016-10-29 12:28:15,804 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source tailSource
2016-10-29 12:28:15,805 (lifecycleSupervisor-1-9) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:169)] Exec source starting with command:tail -f /usr/local/Cellar/nginx/1.10.2_1/logs/host.access.log
2016-10-29 12:28:15,805 (lifecycleSupervisor-1-7) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: hdfsSink: Successfully registered new MBean.
2016-10-29 12:28:15,806 (lifecycleSupervisor-1-7) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: hdfsSink started
2016-10-29 12:28:15,806 (lifecycleSupervisor-1-9) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SOURCE, name: tailSource: Successfully registered new MBean.
2016-10-29 12:28:15,806 (lifecycleSupervisor-1-9) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SOURCE, name: tailSource started
2016-10-29 12:28:19,813 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,813 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,814 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,814 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,814 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,815 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,814 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSSequenceFile.configure(HDFSSequenceFile.java:63)] writeFormat = Writable, UseRawLocalFileSystem = false
2016-10-29 12:28:19,815 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,816 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,816 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 31 32 37 2E 30 2E 30 2E 31 5E 41 31 34 37 37 37 127.0.0.1^A14777 }
2016-10-29 12:28:19,861 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:261)] Creating hdfs://mac.cn:8020/user/mac/logs/2016/10/29//FlumeData.1477715299815.tmp
2016-10-29 12:28:49,991 (hdfs-hdfsSink-roll-timer-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:390)] Closing hdfs://mac.cn:8020/user/mac/logs/2016/10/29//FlumeData.1477715299815.tmp
2016-10-29 12:28:50,008 (hdfs-hdfsSink-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:656)] Renaming hdfs://mac.cn:8020/user/mac/logs/2016/10/29/FlumeData.1477715299815.tmp to hdfs://mac.cn:8020/user/mac/logs/2016/10/29/FlumeData.1477715299815
2016-10-29 12:28:50,014 (hdfs-hdfsSink-roll-timer-0) [INFO - org.apache.flume.sink.hdfs.HDFSEventSink$1.run(HDFSEventSink.java:402)] Writer callback called.

打开8080端口进行访问页面，后台日志会刷新数据，hdfs文件系统中新增文件如下：

如果执行过程中报hdfs相关错误，估计是缺少hdfs相关的jar包，拷贝下面包到flume目录下的lib包目录中即可

6.将hadoop的相关包放进lib目录下

cd /Users/mac/software/apache-flume-1.5.0-cdh5.3.6-bin/lib  

cp ~/software/cdh/hadoop-2.5.0-cdh5.3.6/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.6.jar ./

cp ~/software/cdh/hadoop-2.5.0-cdh5.3.6/share/hadoop/common/lib/commons-configuration-1.6.jar ./

cp ~/software/cdh/hadoop-2.5.0-cdh5.3.6/share/hadoop/common/lib/hadoop-auth-2.5.0-cdh5.3.6.jar ./

cp ~/software/cdh/hadoop-2.5.0-cdh5.3.6/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.6.jar ./

mac安装redis

2016-10-28T12:35:00+08:00

1.安装wget

mac:macaca mac$ brew install wget
==> Downloading https://homebrew.bintray.com/bottles/wget-1.18.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring wget-1.18.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/wget/1.18: 9 files, 1.6M

2.下载redis

wget http://download.redis.io/releases/redis-3.2.5.tar.gz
tar xzf redis-3.2.5.tar.gz
cd redis-3.2.5
make

3.启动reddis服务即可

src/redis-server

 20791:C 28 Oct 12:34:56.784 # Warning: no config file specified, using the default config. In order to specify a config file use src/redis-server /path/to/redis.conf
20791:M 28 Oct 12:34:56.785 * Increased maximum number of open files to 10032 (it was originally set to 2560).
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.2.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 20791
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

20791:M 28 Oct 12:34:56.786 # Server started, Redis version 3.2.5
20791:M 28 Oct 12:34:56.786 * The server is now ready to accept connections on port 6379

4.打开redis客户端测试

 ./src/redis-cli

127.0.0.1:6379> set 'hello' 'wrold'
OK
127.0.0.1:6379> get 'hello'
"wrold"
127.0.0.1:6379>

mac系统搭建master-slave环境搭建(非docker版本)

2016-10-28T10:57:00+08:00

配置mongodb数据库

1.首先安装mongodb，教程参考

https://codetosurvive1.github.io/posts/mac-mongodb-install.html

2.安装完成后，首先启动mongodb服务端,这里启动如下


mongod --dbpath software/mongodb/data/

3.首先从github上下载reliable-macaca-docker-compose，然后进入后，解压backup.tar.gz文件，并恢复数据库

下载reliable-macaca-docker-compose，获取表结构数据  


git clone https://github.com/reliablejs/reliable-macaca-docker-compose.git
Cloning into 'reliable-macaca-docker-compose'...
remote: Counting objects: 37, done.
remote: Total 37 (delta 0), reused 0 (delta 0), pack-reused 37
Unpacking objects: 100% (37/37), done.
Checking connectivity... done.

解压数据结构文件
mac:reliable-macaca-docker-compose mac$ pwd
/Users/mac/software/github/macaca/reliable-macaca-docker-compose
tar -zxvf backup.tar.gz
x backup/
x backup/reliable/
x backup/reliable/projects.bson
x backup/reliable/projects.metadata.json
x backup/reliable/subscribes.bson
x backup/reliable/subscribes.metadata.json
x backup/reliable/system.indexes.bson
x backup/reliable/tasks.bson
x backup/reliable/tasks.metadata.json
x backup/reliable/users.bson
x backup/reliable/users.metadata.json

恢复数据库到本地安装的mongodb数据库中

mongorestore -d reliable backup/reliable
2016-10-28T11:23:15.360+0800	building a list of collections to restore from backup/reliable dir
2016-10-28T11:23:15.361+0800	reading metadata for reliable.projects from backup/reliable/projects.metadata.json
2016-10-28T11:23:15.361+0800	reading metadata for reliable.tasks from backup/reliable/tasks.metadata.json
2016-10-28T11:23:15.362+0800	reading metadata for reliable.users from backup/reliable/users.metadata.json
2016-10-28T11:23:15.362+0800	reading metadata for reliable.subscribes from backup/reliable/subscribes.metadata.json
2016-10-28T11:23:15.414+0800	restoring reliable.projects from backup/reliable/projects.bson
2016-10-28T11:23:15.465+0800	restoring reliable.tasks from backup/reliable/tasks.bson
2016-10-28T11:23:15.513+0800	restoring reliable.subscribes from backup/reliable/subscribes.bson
2016-10-28T11:23:15.513+0800	restoring indexes for collection reliable.subscribes from metadata
2016-10-28T11:23:15.568+0800	restoring reliable.users from backup/reliable/users.bson
2016-10-28T11:23:15.568+0800	restoring indexes for collection reliable.tasks from metadata
2016-10-28T11:23:15.569+0800	restoring indexes for collection reliable.projects from metadata
2016-10-28T11:23:15.644+0800	restoring indexes for collection reliable.users from metadata
2016-10-28T11:23:15.646+0800	finished restoring reliable.tasks (4 documents)
2016-10-28T11:23:15.647+0800	finished restoring reliable.subscribes (0 documents)
2016-10-28T11:23:15.647+0800	finished restoring reliable.projects (1 document)
2016-10-28T11:23:15.710+0800	finished restoring reliable.users (1 document)
2016-10-28T11:23:15.711+0800	done

切换到reliable数据库
> use reliable
switched to db reliable

查看reliable的所有表结构

> show tables
projects
subscribes
tasks
users
>

4.验证表结构数据是否成功，新开终端打开mongo客户端

打开mongo客户端
mongo

显示数据库结构
show databases;
local     0.000GB
reliable  0.000GB

切换到reliable数据库
use reliable

显示表
show tables;
projects
subscribes
tasks
users

配置redis

1.首先按照安装教程进行安装

2.启动redis


src/redis-server

配置reliable-master

1.首先下载最新版reliable-master


git clone https://github.com/reliablejs/reliable-master.git
Cloning into 'reliable-master'...
remote: Counting objects: 956, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 956 (delta 2), reused 0 (delta 0), pack-reused 942
Receiving objects: 100% (956/956), 286.07 KiB | 72.00 KiB/s, done.
Resolving deltas: 100% (512/512), done.
Checking connectivity... done.

2.编译项目

make build  


最后出现
Hash: 0570f1997352375ae79c
Version: webpack 1.13.3
Time: 3629ms
       Asset     Size  Chunks             Chunk Names
     auth.js  3.78 kB       0  [emitted]  auth
dashboard.js  1.14 MB       1  [emitted]  dashboard
  history.js  1.14 MB       2  [emitted]  history
     home.js  1.14 MB       3  [emitted]  home
    index.js  2.42 kB       4  [emitted]  index
    login.js  3.34 kB       5  [emitted]  login
 password.js  4.49 kB       6  [emitted]  password
   signup.js  3.99 kB       7  [emitted]  signup
     task.js  1.14 MB       8  [emitted]  task
     user.js   1.4 kB       9  [emitted]  user
    + 191 hidden modules
编译成功

3.启动服务

make server

./bin/reliable-master server -p 3333 -w 1 --verbose

*****************************************
*  version 1.0.0 is outdate             *
*  run: npm i -g reliable-master@1.0.8  *
*****************************************

>> index.js:44:14 [master] pid:2244 worker fork success
>> index.js:62:14 [master] pid:2244 worker online
>> index.js:21:10 [worker:1] pid:2246 workder init with config:
 { server: { worker: '1', port: '3333', protocol: 'http', verbose: true },
  database: 'mongodb://localhost/reliable',
  site:
   { title: 'Reliable',
     baseurl: 'http://reliable-test.com',
     locale: 'en-US',
     docurl: 'https://macacajs.github.io/macaca',
     issueurl: 'https://github.com/reliablejs/reliable-master',
     login: true },
  auth:
   { github:
      { client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' },
     gitlab:
      { protocol: 'http',
        server_url: '127.0.0.1:3000',
        client_id: '8bb3d4f7fa7d3d346a58',
        client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' } },
  mail:
   { name: 'reliable test',
     port: 465,
     host: 'smtp.reliable-test.com',
     secure: true,
     ignoreTLS: true,
     auth: { user: 'test@reliable-test.com', pass: 'reliable' },
     sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200' },
  plugins: {},
  pluginTextPrefix: 'reliable-plugin',
  pkg:
   { name: 'reliable-master',
     version: '1.0.8',
     description: 'Distributed reliable continuous integration service.',
     keywords: [ 'ci', 'automation', [length]: 2 ],
     bin: { 'reliable-master': './bin/reliable-master' },
     repository:
      { type: 'git',
        url: 'git@github.com:reliablejs/reliable-master.git' },
     dependencies:
      { 'ansi-to-html': '~0.3.0',
        babel: '~5.6.14',
        badgeboard: '~0.1.1',
        bcryptjs: '~2.2.1',
        chalk: '~1.1.1',
        co: '~4.6',
        'co-body': '~1.1.0',
        'co-request': '^0.2.1',
        commander: '~2.0.0',
        'detect-port': '~0.1.3',
        koa: '~0.21.0',
        'koa-csrf': '2.1.3',
        'koa-favicon': '~1.2.0',
        'koa-generic-session': '~1.10.0',
        'koa-redis': '~1.0.1',
        'koa-router': '~3.7.0',
        'koa-static': '~1.4.7',
        mongoose: '=4.1.5',
        nodemailer: '~1.4.0',
        'npm-update': '1.0.2',
        react: '~0.14.0',
        'react-dom': '~0.14.2',
        redis: '~2.3.1',
        'reliable-events': '^0.1.4',
        'reliable-github-oauth': '^1.0.0',
        'reliable-gitlab-oauth': '^1.0.0',
        'reliable-logger': '^1.0.7',
        'reliable-mail': '^0.2.0',
        'reliable-plugin': '^0.1.8',
        validator: '~3.37.0',
        xutil: '~1.0.0',
        yamljs: '~0.2.4',
        zmq: '2.13.0' },
     devDependencies:
      { 'co-mocha': '~1.1.2',
        'command-line-test': '^1.0.5',
        eslint: '~1.1.0',
        'eslint-plugin-react': '~3.2.3',
        istanbul: '*',
        'json-loader': '~0.5.2',
        'jsx-loader': '~0.13.2',
        killing: '~1.0.1',
        mocha: '~2.4.5',
        'pre-commit': '~1.1.1',
        'react-d3-components': '~0.6.0',
        should: '*',
        webpack: '^1.13.1' },
     scripts: { test: 'make test', lint: 'make lint' },
     'pre-commit': [ 'lint', [length]: 1 ],
     engines: { node: '>=4.2.1' },
     homepage: 'https://reliablejs.github.io/reliable',
     license: 'MIT' } }
>> index.js:32:10 [worker:1] pid:2246 base middlewares attached
>> index.js:36:12 [worker:1] pid:2246 middleware: i18n registed
>> render.js:15:8 [worker:1] pid:2246 render view path: /Users/mac/software/github/macaca/reliable-master/web/views
>> index.js:36:12 [worker:1] pid:2246 middleware: inject registed
>> index.js:36:12 [worker:1] pid:2246 middleware: favicon registed
>> index.js:36:12 [worker:1] pid:2246 middleware: powerby registed
>> index.js:36:12 [worker:1] pid:2246 middleware: static registed
>> router.js:41:10 [worker:1] pid:2246 router set
>> index.js:67:14 [master] pid:2244 listening worker id: 1, pid: 2246, address: 172.31.218.249:3333
>> Server start at 2016-10-28 13:17:03 | http://172.31.218.249:3333

4.查看页面http://172.31.218.249:3333

发现是英文版本，断掉服务修改文件
reliable-master/common/config.j文件中的
locale: ‘en-US’,为locale: ‘zh-CN’,

重新编译并启动

make build

make server

5.创建管理员用户

mac:reliable-master mac$ make adduser
./bin/reliable-master adduser

*****************************************
*  version 1.0.0 is outdate             *
*  run: npm i -g reliable-master@1.0.8  *
*****************************************


 Do you want create an account? [Y/N]Y

  User Id: root

  Email address: lixiaojiao_hit@163.com

  Password: ********

  isAdmin [Y/N]: Y

  Add user success!

  Goodbye! Have a great day!

6.接入slave节点,这里注意，master跟slave都没使用docker，master的端口为3333,对于slave的配置请参考docker篇中的讲解，连接


mac:reliable-master mac$ reliable server -m mac.cn:3333 --verbose

7.使用管理员登录后，查看页面信息如下

如何创建分支并提交代码

2016-10-27T08:05:00+08:00

如何创建分支并提交代码

1.首先查看分支信息


git branch
* create
  dev
  master

2.创建分支test


git branch test

查看分支是否创建成功  
git branch

* create
  dev
  master
  test

3.切换分支到test


git checkout test
Switched to branch 'test'

查看是否切换成功
git branch
  create
  dev
  master
* test

4.修改文件信息，比如这里添加一个文件


touch test
ls
macaca-test	package.json	test

5.由于添加了新文件，所以需要提交新文件上去

添加当前目录
git add .

提交到本地仓库
git commit -am '添加test文件'
[test 4d485cd] 添加test文件
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test

推送到远程仓库
git push origin test
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 286 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local objects.
To https://github.com/CodeToSurvive1/bank_sample.git
 * [new branch]      test -> test

6.直接通过web页面管理器查看是否已经提交到git上

7.如果想删除本地分支或者远程仓库分支


git branch -D test
Deleted branch test (was e7c582b).

删除远程仓库分支

$ git push origin --delete test
To https://github.com/CodeToSurvive1/bank_sample.git
 - [deleted]         test

mapreduce程序如何设置没有reducer

2016-10-24T19:05:00+08:00

两种设置方法

1.直接在job中设置

job.setNumReduceTasks(0);

2.通过configuration设置

conf.set(“mapred.reduce.tasks”,”0”);

macaca持续集成框架CI安装master与slave环境配置

2016-10-24T14:43:00+08:00

安装docker

直接下载安装 https://download.docker.com/mac/stable/Docker.dmg

安装master

1.安装reliable-docker-base基础镜像

git clone https://github.com/reliablejs/reliable-docker-base --depth=1
cd reliable-docker-base
docker build -t="reliable-docker-base" .

2.安装reliable-master镜像

git clone https://github.com/reliablejs/reliable-master --depth=1
cd reliable-master
docker build -t="reliable-master" .

3.安装reliable-macaca-docker-compose

git clone https://github.com/reliablejs/reliable-macaca-docker-compose.git --depth=1
cd reliable-macaca-docker-compose
make start

注意：

在make start之前需要修改docker-compose.yml文件，注释掉下面的代码

# slave_1:
#   container_name: "slave_1"
#   restart: always
#   image: "reliable-macaca-slave"
#   links:
#     - reliable-master:reliable-master
#   volumes:
#     - /etc/localtime:/etc/localtime:ro
#   command: /reliable-macaca-slave/bin/reliable-macaca-slave server -m reliable-master:8083 --verbose

同时需要注意这里的mongo的volumes中的路径修改为自己用户下的新的文件夹下，以供docker使用

mongo:
  container_name: "reliable_mongo"
  restart: always
  image: mongo
  volumes:
    - /Users/mac/software/mongodb/data:/data/db
    - /etc/localtime:/etc/localtime:ro
    -

如果不改会报错

ERROR: for mongo  Cannot start service mongo: Mounts denied:
The path /var/mongo/data/db
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
ERROR: Encountered errors while bringing up the project.

可以安装docker GUI工具来kitematic查看docker中启动的容器,下载地址为https://github.com/docker/kitematic/releases

安装后可以看到docker中的容器信息如下

直接通过浏览器访问页面http://mac.cn:8083可以看到登录页面

点击sign up直接通过页面注册

进入登录页面

首页页面如下

可以通过下面指令添加管理员帐号：

docker exec -it reliable-master make adduser

./bin/reliable-master adduser

*****************************************
*  version 1.0.0 is outdate             *
*  run: npm i -g reliable-master@1.0.6  *
*****************************************

 Do you want create an account? [Y/N]Y

  User Id: root

  Email address: lixiaojiao_hit@163.com

  Password: qwer1234

  isAdmin [Y/N]: Y

  Add user success!

  Goodbye! Have a great day!

安装slave

brew install pkg-config

brew install zeromq

npm install reliable-macaca-slave -g  

安装成功后直接通过命令可以连接master服务端,服务端默认端口为8083


reliable server -m mac.cn:8083 --verbose

启动成功后台日志信息如下

mac:reliable-macaca-docker-compose mac$ reliable server -m mac.cn:8083 --verbose


>> request.js:23:12 [master] pid:18005 get remote update info failed.
>> index.js:57:14 [master] pid:18008 worker fork success
>> index.js:60:14 [master] pid:18008 worker online
>> index.js:63:14 [master] pid:18008 listening worker id: 1, pid: 18009, address: 172.31.218.249:4000
>> index.js:149:12 [worker:1] pid:18009 slave init with
 { webPort: 8080,
  verbose: true,
  port: 4000,
  registry: 'http://registry.npmjs.org',
  master: 'http://mac.cn:8083',
  ip: '172.31.218.249' }
>> index.js:54:12 [worker:1] pid:18009 Zmq Reconnecting...
>> index.js:78:10 [worker:1] pid:18009 Connected to the master server.
>> index.js:9:10 [worker:1] pid:18009 base middlewares attached
>> render.js:15:8 [worker:1] pid:18009 render view path: /usr/local/lib/node_modules/reliable-macaca-slave/web/views
>> index.js:15:12 [worker:1] pid:18009 middleware: inject registed
>> index.js:15:12 [worker:1] pid:18009 middleware: favicon registed
>> index.js:15:12 [worker:1] pid:18009 middleware: powerby registed
>> index.js:15:12 [worker:1] pid:18009 middleware: static registed
>> channel.js:44:53 [worker:1] pid:18009 accept, endpoint: tcp://172.31.218.249:4000
>> Temp directory not existed, check after one day.
>> index.js:63:14 [master] pid:18008 listening worker id: 1, pid: 18009, address: 172.31.218.249:8080
>> Slave Web Server start with options
 { webPort: 8080,
  verbose: true,
  port: 4000,
  registry: 'http://registry.npmjs.org',
  master: 'http://mac.cn:8083',
  ip: '172.31.218.249' }
>> index.js:63:14 [master] pid:18008 listening worker id: 1, pid: 18009, address: 172.31.218.249:8080
>> channel.js:73:14 [worker:1] pid:18009
<--- zmq message
{ sysInfo:
   { cpus:
      [ { model: 'Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz',
          speed: 2700,
          times: { user: 2608850, nice: 0, sys: 2004090, idle: 6215740, irq: 0 } },
        { model: 'Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz',
          speed: 2700,
          times: { user: 1582270, nice: 0, sys: 990180, idle: 8255360, irq: 0 } },
        { model: 'Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz',
          speed: 2700,
          times: { user: 2588680, nice: 0, sys: 1728290, idle: 6510850, irq: 0 } },
        { model: 'Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz',
          speed: 2700,
          times: { user: 1601170, nice: 0, sys: 1009540, idle: 8217100, irq: 0 } },
        [length]: 4 ],
     memory: 0.0027742385864257812,
     type: 'Darwin',
     platform: 'darwin',
     uptime: '0d 3h 11m',
     release: '16.0.0',
     hostname: 'mac.local',
     port: undefined },
  type: 'ack',
  data: 'hello master',
  status: 'ack' }

刷新前台页面看到有一个设备已经连接成功

自此，macaca的master，slave中CI持续集成框架搭建完毕。

gitlab root/12345678

ClassNotFoundException,...$MapClass not found

2016-10-23T08:21:00+08:00

idea工具调试

如果说idea或者eclipse之前在编写的map/reduce程序后直接通过run as的方式能够运行，但是突然某天运行后报错，报错信息却下Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mac.cn.SecondSort.advanced.after.SecondSortRunner1$MapClass not found，这里分析下为何出现这个问题。

1.之前为何没有问题？那么为何之前没有问题呢，下面的目录结构是没有问题的目录结构

对比两张图片我们发现，在resources目录下多了一个文件，文件名称为mapred-site.xml文件，该文件是我直接从hadoop集群环境中拷贝过来的，那么该文件是做什么的？又有什么用呢？

2.mapred-site.xml文件内容:
这里我们在集群中配置的mapred-site.xml文件很简单，内容如下：

<configuration>
<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>
</configuration>

也就是说文件中只有这没一行内容，再无其他。

3.原因分析

那么这一样内容有何作用呢？原来该property文件中的mapreduce.framework.name属性是用来指定mapreduce任务运行的方式，该属性该如何配置呢，通过查看mapred-default.xml我们发现该属性定义为：

<property>
  <name>mapreduce.framework.name</name>
  <value>local</value>
  <description>The runtime framework for executing MapReduce jobs.
  Can be one of local, classic or yarn.
  </description>
</property>

也就是说属性mapreduce.framework.name有三个取值，分别是local，classic，yarn.

local本地模式，不会使用yarn集群来分配资源，直接在本地节点执行，在本地模式下执行的任务无法发挥集群的作用，而且执行的任务是不会在web ui中查看到的

classic是针对mapreduce1而言，为了兼容老版本的mapreuce框架而定

yarn集群管理模式，会通过YarnRunner与resourcemanager通信

所以上面在加入该文件之前，没有显示指定mapreduce.framework.name，因此默认为local本地模式，这样任务是不会提交到集群中，所以本地肯定有map，rducer及runner的相关类文件。当加入该文件后，mapreduce.framework.name由默认的local本地模式改为yarn，运行后，hadoop框架需要将job提交到集群中，而集群中是不存在相关的map，reducer类的。

4.解决方案
如果显示的将mapreduce.framework.name改为了yarn，idea则可通过下面步骤来解决(eclipse可以变通修改)： file->project structure->artifacts->+->jar->from modules with dependiences

设置主类

勾选build on make

最后在java代码中添加下面代码来指定jar的路径

job.setJar("/Users/mac/software/bigdataproject/out/artifacts/bigdata_jar/bigdata.jar");

5.最后运行runner或者driver启动类，不会报错。

hadoop二次排序(高级)

2016-10-22T18:21:00+08:00

hadoop二次排序问题

从hadoop二次排序中可以看出，虽然通过设计Reducer<CustomDatatype, NullWritable, CustomDatatype, NullWritable>这样的输出可以实现，但是发现reduce的输入key为CustomDatatype，也就是说只有map的输入的两列数据完全相同的时候才会进行合并，这样效率是很低的。因为数据到了reduce阶段会首先按照定义的group的规则进行分组，这里没有设置规则，因此按照默认的规则进行分组，只有相同的数据分组后进入reduce，重复该操作，但是两列重复的概率会有多大呢？这完全取决于数据本身。

map阶段的排序逻辑规则

map阶段的排序规则如下：

1.首先查看job是否有设置过setSortComparatorClass,如果有设置，则直接根据该类的实现规则。

2.如果1中没有设置该属性，那么会去自定义的key中查找是否注册过compartor类，如果注册过则使用。

3.如果2中也没有进行注册，则直接使用key中的comparable接口。

下面是改良后的二次排序的代码：

package mac.cn.SecondSort.advanced;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.StringTokenizer;

/**
 * 二次排序启动类或者驱动类
 * http://www.superwu.cn/2013/08/18/492/
 * http://blog.csdn.net/w1014074794/article/details/51821727
 * https://www.iteblog.com/archives/1415
 *
 * @author lixiaojiao
 * @create 2016-10-22 12:36
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class AdvancedSecondSortRunner {

    /**
     * 把第一列整数和第二列作为类的属性，并且实现WritableComparable接口
     */
    public static class IntPair implements WritableComparable<IntPair> {
        private int first = 0;
        private int second = 0;

        public void set(int left, int right) {
            first = left;
            second = right;
        }

        public int getFirst() {
            return first;
        }

        public int getSecond() {
            return second;
        }

        public void readFields(DataInput in) throws IOException {
            first = in.readInt();
            second = in.readInt();
        }

        public void write(DataOutput out) throws IOException {
            out.writeInt(first);
            out.writeInt(second);
        }

        @Override
        public int hashCode() {
            return first + "".hashCode() + second + "".hashCode();
        }

        @Override
        public boolean equals(Object right) {
            if (right instanceof IntPair) {
                IntPair r = (IntPair) right;
                return r.first == first && r.second == second;
            } else {
                return false;
            }
        }

        //这里的代码是关键，因为对key排序时，调用的就是这个compareTo方法
        public int compareTo(IntPair o) {
            if (first != o.first) {
                return first - o.first;
            } else if (second != o.second) {
                return second - o.second;
            } else {
                return 0;
            }
        }


        /** A Comparator that compares serialized IntPair. */
        public static class Comparator extends WritableComparator {
            public Comparator() {
                super(IntPair.class);
            }

            public int compare(byte[] b1, int s1, int l1,
                               byte[] b2, int s2, int l2) {
                return compareBytes(b1, s1, l1, b2, s2, l2);
            }
        }

        static {
            WritableComparator.define(IntPair.class, new Comparator());
        }

    }


    public static class SortComparator extends WritableComparator{


        public SortComparator(){
            super(IntPair.class, true);
        }

        @Override
        public int compare(WritableComparable a, WritableComparable b) {
            IntPair o1 = (IntPair) a;
            IntPair o2 = (IntPair) b;

            if (o1.first != o2.first) {
                return o1.first - o2.first;
            } else {
                return o1.second - o2.second;
            }
        }
    }


    /**
     * 在分组比较的时候，只比较原来的key，而不是组合key。
     */
    public static class GroupingComparator implements RawComparator<IntPair> {

        public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
            return WritableComparator.compareBytes(b1, s1, Integer.SIZE / 8, b2, s2, Integer.SIZE / 8);
        }

        public int compare(IntPair o1, IntPair o2) {
            int first1 = o1.getFirst();
            int first2 = o2.getFirst();
            return first1 - first2;
        }

    }


    public static class MapClass extends Mapper<LongWritable, Text, IntPair, IntWritable> {

        private final IntPair key = new IntPair();
        private final IntWritable value = new IntWritable();

        @Override
        public void map(LongWritable inKey, Text inValue,
                        Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(inValue.toString());
            int left = 0;
            int right = 0;
            if (itr.hasMoreTokens()) {
                left = Integer.parseInt(itr.nextToken());
                if (itr.hasMoreTokens()) {
                    right = Integer.parseInt(itr.nextToken());
                }
                key.set(left, right);
                value.set(right);
                context.write(key, value);
            }
        }
    }

    public static class Reduce extends Reducer<IntPair, IntWritable, Text, IntWritable> {
        private static final Text SEPARATOR = new Text("------------------------------------------------");
        private final Text first = new Text();

        @Override
        public void reduce(IntPair key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
            context.write(SEPARATOR, null);
            first.set(Integer.toString(key.getFirst()));
            for (IntWritable value : values) {
                context.write(first, value);
                System.out.println("----");
            }
        }
    }


    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://mac.cn:8020");

        Job job = Job.getInstance(conf, "secondary sort");
        job.setJarByClass(AdvancedSecondSortRunner.class);
        job.setMapperClass(MapClass.class);
        job.setReducerClass(Reduce.class);

       job.setSortComparatorClass(SortComparator.class);

        job.setGroupingComparatorClass(GroupingComparator.class);

        job.setMapOutputKeyClass(IntPair.class);
        job.setMapOutputValueClass(IntWritable.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path("/user/mac/mapreduce/sort/input"));
        FileOutputFormat.setOutputPath(job, new Path("/user/mac/mapreduce/sort/secondout/"));


        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

}

对应上面总结的三点查看该改良后的二次排序的代码总结如下：
1.如果在main方法中的，没有注释掉job.setSortComparatorClass(SortComparator.class);那么map阶段将key进行排序的时候会使用SortComparator中定义的方法。
2.如果注释掉该行代码，则是使用IntPair中定义的static域中的 WritableComparator.define(IntPair.class, new Comparator());的Comparator 3.如果没有该注册过的Comparator，则直接使用IntPair中的compareto方法。

reduce阶段

查看代码发现，在reduce阶段注册了一个叫做GroupingComparator的比较类，该类的作用是将reduce拉取的数据进行合并，比如<1,2>，<1,3>合并为<1,2,3>，即按照key进行分组，比较相同的分为一组，不同的则不进行分组。

这里需要主意的是，并不是全部数据全部分组完毕后才进入到reduce阶段的，而是在合并完一个分组后会进入reduce方法处理数据，然后继续合并下一个分组，再进入reduce。

比如，到达reduce阶段的数据如下，

那么reduce阶段会先进入分组方法中将<1 2>,<1 3>,<1 6>进行合并为<1,{2,3,6}>,然后进入reduce。然后再进行分组<5,7>合并为<5,7>，然后进入reduce，然后再进行分组<6 10>,<6 11>,<6 13>分组为<6,{10,11,13}>然后继续进入reduce中。

而上面的group为何只比较key呢，因为我们想在reduce阶段中按照key记性分组，这样效率更高。

本文源码地址为:https://github.com/CodeToSurvive1/bigdataproject/blob/master/src/main/java/mac/cn/SecondSort/advanced/AdvancedSecondSortRunner.java

hadoop二次排序

2016-10-22T08:21:00+08:00

map/reduce简单介绍

正常的map/reduce程序而言，在map阶段会根据key自动进行排序，因此key必须会实现CompareWritable接口，即要能排序又要能够序列化，因此为了测试map的输出能够按照key自动进行排序，下面简单的例子是将文本中两列文本按照第一列进行排序，这里权且叫做一次排序。

hdfs上/user/mac/mapreduce/sort/input/input.txt文件中的内容信息为：


hello world
word hello
li xiaojiao
angle baby
good study
china shanghai
safari chrome
hello li
hello xiao
hello xiao
good study
hello jiao
china shandong
china qingdao

一次排序实现逻辑

1.编写mapper类

该类是将上面的文本由hadoop自己内部实现的机制(LineRecordReader)实现的将文件按照行为单位解析为<key,value>-><偏移量，每行内容>的结构作为map阶段的输入，比如上面的保存在/user/mac/mapreduce/sort/input/input.txt会被解析为：

0---->hello world
12---->word hello
23---->li xiaojiao
35---->angle baby
46---->good study
57---->china shanghai
72---->safari chrome
86---->hello li
95---->hello xiao
106---->hello xiao
117---->good study
128---->hello jiao
139---->china shandong
154---->china qingdao

因此map阶段的工作需要将上面的结果转变为正常的/user/mac/mapreduce/sort/input/input.txt内容输出因此map阶段的输出应该为：

hello world
word hello
li xiaojiao
angle baby
good study
china shanghai
safari chrome
hello li
hello xiao
hello xiao
good study
hello jiao
china shandong
china qingdao

因此这里map阶段的逻辑代码可以按照下面实现

package mac.cn.SecondSort.before;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * 排序mapper
 *
 * @author lixiaojiao
 * @create 2016-10-22 09:52
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class SortMapper extends Mapper<LongWritable, Text, Text, Text> {

    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
        System.out.println("----mapper---start");
    }

    /**
     * 输入为1->key    value
     * 35->key    value
     * 转换为key1 value1
     * key2  value2
     *
     * @param key
     * @param value
     * @param context
     * @throws IOException
     * @throws InterruptedException
     */
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        System.out.println(key.toString() + "---->" + value.toString());
        String[] values = value.toString().split(" ");
        String valueKey = values[0];
        String valueValue = values[1];

        context.write(new Text(valueKey), new Text(valueValue));
    }


    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        System.out.println("----mapper---end");
    }
}

2.编写reducer类

由于在map阶段和reduce阶段中存在shuffle阶段，而shuffle阶段中会包含partition分区，sort排序，，分组组合（group），combine合并(默认没有该操作，需要指定)等操作，因此到了reduce阶段后的数据会按照key进行了相关的合并操作，以便节省网络开销及本地磁盘的开销，因此到了reduce阶段后的数据也就变为了下面的格式:

angle------baby,
china------shandong,qingdao,shanghai,
good------study,study,
hello------world,jiao,xiao,xiao,li,
li------xiaojiao,
safari------chrome,
word------hello,

这里可以看出key已经按照字母顺序进行了排序，因此reduce阶段的输出只需要再重新将上面的数据解析为key，value的格式即可。因此这里reducer阶段的逻辑代码可以按照下面实现

package mac.cn.SecondSort.before;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

/**
 * sortReducer
 *
 * @author lixiaojiao
 * @create 2016-10-22 09:55
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class SortReducer extends Reducer<Text, Text, Text, Text> {

    @Override
    protected void setup(Context context) throws IOException, InterruptedException {

        System.out.println("----reduce---start");
    }

    /**
     * <key,value,value>
     * <key1,value1,value1>
     * 转换为
     * key value
     * key value
     * key1 value1
     * key1 value1
     *
     * @param key
     * @param values
     * @param context
     * @throws IOException
     * @throws InterruptedException
     */
    @Override
    protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {

        StringBuilder valueStr = new StringBuilder();
        for (Text value : values) {
            valueStr.append(value).append(",");
            context.write(key, value);
        }

        System.out.println(key.toString() + "------" + valueStr);
    }

    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        System.out.println("----reduce---end");
    }
}

3.编写驱动类进行测试

package mac.cn.SecondSort.before;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
 * sort启动类
 *
 * @author lixiaojiao
 * @create 2016-10-22 09:59
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class SortRunner implements Tool {

    private Configuration conf;

    public int run(String[] args) throws Exception {


        Configuration conf = this.getConf();

        Job job = Job.getInstance(conf, "sort test");


        job.setMapperClass(SortMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);

        job.setReducerClass(SortReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        FileInputFormat.addInputPath(job, new Path("/user/mac/mapreduce/sort/input"));
        FileOutputFormat.setOutputPath(job, new Path("/user/mac/mapreduce/sort/output"));

        return job.waitForCompletion(true) ? 0 : 1;
    }

    public void setConf(Configuration conf) {
        this.conf = conf;
    }

    public Configuration getConf() {
        return conf;
    }


    public static void main(String args[]) throws Exception {

        Configuration config = new Configuration();
        config.set("fs.defaultFS", "hdfs://mac.cn:8020");

        ToolRunner.run(config, new SortRunner(), args);
    }

}

4.准备数据进行测试

//新建sort目录
./bin/hdfs dfs -mkdir /user/mac/mapreduce/sort/input
//创建输入文件
./bin/hdfs dfs -touchz /user/mac/mapreduce/sort/input/input.txt
//在输入文件中追加文本信息
./bin/hdfs dfs -appendToFile - /user/mac/mapreduce/sort/input/input.txt

hello world
word hello
li xiaojiao
angle baby
good study
china shanghai
safari chrome
hello li
hello xiao
hello xiao
good study
hello jiao
china shandong
china qingdao

5.运行SortRunner类进行测试

测试结果为

mac:hadoop-2.5.0-cdh5.3.6 mac$ ./bin/hdfs dfs -cat /user/mac/mapreduce/sort/output/*
16/10/22 11:28:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hello world
word hello
li xiaojiao
angle baby
good study
china shanghai
safari chrome
hello li
hello xiao
hello xiao
good study
hello jiao
china shandong
china qingdao

6.总结：

hadoop中的map/reduce的map阶段确实经过了排序的操作，而且是按照key进行排序，但是value是没有经过排序的。上面一次排序源代码连接：一次排序源代码

二次排序

从上面的一次排序中可以看出，在map阶段会把key按照排序规则进行排序，因此，如果我想对上面的原始数据进行二次排序呢，也就是说在一次排序把第一列进行排序的基础上，也要求把第二列同样按照排序的规则进行排序，这里该如何做呢？其实很简单，就是将每行数据拆分成多个字段，将这多个字段定义为map的一个key，也就是自定义key数据类型，该类型中包含要排序的这几列而已。

1.自定义数据类型，封装该行记录

package mac.cn.SecondSort.after;

import org.apache.hadoop.io.WritableComparable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

/**
 * 二次排序中自定义数据类型
 *
 * @author lixiaojiao
 * @create 2016-10-22 11:55
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class CustomDatatype implements WritableComparable<CustomDatatype> {

    private String columnA;

    private String columnB;

    public int compareTo(CustomDatatype o) {
        int result = this.getColumnA().compareTo(o.getColumnA());
        if (result == 0) {
            return this.getColumnB().compareTo(o.getColumnB());
        }
        return result;
    }

    public void write(DataOutput out) throws IOException {
        out.writeUTF(columnA);
        out.writeUTF(columnB);
    }

    public void readFields(DataInput in) throws IOException {
        this.columnA = in.readUTF();
        this.columnB = in.readUTF();
    }

    @Override
    public String toString() {
        return this.columnA+" "+this.columnB;
    }

    public String getColumnA() {
        return columnA;
    }

    public void setColumnA(String columnA) {
        this.columnA = columnA;
    }

    public String getColumnB() {
        return columnB;
    }

    public void setColumnB(String columnB) {
        this.columnB = columnB;
    }
}

2.自定义mapper类，mapper的输出的key设置为自定义数据类型CustomDatatype即可，而输出类型直接设置为空NullWritable就可以了，这样mapper输出到reducer阶段的时候会自动的进行分区，排序，分组组合（group），combine合并(默认没有该操作，需要指定)等操作。

package mac.cn.SecondSort.after;

import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

import java.io.IOException;

/**
 * 二次排序mapper类
 *
 * @author lixiaojiao
 * @create 2016-10-22 12:04
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class SecondSortMapper extends Mapper<LongWritable, Text, CustomDatatype, NullWritable> {

    private CustomDatatype customDatatype = new CustomDatatype();

    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
        System.out.println("-----mapper-----start");
    }

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        String[] valueLine = value.toString().split(" ");
        String columnA = valueLine[0];
        String columnB = valueLine[1];

        customDatatype.setColumnA(columnA);
        customDatatype.setColumnB(columnB);
        context.write(customDatatype,NullWritable.get());
    }

    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        System.out.println("-----mapper-----end");
    }
}

3.自定义reducer类，该类的输出key也是CustomDatatype，value为空

package mac.cn.SecondSort.after;

import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

/**
 * 二次排序reducer类
 *
 * @author lixiaojiao
 * @create 2016-10-22 12:19
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class SecondSortReducer extends Reducer<CustomDatatype, NullWritable, CustomDatatype, NullWritable> {
    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
        System.out.println("-----mapper----start");
    }

    /**
     * 按照注释中的实现逻辑会遗漏掉行记录相同的数据,比如有三行数据相同,那么结果就合并为了一行
     *
     * @param key
     * @param values
     * @param context
     * @throws IOException
     * @throws InterruptedException
     */
    @Override
    protected void reduce(CustomDatatype key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException {
        // context.write(key,NullWritable.get());

        for (NullWritable value : values) {
            context.write(key, NullWritable.get());
        }
    }

    @Override
    protected void cleanup(Context context) throws IOException, InterruptedException {
        System.out.println("-----mapper----end");
    }
}

4.编写启动类

package mac.cn.SecondSort.after;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
 * 二次排序启动类或者驱动类
 *
 * @author lixiaojiao
 * @create 2016-10-22 12:36
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class SecondSortRunner implements Tool {

    private Configuration config;

    public int run(String[] args) throws Exception {

        Configuration conf = this.getConf();

        Job job = Job.getInstance(conf,"second sort");


        job.setMapperClass(SecondSortMapper.class);
        job.setMapOutputKeyClass(CustomDatatype.class);
        job.setMapOutputValueClass(NullWritable.class);

        job.setReducerClass(SecondSortReducer.class);
        job.setOutputKeyClass(CustomDatatype.class);
        job.setOutputValueClass(NullWritable.class);

        FileInputFormat.addInputPath(job,new Path("/user/mac/mapreduce/sort/input"));
        FileOutputFormat.setOutputPath(job,new Path("/user/mac/mapreduce/sort/secondout"));

        return job.waitForCompletion(true)?0:1;
    }

    public void setConf(Configuration conf) {
        this.config = conf;
    }

    public Configuration getConf() {
        return config;
    }

    public static void main(String args[]) throws Exception {
        Configuration config = new Configuration();
        config.set("fs.defaultFS", "hdfs://mac.cn:8020");
        int exit = ToolRunner.run(config, new SecondSortRunner(),args);
        System.exit(exit);
    }


}

5.运行后查看输出结果

mac:hadoop-2.5.0-cdh5.3.6 mac$ ./bin/hdfs dfs -cat /user/mac/mapreduce/sort/secondout/*
16/10/22 12:55:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
angle baby
china qingdao
china shandong
china shanghai
good study
good study
hello jiao
hello li
hello world
hello xiao
hello xiao
li xiaojiao
safari chrome
word hello

6.总结

注意reduce阶段的输出，要根据nullwritable的个数循环输出，不然会漏掉记录相同的数据。源代码为：二次排序元源代码

hadoop自定义InputFormat

2016-10-15T12:47:00+08:00

数据准备

执行下面sql语句

use hadoop
db.persons.insert({name:"zhangsan",age:15,sex:"M"})
db.persons.insert({name:"lisi",sex:"M"})
db.persons.insert({name:"wangwu",age:18})
db.persons.insert({name:"xiaoer",age:18,sex:"M"})
db.persons.insert({name:"tom",age:17})
db.persons.insert({name:"Helen",age:15,sex:"F"})
db.persons.insert({name:"Judy",age:15,sex:"F"})

db.persons.find().toArray()
[
	{
		"_id" : ObjectId("5801b5bcaed1f2ef17d38f47"),
		"name" : "zhangsan",
		"age" : 15,
		"sex" : "M"
	},
	{
		"_id" : ObjectId("5801b62aaed1f2ef17d38f48"),
		"name" : "lisi",
		"sex" : "M"
	},
	{
		"_id" : ObjectId("5801b62aaed1f2ef17d38f49"),
		"name" : "wangwu",
		"age" : 18
	},
	{
		"_id" : ObjectId("5801b62aaed1f2ef17d38f4a"),
		"name" : "xiaoer",
		"age" : 18,
		"sex" : "M"
	},
	{
		"_id" : ObjectId("5801b62aaed1f2ef17d38f4b"),
		"name" : "tom",
		"age" : 17
	},
	{
		"_id" : ObjectId("5801b62aaed1f2ef17d38f4c"),
		"name" : "Helen",
		"age" : 15,
		"sex" : "F"
	},
	{
		"_id" : ObjectId("5801b62caed1f2ef17d38f4d"),
		"name" : "Judy",
		"age" : 15,
		"sex" : "F"
	}
]

业务逻辑

本例子是在mongodb数据库中存在person表，该表中存在三个字段，分别为名字，年龄，性别。通过编写map/reduce来实现读取数据库中的数据，然后通过map/reduce计算没个年龄的人数，并将结果输出到hdfs文件系统中。

实现逻辑

由于从mongodb中读取person表中的数据，因此对于map阶段的输入来说，输入的key应该为行号，value应该为表中的一行记录。而reduce阶段的输出key应该为年龄，value为数量

代码逻辑

自定义InputFormat要点：

1.继承InputFormat类并进行相关实现

InputFormat需要实现两个方法:

方法名称	作用
InputSplit[] getSplits(JobConf job, int numSplits);	定义了分片信息如何获取
RecordReader<K, V> getRecordReader(InputSplit split,JobConf job,Reporter reporter);	定义了如何读取记录数据

2.继承RecordReader类并进行相关实现 RecordReader需要实现两个方法:

方法名称	作用
public abstract void initialize(InputSplit split, TaskAttemptContext context );	对split初始化
public abstract boolean nextKeyValue();	判断是否还有数据
public abstract KEYIN getCurrentKey();	获取当前数据的key
public abstract VALUEIN getCurrentValue();	获取当前数据的value
public abstract float getProgress()	获取当前执行的进度
public abstract void close()	释放资源

代码编写

在pom.xml文件中添加mongod的依赖包

<dependency>
	<groupId>org.mongodb</groupId>
	<artifactId>mongo-java-driver</artifactId>
	<version>3.2.1</version>
</dependency>

2.根据数据库表person自定义数据类型

package mac.cn.CustomInputFormat;

import org.apache.hadoop.io.Writable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

/**
 * 自定义MongoDB数据类型
 *
 * @author lixiaojiao
 * @create 2016-10-17 20:55
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class MongoDBWritable implements Writable {

    private String name;

    private Integer age;

    public void write(DataOutput out) throws IOException {
        out.writeUTF(name);
        //age为空时不序列化
        if (age == null) {
            out.writeBoolean(false);
        } else {
            out.writeBoolean(true);
            out.writeInt(age);
        }
    }

    public void readFields(DataInput in) throws IOException {
        this.name = in.readUTF();
        if (in.readBoolean()) {
            this.age = in.readInt();
        }
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public Integer getAge() {
        return age;
    }

    public void setAge(Integer age) {
        this.age = age;
    }
}

3.自定义InputSplit分片信息

package mac.cn.CustomInputFormat;

import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.InputSplit;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

/**
 * 自定义mongodb分片
 *
 * @author lixiaojiao
 * @create 2016-10-17 21:03
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class MongoDBInputSpilt extends InputSplit implements Writable {

    //定义分片起始位置
    private Long start;

    //定义分片终止位置
    private Long end;

    public MongoDBInputSpilt() {
        super();
    }

    public MongoDBInputSpilt(Long start, Long end) {
        super();
        this.start = start;
        this.end = end;
    }

    public long getLength() throws IOException, InterruptedException {
        return end - start;
    }

    public String[] getLocations() throws IOException, InterruptedException {
        return new String[0];
    }

    public void write(DataOutput out) throws IOException {
        out.writeLong(start);
        out.writeLong(end);
    }

    public void readFields(DataInput in) throws IOException {
        this.start = in.readLong();
        this.end = in.readLong();
    }

    public Long getStart() {
        return start;
    }

    public void setStart(Long start) {
        this.start = start;
    }

    public Long getEnd() {
        return end;
    }

    public void setEnd(Long end) {
        this.end = end;
    }
}

4.自定义RecordReader

package mac.cn.CustomInputFormat;

import com.mongodb.MongoClient;
import com.mongodb.MongoClientOptions;
import com.mongodb.MongoClientURI;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.bson.Document;

import java.io.IOException;

/**
 * 自定义MongoDB的RecordReader封装类
 *
 * @author lixiaojiao
 * @create 2016-10-17 21:24
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class MongoDBRecordReader<V extends MongoDBWritable> extends RecordReader<LongWritable, V> {

    private MongoDBInputSpilt inputSpilt;

    private Configuration config;

    private int index = 0;

    private MongoCursor<Document> documentIterator;

    private LongWritable key;

    private V value;

    public MongoDBRecordReader() {
        super();
    }

    public MongoDBRecordReader(InputSplit inputSpilt, TaskAttemptContext context) throws IOException, InterruptedException {
        this.initialize(inputSpilt, context);
    }

    public void initialize(InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException {
        this.inputSpilt = (MongoDBInputSpilt) split;
        config = context.getConfiguration();

        this.key = new LongWritable();
        this.value = (V) new MongoDBWritable();
    }

    public boolean nextKeyValue() throws IOException, InterruptedException {

        if (documentIterator == null) {
            //定义url
            MongoClientURI uri = new MongoClientURI("mongodb://127.0.0.1:27017/hadoop",
                    MongoClientOptions.builder().cursorFinalizerEnabled(false));

            //获取连接
            MongoClient client = new MongoClient(uri);

            //获取数据库
            MongoDatabase hadoopDb = client.getDatabase("hadoop");

            //获取集合persons
            MongoCollection<Document> personsCollection = hadoopDb.getCollection("persons");

            FindIterable<Document> result = personsCollection.find().skip(this.inputSpilt.getStart().intValue()).limit((int) this.inputSpilt.getLength());

            documentIterator = result.iterator();
        }

        if (documentIterator.hasNext()) {
            Document doc = documentIterator.next();

            this.key = new LongWritable(this.inputSpilt.getStart() + index);

            value.setName(doc.get("name").toString());

            if (doc.get("age") != null) {
                value.setAge(Double.valueOf(doc.get("age").toString()).intValue());
            } else {
                value.setAge(null);
            }

            index++;

            return true;
        }

        return false;
    }

    public LongWritable getCurrentKey() throws IOException, InterruptedException {
        return key;
    }

    public V getCurrentValue() throws IOException, InterruptedException {
        return value;
    }

    public float getProgress() throws IOException, InterruptedException {
        return 0;
    }

    public void close() throws IOException {
        documentIterator.close();
    }
}

5.自定义InputFormat

package mac.cn.CustomInputFormat;

import com.mongodb.MongoClient;
import com.mongodb.MongoClientOptions;
import com.mongodb.MongoClientURI;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.*;
import org.bson.Document;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

/**
 * 自定义input format输入格式化器
 *
 * @author lixiaojiao
 * @create 2016-10-17 21:00
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class MongoDBInputFormat<V extends MongoDBWritable> extends InputFormat<LongWritable, V> {

    public List<InputSplit> getSplits(JobContext context) throws IOException, InterruptedException {

        //定义url
        MongoClientURI uri = new MongoClientURI("mongodb://127.0.0.1:27017/hadoop",
                MongoClientOptions.builder().cursorFinalizerEnabled(false));

        //获取连接
        MongoClient client = new MongoClient(uri);

        //获取数据库
        MongoDatabase hadoopDb = client.getDatabase("hadoop");

        //获取集合persons
        MongoCollection<Document> personsCollection = hadoopDb.getCollection("persons");

        //获取总记录数
        long count = personsCollection.count();

        //定义分片的大小为2
        long chunkSize = 2;

        //计算分片数量
        long mappers = count / chunkSize;

        List<InputSplit> splits = new ArrayList<InputSplit>();

        //组装所有的分片位置信息
        for (int i = 0; i < mappers; i++) {
            if (i + 1 == mappers) {
                //如果最后一组数据不恰巧可分,则最后的数据全部放在最后一个mapper中,比如七个数据,最后一组为5,6,7
                splits.add(new MongoDBInputSpilt(i * chunkSize, count));
            } else {
                splits.add(new MongoDBInputSpilt(i * chunkSize, i * chunkSize + chunkSize));
            }
        }

        return splits;
    }

    public RecordReader createRecordReader(InputSplit split, TaskAttemptContext context) throws IOException, InterruptedException {
        return new MongoDBRecordReader(split, context);
    }
}

6.编写测试类

package mac.cn.CustomInputFormat;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;

/**
 * MongoDBInputFormat启动类
 *
 * @author lixiaojiao
 * @create 2016-10-17 21:59
 * @email lixiaojiao_hit@163.com
 * @QQ 2305886442
 * @graduate 哈尔滨工业大学
 * @address shanghai china
 * @blog https://CodeToSurvive1.github.io
 * @github https://github.com/CodeToSurvive1
 */
public class MongoDBInputFormatRunner {

    static class MongoDBMapper extends Mapper<LongWritable, MongoDBWritable, LongWritable, MongoDBWritable> {
        @Override
        protected void map(LongWritable key, MongoDBWritable value, Context context) throws IOException, InterruptedException {
            if(value.getAge()==null){
                return;
            }
            context.write(new LongWritable(value.getAge()), value);
        }
    }

    static class MongoDBReducer extends Reducer<LongWritable, MongoDBWritable, IntWritable, IntWritable> {

        @Override
        protected void reduce(LongWritable key, Iterable<MongoDBWritable> values, Context context) throws IOException, InterruptedException {

            Map<String,Integer> ages = new HashMap<String, Integer>();

            for(MongoDBWritable value : values){
                if(ages.containsKey(value.getAge().toString())){
                    Integer number = ages.get(value.getAge().toString());
                    number++;
                    ages.put(value.getAge().toString(),number);
                }else{
                    ages.put(value.getAge().toString(),1);
                }
            }

            Set<Map.Entry<String, Integer>> entries = ages.entrySet();

            Iterator<Map.Entry<String, Integer>> iterator = entries.iterator();
            while(iterator.hasNext()){
                Map.Entry<String, Integer> next = iterator.next();
                context.write(new IntWritable(Integer.parseInt(next.getKey())),new IntWritable(next.getValue()));
            }

        }
    }

    public static void main(String args[]) throws IOException, ClassNotFoundException, InterruptedException {

        Configuration config = new Configuration();
        config.set("fs.defaultFS","hdfs://mac.cn:8020");


        Job job = Job.getInstance(config, "MongoDb input format");

        job.setInputFormatClass(MongoDBInputFormat.class);

        job.setMapperClass(MongoDBMapper.class);
        job.setMapOutputKeyClass(LongWritable.class);
        job.setMapOutputValueClass(MongoDBWritable.class);


        job.setReducerClass(MongoDBReducer.class);
        job.setOutputKeyClass(IntWritable.class);
        job.setOutputValueClass(IntWritable.class);

        FileOutputFormat.setOutputPath(job,new Path("/user/mac/mapreduce/output3"));

        System.exit(job.waitForCompletion(true)?0:1);
    }

}

代码测试

启动mongodb服务端
运行runner测试类
查看hdfs中的数据
./bin/hdfs dfs -cat /user/mac/mapreduce/output3/*

年龄	数量
15	3
17	1
18	2

代码地址: https://github.com/CodeToSurvive1/bigdataproject/tree/master/src/main/java/mac/cn/CustomInputFormat

Mac系统mongodb安装

2016-10-10T21:58:00+08:00

mac系统安装步骤

1.通过homebrew安装mongodb

  1.1 brew update |更新Homebrew的包数据库
  1.2 brew install mongodb |安装mongodb数据库

2.启动数据库

  2.1 首先指定mongodb的数据目录，默认是/data/db，这里改为/Users/mac/software/mongodb/data
  2.2 启动mongodb mongod --dbpath /Users/mac/software/mongodb/data

3.启动客户端连接测试成功与否

mac:data mac$ mongo
MongoDB shell version: 3.2.10
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Server has startup warnings:
2016-10-11T08:29:52.538+0800 I CONTROL  [initandlisten]
2016-10-11T08:29:52.538+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
>

show databases;//查看数据库
show collections;//查看当前数据库的集合，对应的应该是表的概念
use hadoop;//如果没有则创建数据库，有则切换
show dbs;//显示有数据的数据库
db.hadoop.insert({'name':'hadoop'});//向hadoop数据库中插入数据
db.dropDatabase();//删除当前数据库
db.createCollection("users");//在当前连接的数据库中创建集合users
show collections;//显示当前连接的数据库的所有集合
db.users.drop();//删除当前连接数据库的Users集合

参考文档

homebrew安装mongodb官方网站参考连接

mongodb shell 使用

macaca源码分析-android-uiautomator

2016-10-05T07:54:00+08:00

  由于新手接触自动化来说，很多知识需要恶补，其中对于自动化脚本是如何通过指令发送到手机，感觉很是神奇，因此对于macaca中如何通过脚本操控手机这块进行了相关的研究，发现，在macaca中，操控安卓手机主要有两种方式：
  1.  直接对底层adb指令进行封装，上层进行相关调用，比如截图啊、查询设备啊之类
  2.  通过安卓自动化框架封装的Uiautomator进行相关封装
  本文主要分析macaca中对于Uiautomator的封装的讨论

前提准备

 macaca-adb
 macaca-android
 uiautomator-client

软件下载方式

可以通过两种方式获取：

      1. 安装macaca-cli及android驱动，[可以参考macaca环境搭建](https://codetosurvive1.github.io/posts/macaca-environment.html)，安装成功后会在对应的目录下生成node_moduels下,可以直接查看/usr/local/lib/node_modules/macaca-android驱动，该驱动依赖于macaca-adb以及uiautomator-client
      2. 源码方式查看，直接从github下载源代码并安装相关依赖包

软件下载

这里直接使用上面的第二种方式，源码下载安装方式查看

    1.下载源代码
      mac:macaca mac$ git clone https://github.com/macacajs/macaca-android.git

      Cloning into 'macaca-android'...
      remote: Counting objects: 337, done.
      remote: Compressing objects: 100% (35/35), done.
      remote: Total 337 (delta 22), reused 0 (delta 0), pack-reused 302
      Receiving objects: 100% (337/337), 54.47 KiB | 64.00 KiB/s, done.
      Resolving deltas: 100% (222/222), done.
      Checking connectivity... done.
    2.安装依赖
      mac:macaca mac$ cd macaca-android/
      mac:macaca-android mac$ npm install
    3.这里使用webstorm打开macaca-android工程

webstorm中工程截图如下

源码分析

macaca-android代码目录lib下核心代码为controllers.js和macaca-android.js

macaca-android.js中构造函数

class Android extends DriverBase {
  constructor() {
    super();
    this.adb = null;
    this.apkInfo = null;
    this.args = null;
    this.chromedriver = null;
    this.chromeDriverPort = null;
    this.proxy = null;
    this.udid = null;
    this.uiautomator = null;
    this.isChrome = null;
    this.isVirtual = true;
    this.moveToPosition = null;
    this.contexts = [];
  }
}

启动设备

Android.prototype.startDevice = function *(caps) {
  this.args = _.clone(caps);
  this.isChrome = this.args.browserName && this.args.browserName.toLowerCase() === 'chrome';
  yield JAVA.getVersion();
  this.initReuse();
  this.initAdb();
  yield this.initDevice();
  yield this.initUiautomator();//初始化UIautomator
  yield this.getApkInfo();
  yield this.unlock();
  yield this.setIME();
  yield this.launchApk();
  yield this.waitActivityReady();

  if (this.isChrome) {
    yield this.getWebviews();
  }
};

  Android.prototype.initUiautomator = function *() {
  this.uiautomator = new UIAutomator();
  yield this.uiautomator.init(this.adb);
  };

UIAutomator引入包uiautomator-client

  const UIAutomator = require('uiautomator-client');    

查看uiautomator-client包下lib下的uiautomator-client.js

UIAutomator的构造函数及init方法

function UIAutomator() {
  this.adb = null;
  this.socket = null;
  this.socketPort = 6789;
  this.queue = [];
}

UIAutomator.prototype.init = function *(adb) {
  this.adb = adb;
  const ANDROID_TMP_DIR = this.adb.getTmpDir();

  try {
    yield this.adb.rm(`${ANDROID_TMP_DIR}/${FILE_NAME}.jar`);
  } catch (e) {

  }
  if (!_.isExistedFile(binPath)) {
    logger.error(`uiautomator-bootstrap was not found in: ${binPath}, please check your Android ENV`);
    return;
  }
  yield this.adb.push(binPath, ANDROID_TMP_DIR);
  try {
    yield this.adb.killProcess('uiautomator');
  } catch (e) {

  }
  yield this.adb.forward(this.socketPort, this.socketPort);
  yield this.initSocketServer();
  yield this.initSocketClient();
};

从上面的构造函数及init方法中可以看出，默认构造了6789端口，然后调用init初始化方法中，首先删除了安卓手机临时目录’/data/local/tmp’中的uiautomator-bootstrap.jar包,然后上传jar包到临时目录下，并杀掉手机中的uiautomator进程，并将手机端口与电脑端口进行映射，最后初始化手机端的服务器及电脑端的客户端连接。

UIAutomator.prototype.initSocketServer = function() {
  return new Promise((resolve, reject) => {
    const ANDROID_TMP_DIR = this.adb.getTmpDir();
    let args = `shell uiautomator runtest ${ANDROID_TMP_DIR}/${FILE_NAME}.jar -c ${CLASS_NAME} -e port ${this.socketPort} -e flag ${READY_FLAG}`.split(' ');

    var proc = this.adb.spawn(args, {
      path: process.cwd(),
      env: process.env
    });

    proc.stderr.setEncoding('utf8');
    proc.stdout.setEncoding('utf8');
    proc.stdout.on('data', data => {
      if (!!~data.indexOf(READY_FLAG)) {
        logger.info('socket server ready');
        resolve();
      } else {
        console.log(data);
      }
    });
    proc.stderr.on('data', data => {
      console.log(data);
    });
  });
};

初始化手机服务端方法中，可以看出是直接调用通过adb shell调用手机中的uiautomator执行uiautomator的测试集adb shell uiautomator runtest ${ANDROID_TMP_DIR}/${FILE_NAME}.jar -c ${CLASS_NAME} -e port ${this.socketPort} -e flag ${READY_FLAG}，这里的${ANDROID_TMP_DIR}/${FILE_NAME}.jar指的是/data/local/tmp/uiautomator-bootstrap.jar,-c指定要运行的测试类，即为com.android.uiautomator.client.Initialize，-e指定参数，这里指定了两个参数，一个是port，一个是flag

UIAutomator.prototype.initSocketClient = function() {
  return new Promise((resolve, reject) => {
    this.socket = net.connect(this.socketPort, () => {
      logger.info('socket client ready');
      resolve();
    });

    var tempData = '';
    this.socket.setEncoding('utf8');
    this.socket.on('data', data => {
      try {
        let res = JSON.parse(tempData + data);
        tempData = '';
        let success = res.success;

        if (success) {
          this.queue.shift().resolve(res.data);
        } else {
          this.queue.shift().reject();
        }
      } catch (e) {
        tempData += data;
      }
    });
    this.socket.on('end', () => {
      logger.warn('connect lost');
    });
  });
};

电脑客户端连接比较简单，直接连接上服务端，并注册监听事件。

UIAutomator.prototype.send = function *(cmd) {
  var defer = new _.Defer();
  defer.promise.then(data => {
  });
  this.queue.push(defer);
  this.socket.write(`${JSON.stringify(cmd)}\n`);
  return yield defer.promise;
};

这是另外的一个发送的方法send，直接将指令发送到服务端,该方法的调用在macaca-android.js中,这里很简单直接调用并返回响应码

Android.prototype.send = function *(data) {
  try {
    var result = yield this.uiautomator.send(data);
  } catch (err) {
    logger.debug(`UnknownError from uiautomator ${err}`);
    throw new errors.UnknownError();
  }
  var statusCode = result.status;
  if (statusCode === 0) {
    return result.value;
  } else {
    var errName = getErrorByCode(statusCode);
    if (!errName) {
      throw new errors.UnknownError();
    }
    throw new errors[errName](result.value);
  }
};

手机端uiautomator源码导入

从这里的目录可以看出macaca中对于uiautomator的java源代码封装，下面使用idea进行相关源码导入或者直接通过github下载并添加lib下的jar包

1.建立普通java工程

2.uiautomator-client目录下的src拷贝到创建的java工程中

3.添加uiatuomator，android，及fastjson的jar包,添加成功后，代码中不报错

手机端uiautomator源码分析

1.直接查看上面js中指定的com.android.uiautomator.client.Initialize源码

public class Initialize extends UiAutomatorTestCase {

	/**
	 * @throws IOException
	 * @throws JSONException
	 * @throws ParserConfigurationException
	 */
	public void testStartServer() throws IOException, JSONException,
			ParserConfigurationException {
		Utils.output("uiautomator start socket server.");
		int port = Integer.parseInt(getParams().getString("port"));
		String readyFlag = getParams().getString("flag");
		SocketServer server = new SocketServer(port);
		server.listen(readyFlag);
	}
}

可以看出，该类直接继承自uiautomator框架中的UiAutomatorTestCase，该方法直接获取传递进来的参数port和flag，并传递给SocketServer类

2.SocketServer类是简单的java tcp服务端实现类

public class SocketServer {

	private ServerSocket server;

	public SocketServer(int port) throws IOException {
		setServer(new ServerSocket(port));
	}

	/**
	 * @param readyFlag
	 * @throws JSONException
	 * @throws ParserConfigurationException
	 */
	public void listen(String readyFlag) throws JSONException,
			ParserConfigurationException {
		Utils.output(readyFlag);
		try {
			Socket client = server.accept();
			BufferedReader input = new BufferedReader(new InputStreamReader(
					client.getInputStream(), "UTF-8"));
			BufferedWriter output = new BufferedWriter(new OutputStreamWriter(
					client.getOutputStream(), "UTF-8"));
			StringBuilder swap = new StringBuilder();

			while (true) {
				swap.setLength(0);
				int chunk;
				while ((chunk = input.read()) != -1 && input.ready()) {
					swap.append((char) chunk);
				}
				String str = swap.toString();
				Utils.output("recive: " + str);
				String res = Command.handleInput(str);
				Utils.output("return: " + res);
				output.write(res);
				output.flush();
			}
		} catch (final IOException e) {

		}
	}

	/**
	 * @return res
	 */

	public ServerSocket getServer() {
		return server;
	}

	/**
	 * @param server
	 */
	public void setServer(ServerSocket server) {
		this.server = server;
	}
}

该类的核心代码在listen方法，该方法即为死循环，不停的接受发送过来的指令，然后处理执行Command.handleInput，并将处理的结果返回

3.Command.handleInput处理接受到的指令并解析，并根据命令将命令参数传递给不同的实现类

public class Command {
	/**
	 *
	 */
	private static HashMap<String, CommandBase> map = new HashMap<String, CommandBase>();

	static {
		getMap().put("ping", new Ping());
		getMap().put("wake", new Wake());
		getMap().put("find", new Find());
		getMap().put("setText", new SetText());
		getMap().put("getText", new GetText());
		getMap().put("click", new Click());
		getMap().put("clearText", new ClearText());
		getMap().put("swipe", new Swipe());
		getMap().put("getWindowSize", new GetWindowSize());
		getMap().put("getProperties", new GetProperties());
		getMap().put("getSource", new GetSource());
	}

	/**
	 * @param input
	 * @return res
	 * @throws JSONException
	 * @throws ParserConfigurationException
	 */
	static String handleInput(String input) throws JSONException,
			ParserConfigurationException {
		JSONObject json = Utils.parseJSON(input);
		String cmd = (String) json.get("cmd");
		JSONObject args = (JSONObject) json.get("args");
		return getMap().get(cmd).execute(args);
	}

	/**
	 * @return res
	 */
	public static HashMap<String, CommandBase> getMap() {
		return map;
	}

	/**
	 * @param map
	 */
	public static void setMap(HashMap<String, CommandBase> map) {
		Command.map = map;
	}
}

其中的handleInput即接受指令，然后从指令中获取命令和命令参数，并调用getMap().get(cmd).execute(args)，而getMap()中的map的初始化在static域中

命令	处理类	作用
ping	Ping()	测试连通
wake	Wake()	模拟按电源键
find	Find()	查找页面元素
setText	SetText()	设置文本内容
getText	GetText()	获取文本内容
cick	Click()	点击元素
clearText	ClearText()	清空元素内容
swipe	Swipe()	滑动
getWindowSize	GetWindowSize()	获取手机长宽
getProperties	GetProperties()	获取手机属性
getSource	GetSource()	获取当前画面的dump的xml文件

4.各个处理类分析

4.1 Ping代码分析，该方法直接返回成功标志，实际上没有具体的业务含义，可能只是为了测试手机服务端启动的成功与否

public class Ping extends CommandBase {
	@Override
	public String execute(JSONObject args) throws JSONException {
		try {
			return success((Object) "it works!");
		} catch (JSONException e) {
			e.printStackTrace();
			return failed("UnknownError");
		}
	}
}

4.2 Wake 模拟电源按键，核心调用的UiDevice中的wakeUp方法

public class Wake extends CommandBase {
	@Override
	public String execute(JSONObject args) throws JSONException {
		try {
			UiDevice.getInstance().wakeUp();
			return success((Object) true);
		} catch (final RemoteException e) {
			Utils.output(e.toString());
		}
		return failed("UnknownError");
	}
}

4.3 SetText 设置文本内容，该处理类也是同样根据传递进来的ElementId获取到macaca封装的Element元素后，调用setText方法，核心也是调用的UiObject中的setText方法

public class SetText extends CommandBase {
    @Override
    public String execute(JSONObject args) throws JSONException {
        try {

            String elementId = (String) args.get("elementId");
            String text = (String) args.get("text");

            Element element = Elements.getGlobal().getElement(elementId);

            boolean needPressEnter = false;

            if (text.endsWith("\\n")) {
                needPressEnter = true;
                text = text.replace("\\n", "");
            }

            Charset UTF7 = Charset.forName("UTF-7");
            Charset ASCII = Charset.forName("US-ASCII");

            byte[] encoded = text.getBytes(UTF7);
            String str = new String(encoded, ASCII);

            boolean result = element.setText(str);

            if (needPressEnter) {
                final UiDevice d = UiDevice.getInstance();
                d.pressEnter();
            }

            return success(result);
        } catch (final UiObjectNotFoundException e) {
            return failed("NoSuchElement");
        } catch (final Exception e) {
            return failed("UnknownError");
        }
    }
}

4.4 GetText 获取文本内容，该处理类也是同样根据传递进来的ElementId获取到macaca封装的Element元素后，调用getText方法，核心也是调用的UiObject中的getText方法

public class GetText extends CommandBase {
	@Override
	public String execute(JSONObject args) throws JSONException {

		try {
			String elementId = (String) args.get("elementId");
			Element element = Elements.getGlobal().getElement(elementId);
			return success(element.getText());
		} catch (final Exception e) {
			return failed("UnknownError");
		}
	}
}

4.5 Click 点击操作该处理类也是同样根据传递进来的ElementId获取到macaca封装的Element元素后，调用click方法，核心也是调用的UiObject中的click方法

public class Click extends CommandBase {
    @Override
    public String execute(JSONObject args) throws JSONException {
        try {
            String elementId = (String) args.get("elementId");
            Element el = Elements.getGlobal().getElement(elementId);

            el.click();
            return success(true);
        } catch (final UiObjectNotFoundException e) {
            return failed("NoSuchElement");
        } catch (final Exception e) {
            return failed("UnknownError");
        }
    }
}

4.6 ClearText 清空文本该操作类通过参数中获取的元素id获取到元素后，直接调用Element元素的clearText方法，而Element元素是macaca自己封装的Uiautomator中的UIObject类中的clearTextField方法

public class ClearText extends CommandBase {
    @Override
    public String execute(JSONObject args) throws JSONException {
        try {
            String elementId = (String) args.get("elementId");
            Element el = Elements.getGlobal().getElement(elementId);

            el.clearText();
            return success(true);
        } catch (final UiObjectNotFoundException e) {
            return failed("NoSuchElement");
        } catch (final Exception e) {
            return failed("UnknownError");
        }
    }
}

  public UiObject element;
  /**
  * @throws UiObjectNotFoundException
  */
  public void clearText() throws UiObjectNotFoundException {
    element.clearTextField();
  }

4.7 Swipe 滑动该处理类直接根据参数获取（开始位置坐标，结束位置坐标）+持续时间，然后直接调用UiDevice中的swipe滑动方法实现

public class Swipe extends CommandBase {
    @Override
    public String execute(JSONObject args) throws JSONException {
        try {
            Integer startX = (Integer) args.get("startX");
            Integer startY = (Integer) args.get("startY");
            Integer endX = (Integer) args.get("endX");
            Integer endY = (Integer) args.get("endY");
            Integer duration = (Integer) args.get("duration");
            boolean result = UiDevice.getInstance().swipe(startX, startY, endX, endY, duration);

            return success(result);
        } catch (final Exception e) {
            return failed("UnknownError");
        }
    }
}

4.8 GetWindowSize 获取手机长宽直接调用UiDevice获取长度和宽度返回

public class GetWindowSize extends CommandBase {
	@Override
	public String execute(JSONObject args) throws JSONException {
		try {
			Integer width = UiDevice.getInstance().getDisplayWidth();
			Integer height = UiDevice.getInstance().getDisplayHeight();
			JSONObject size = new JSONObject();
			size.put("width", width);
			size.put("height", height);
			return success(size.toString());
		} catch(JSONException e) {
			return failed("UnknownError");
		}
	}
}

4.9 GetProperties 获取手机属性包括元素长度、宽度、中心坐标、左上位置坐标、面积

public class GetProperties extends CommandBase {
    @Override
    public String execute(JSONObject args) throws JSONException {
        try {
            String elementId = (String) args.get("elementId");
            Element el = Elements.getGlobal().getElement(elementId);
            final Rect rect = el.element.getBounds();
            JSONObject size = new JSONObject();
            size.put("width", rect.width());
            size.put("height", rect.height());
            size.put("centerX", rect.centerX());
            size.put("centerY", rect.centerY());
            JSONObject origin = new JSONObject();
            origin.put("x", rect.left);
            origin.put("y", rect.top);
            JSONObject props = new JSONObject();
            props.put("origin", origin);
            props.put("size", size);
            return success(props.toString());
        } catch (final UiObjectNotFoundException e) {
            return failed("NoSuchElement");
        } catch (final Exception e) {
            return failed("UnknownError");
        }
    }
}

4.10 GetSource 获取当前画面的dump的xml文件直接调用UiDevice中的dump方法生成macaca-dump.xml文件，然后在js中通过adb命令pull拉取到电脑本地文件系统中

public class GetSource extends CommandBase {

    private static final String dumpFileName = "macaca-dump.xml";

    @Override
    public String execute(JSONObject args) throws JSONException {
        try {

            final File dump = new File(new File(Environment.getDataDirectory(),
                    "local/tmp"), dumpFileName);
            dump.mkdirs();
            if (dump.exists()) {
                dump.delete();
            }
            UiDevice.getInstance().dumpWindowHierarchy(dumpFileName);
            return success(true);
        } catch (final Exception e) {
            return failed("UnknownError");
        }
    }
}

4.11 Find 查找页面元素首先传递参数需要三个

策略：即为通过什么来获取元素，这里有class_Name、name、id、xpath

值:即策略所对应的取值

是否支持多个查询:是否支持多个查询或者找到匹配元素即返回

public class Find extends CommandBase {

	private Elements elements = Elements.getGlobal();

	@Override
	public String execute(JSONObject args) throws JSONException,
			ParserConfigurationException {
		try {
			String strategy = (String) args.get("strategy");
			String selector = (String) args.get("selector");
			Boolean multiple = (Boolean) args.get("multiple");

			strategy = strategy.trim().replace(" ", "_").toUpperCase();
			Object result = null;
			List<UiSelector> selectors = null;
			try {
				selectors = getSelectors(strategy, selector,
						multiple);
			} catch (Exception e) {
				e.printStackTrace();
			}
			boolean found = false;

			if (multiple) {
				List<Element> foundElements = new ArrayList<Element>();
				for (UiSelector item : selectors) {
					try {
						final List<Element> elementsFromSelector = getElements(item);
						foundElements.addAll(elementsFromSelector);
					} catch (final UiObjectNotFoundException ignored) {
					}
				}
				found = foundElements.size() > 0;
				result = elementsToJSONArray(foundElements);

			} else {
				for (int i = 0; i < selectors.size() && !found; i++) {
					try {
						result = getElement(selectors.get(i));
						found = result != null;
					} catch (Exception ignored) {
						Utils.output("ignored selector");
					}
				}
			}

			if (!found) {
				return failed("ElementNotFound");
			}

			return success((Object) result);
		} catch (JSONException e) {
			e.printStackTrace();
			return failed("UnknownError");
		}
	}

	/**
	 * @param strategy
	 * @param text
	 * @param multiple
	 * @return res
	 * @throws ParserConfigurationException
	 */
	private List<UiSelector> getSelectors(String strategy, String text,
			boolean multiple) throws Exception {

		final List<UiSelector> list = new ArrayList<UiSelector>();

		UiSelector selectors = new UiSelector();

		if (strategy.equals("CLASS_NAME")) {
			selectors = selectors.className(text);

			if (!multiple) {
				selectors = selectors.instance(0);
			}
			list.add(selectors);
		} else if (strategy.equals("NAME")) {
			selectors = new UiSelector().description(text);
			if (!multiple) {
				selectors = selectors.instance(0);
			}
			list.add(selectors);

			selectors = new UiSelector().text(text);
			if (!multiple) {
				selectors = selectors.instance(0);
			}
			list.add(selectors);
		} else if (strategy.equals("ID")) {
			selectors = selectors.resourceId(text);

			if (!multiple) {
				selectors = selectors.instance(0);
			}
			list.add(selectors);
		} else if(strategy.equals("XPATH")) {
			final ArrayList<UiSelector> pairs = XmlUtils.getSelectors(text);

			if (!multiple) {
				if (pairs.size() == 0) {
					throw new Exception("Could not find an element using given xpath expression.");
				}
				list.add(pairs.get(0));
			} else {
				for (final UiSelector pair : pairs) {
					list.add(pair);
				}
			}
		}

		return list;
	}

	/**
	 * @param sel
	 * @return res
	 * @throws JSONException
	 * @throws Exception
	 */
	private JSONObject getElement(final UiSelector sel) throws JSONException,
			Exception {
		final JSONObject res = new JSONObject();
		final Element element = getElements().getElement(sel);
		return res.put("ELEMENT", element.getId());
	}

	/**
	 * @param sel
	 * @return res
	 * @throws UiObjectNotFoundException
	 */
	private ArrayList<Element> getElements(final UiSelector sel)
			throws UiObjectNotFoundException {
		return elements.getElements(sel);
	}

	/**
	 * @param elems
	 * @return res
	 * @throws JSONException
	 */
	private JSONArray elementsToJSONArray(final List<Element> elems)
			throws JSONException {
		JSONArray resArray = new JSONArray();
		for (Element element : elems) {
			resArray.put(new JSONObject().put("ELEMENT", element.getId()));
		}
		return resArray;
	}

	/**
	 * @return res
	 */
	public Elements getElements() {
		return elements;
	}

	/**
	 * @param elements
	 */
	public void setElements(Elements elements) {
		this.elements = elements;
	}
}

疑问

既然macaca封装的时候通过两种方式进行封装，adb命令和uiautomator，那么何时使用adb？何时使用Uiautomator呢？带着疑问向macaca团队提出issue，相关疑问解答参考连接macaca-android驱动封装规则

github从源代码库更新到自己fork的代码库

2016-10-04T22:42:00+08:00

github从源代码库更新到自己fork的代码库

1.以自己从macacajs源代码库中fork的https://github.com/CodeToSurvive1/app-inspector.git为例子

切换到本地app-inspector目录中

mac:app-inspector mac$ git remote -v
origin	https://github.com/CodeToSurvive1/app-inspector.git (fetch)
origin	https://github.com/CodeToSurvive1/app-inspector.git (push)

2.添加上游源代码的仓库地址

mac:app-inspector mac$ git remote add upstream https://github.com/macacajs/app-inspector.git
mac:app-inspector mac$ git remote -v
origin	https://github.com/CodeToSurvive1/app-inspector.git (fetch)
origin	https://github.com/CodeToSurvive1/app-inspector.git (push)
upstream	https://github.com/macacajs/app-inspector.git (fetch)
upstream	https://github.com/macacajs/app-inspector.git (push)

如果不小心添加错了，可以删除

mac:app-inspector mac$ git remote remove upstream

3.从远程下载分支

mac:app-inspector mac$ git fetch upstream
From https://github.com/macacajs/app-inspector
 * [new branch]      fix-bounds-parse -> upstream/fix-bounds-parse
 * [new branch]      gh-pages   -> upstream/gh-pages
 * [new branch]      master     -> upstream/master
 * [new branch]      support-real-device -> upstream/support-real-device

4.检出本地分支

mac:app-inspector mac$ git checkout master
Already on \'master\'
Your branch is up-to-date with \'origin/master\'.

5.合并远程分支的变化到本地master仓库，这一步会合并远程代码库中的更新到本地仓库中，不会丢失本地的修改

mac:app-inspector mac$ git merge upstream/master
Updating e0695e8..6f8c6a7
Fast-forward
 .gitignore                   |  1 +
 .npmignore                   |  1 +
 assets/components/index.jsx  |  6 +++---
 assets/components/screen.jsx | 11 ++---------
 assets/components/tree.less  |  2 +-
 assets/index.less            |  1 +
 assets/libs/bounds.js        | 51 ++++++++++-----------------------------------------
 lib/android.js               | 19 +++++++++++--------
 lib/ios.js                   | 63 +++++++++++++++++++++++++++++++++++++++++----------------------
 package.json                 |  3 ++-
 public/dist/index.js         |  8 --------
 11 files changed, 73 insertions(+), 93 deletions(-)
 delete mode 100644 public/dist/index.js
mac:app-inspector mac$

6.提交到github仓库

mac:app-inspector mac$ git push origin master
Everything up-to-date
mac:app-inspector mac$

app-inspector代码启动

2016-10-04T00:00:00+08:00

app-inspector代码启动

1.从github下载代码

mac:macacajs mac$ git clone https://github.com/macacajs/app-inspector.git
Cloning into \'app-inspector\'...
remote: Counting objects: 352, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 352 (delta 2), reused 0 (delta 0), pack-reused 338
Receiving objects: 100% (352/352), 186.55 KiB | 55.00 KiB/s, done.
Resolving deltas: 100% (160/160), done.
Checking connectivity... done.

2.安装依赖包，安装成功

mac:macacajs mac$ cd app-inspector/
mac:app-inspector mac$ ls
HISTORY.md      README.md       index.js        package.json        views
LICENSE         assets          issue_template.md   public          webpack.config.js
Makefile        bin         lib         test
mac:app-inspector mac$ npm install
npm WARN deprecated cross-spawn-async@2.2.4: cross-spawn no longer requires a build toolchain, use it instead!
npm WARN deprecated win-spawn@2.0.0: use [cross-spawn](https://github.com/IndigoUnited/node-cross-spawn) or [cross-spawn-async](https://github.com/IndigoUnited/node-cross-spawn-async) instead.
npm WARN deprecated tough-cookie@2.2.2: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
。。。。
nunjucks@2.5.2 node_modules/nunjucks
├── asap@2.0.5
├── yargs@3.32.0 (decamelize@1.2.0, y18n@3.2.1, camelcase@2.1.1, window-size@0.1.4, os-locale@1.4.0, string-width@1.0.2, cliui@3.2.0)
└── chokidar@1.6.0 (path-is-absolute@1.0.0, async-each@1.0.1, inherits@2.0.3, glob-parent@2.0.0, is-glob@2.0.1, is-binary-path@1.0.1, readdirp@2.1.0, anymatch@1.3.0, fsevents@1.0.14)
webpack@1.13.2 node_modules/webpack
├── interpret@0.6.6
├── tapable@0.1.10
├── async@1.5.2
├── clone@1.0.2
├── supports-color@3.1.2 (has-flag@1.0.0)
├── loader-utils@0.2.16 (object-assign@4.1.0, big.js@3.1.3, emojis-list@2.0.1, json5@0.5.0)
├── enhanced-resolve@0.9.1 (graceful-fs@4.1.8, memory-fs@0.2.0)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── optimist@0.6.1 (wordwrap@0.0.3, minimist@0.0.10)
├── acorn@3.3.0
├── memory-fs@0.3.0 (errno@0.1.4, readable-stream@2.1.5)
├── webpack-core@0.6.8 (source-map@0.4.4, source-list-map@0.1.6)
├── uglify-js@2.6.4 (async@0.2.10, uglify-to-browserify@1.0.2, source-map@0.5.6, yargs@3.10.0)
├── node-libs-browser@0.6.0 (https-browserify@0.0.0, punycode@1.4.1, string_decoder@0.10.31, tty-browserify@0.0.0, path-browserify@0.0.0, constants-browserify@0.0.1, os-browserify@0.1.2, process@0.11.9, assert@1.4.1, domain-browser@1.1.7, querystring-es3@0.2.1, timers-browserify@1.4.2, stream-browserify@1.0.0, events@1.1.1, readable-stream@1.1.14, util@0.10.3, console-browserify@1.1.0, url@0.10.3, vm-browserify@0.0.4, http-browserify@1.7.0, buffer@4.9.1, browserify-zlib@0.1.4, crypto-browserify@3.2.8)
└── watchpack@0.2.9 (graceful-fs@4.1.8, async@0.9.2, chokidar@1.6.0)

3.修改package.json 中的uuid，并使用npm run start

需要先修改 package.json 里面的 UID：
"start": "APP_INSPECTOR=dev ./bin/app-inspector -u DU2MYN146D002912 --verbose",
使用 npm run start 启动

4.或者使用下面的

如果有 APP_INSPECTOR=dev环境变量，服务端会使用 webpack-dev-middle，静态资源直接从源码动态编译。

mac:app-inspector mac$ APP_INSPECTOR=dev ./bin/app-inspector -u DU2MYN146D002912

或者

如果没有 APP_INSPECTOR=dev环境变量，服务器会从 dist 目录读取打包后的静态文件，而打包文件是没有提交到 git 的。如果要得到 dist 文件需要使用npm run build编译。

mac:app-inspector mac$ npm run build
mac:app-inspector mac$ ./bin/app-inspector -u DU2MYN146D002912

参考来源

https://github.com/alibaba/macaca/issues/228

macaca环境搭建

2016-10-03T00:00:00+08:00

macaca环境搭建

软件安装前提

JAVA环境
安卓环境（如果需要测试安卓设备）
nodejs   
cnpm

前提软件安装

1.JAVA环境安装，一路安装

2.设置JAVA环境变量

mac:~ mac$ sudo vi /etc/profile
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home
export CLASS_PATH=$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$CLASS_PATH

mac:~ mac$ source /etc/profile

3.下载Android sdk,解压后进入tools目录下，打开Android程序出现下面截图，勾选并进行安装

4.安装完成后，配置android环境变量

export ANDROID_HOME=/Users/mac/software/android-sdk-macosx
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools

mac:~ mac$ source /etc/profile

5.安装nodejs,一路下一步

6.安装cnpm国内镜像

mac:~ mac$ npm install -g cnpm --registry=https://registry.npm.taobao.org
npm ERR! Darwin 15.6.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "cnpm" "--registry=https://registry.npm.taobao.org"
npm ERR! node v4.5.0
npm ERR! npm  v2.15.9
npm ERR! path /usr/local/lib/node_modules/cnpm
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall rmdir
npm ERR! Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/cnpm'
npm ERR!     at Error (native)
npm ERR!  { [Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/cnpm']
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'rmdir',
npm ERR!   path: '/usr/local/lib/node_modules/cnpm' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.
npm ERR! error rolling back Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/cnpm'
npm ERR! error rolling back     at Error (native)
npm ERR! error rolling back  { [Error: EACCES: permission denied, rmdir '/usr/local/lib/node_modules/cnpm']
npm ERR! error rolling back   errno: -13,
npm ERR! error rolling back   code: 'EACCES',
npm ERR! error rolling back   syscall: 'rmdir',
npm ERR! error rolling back   path: '/usr/local/lib/node_modules/cnpm' }
npm ERR! Please include the following file with any support request:
npm ERR!     /Users/mac/npm-debug.log

出现上面报错时候可以进行下面操作

mac:~ mac$ sudo mkdir -p /usr/local/lib/node_modules/cnpm
mac:~ mac$ sudo chown -R mac /usr/local/lib/node_modules/
mac:~ mac$ sudo chown -R mac /usr/local/bin/

macaca相关软件安装

1.安装macaca服务端程序

npm install macaca-cli -g

2.如果需要测试安卓设备

cnpm install macaca-android -g

3.测试浏览器需要

cnpm install macaca-chrome -g

4.测试ios设备需要安装

首先需要安装homebrew

mac:~ mac$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

安装ios代理

brew install ios-webkit-debug-proxy
brew install usbmuxd 

安装IOS驱动

cnpm install macaca-ios -g

5.测试电脑端需要安装

cnpm install macaca-electron -g

6.如果想通过页面审查设备元素，可以安装app-inspector

cnpm install app-inspector -g

7.检查环境是否成功

mac:~ mac$ macaca doctor
  macaca-doctor version: 1.0.19
  Node.js checklist:
  node env: /usr/local/bin/node
  node version: v4.5.0
  iOS checklist:
  Xcode is installed at: `/Applications/Xcode.app/Contents/Developer`
  Xcode Command Line Tools is ready, version: 2343.
  ios_webkit_debug_proxy is installed at: /usr/local/bin/ios_webkit_debug_proxy
  Android checklist:
  JAVA version is `1.8.0_91`
  JAVA_HOME is set to `/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home`
  ANDROID_HOME is set to `/Users/mac/software/android-sdk-macosx`
  Platforms is set to `/Users/mac/software/android-sdk-macosx/platforms/android-24`
  Android tools is set to `/Users/mac/software/android-sdk-macosx/tools/android`
  ADB tool is set to `/Users/mac/software/android-sdk-macosx/platform-tools/adb`
  Installed driver list:
  android: 1.0.28
  chrome: 1.0.2
  electron: 1.1.2
  ios: 1.0.37

8.运行脚本过程中报错或者inspector过程中报错，打开xcode->proferences找到下面的accounts,并登陆开发者账号

比如，安装完app-inspector通过inspector查看后台报错

插入ios真机设备，查看序列号

idevice_id -l
9daef85429600e88ea5442ba3d5fee548c02aeac

启动inspector

inspector -u 9daef85429600e88ea5442ba3d5fee548c02aeac --verbose
>> request.js:24:12 [master] pid:3167 get remote update info failed.
>> server.js:16:12 [master] pid:3167 server start with config:
 { port: 5678,
  verbose: true,
  udid: '9daef85429600e88ea5442ba3d5fee548c02aeac',
  ip: '192.168.31.164',
  host: 'mac.local',
  loaded_time: '2016-11-07 21:01:21' }
>> app-inspector.js:42:10 [master] pid:3167 server start at: http://192.168.31.164:5678
>> xctest-client.js:46:14 [master] pid:3167 project path: /usr/local/lib/node_modules/.app-inspector_npminstall/node_modules/.1.0.28@webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj
>> xctest-client.js:215:12 [master] pid:3167 xcode version: 8.1
>> WebDriverAgent version: 1.0.28
>> xctest-client.js:160:14 [master] pid:3167 2016-11-07 21:01:24.258 xcodebuild[3174:78023]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/mac/Library/Developer/Xcode/DerivedData/WebDriverAgent-bigkcildozwtjqcpfxzovxskdqza/Logs/Test/79C8FCCF-B531-40EC-85B8-59D11FA5B4FA/Session-WebDriverAgentRunner-2016-11-07_210124-LVSTWc.log

>> xctest-client.js:160:14 [master] pid:3167 2016-11-07 21:01:24.258 xcodebuild[3174:78017] [MT] IDETestOperationsObserverDebug: (168B0761-A302-4A92-908A-01BD523994ED) Beginning test session WebDriverAgentRunner-168B0761-A302-4A92-908A-01BD523994ED at 2016-11-07 21:01:24.257 with Xcode 8B62 on target <DVTiOSDevice: 0x7fbf64f26870> {
    deviceSerialNumber:         FK2P1TWEG5QY
    identifier:                 9daef85429600e88ea5442ba3d5fee548c02aeac
    deviceClass:                iPhone
    deviceName:                 李小蛟的 iPhone
    deviceIdentifier:           9daef85429600e88ea5442ba3d5fee548c02aeac
    productVersion:             10.1.1
    buildVersion:               14B100
    deviceSoftwareVersion:      10.1.1 (14B100)
    deviceArchitecture:         arm64
    deviceTotalCapacity:        58959704064
    deviceAvailableCapacity:    44852158464
    deviceIsTransient:          NO
    ignored:                    NO
    deviceIsBusy:               NO
    deviceIsActivated:          YES
    deviceActivationState:      Activated
    deviceType:                 <DVTDeviceType:0x7fbf64bb3650 Xcode.DeviceType.iPhone>
    supportedDeviceFamilies:    (
    1
)
    applications:              (null)
    provisioningProfiles:      (null)
    activityProgress:          -2
    activityTitle:             
    hasInternalSupport:        NO
    isSupportedOS:             YES
    developerDiskMountError:   (null)
(null)
  } (10.1.1 (14B100))

>> xctest-client.js:160:14 [master] pid:3167 2016-11-07 21:01:30.215 xcodebuild[3174:78017] Error Domain=IDETestOperationsObserverErrorDomain Code=3 "Test operation was canceled. If you believe this error represents a bug, please attach the log file at /Users/mac/Library/Developer/Xcode/DerivedData/WebDriverAgent-bigkcildozwtjqcpfxzovxskdqza/Logs/Test/79C8FCCF-B531-40EC-85B8-59D11FA5B4FA/Session-WebDriverAgentRunner-2016-11-07_210124-LVSTWc.log" UserInfo={NSLocalizedDescription=Test operation was canceled. If you believe this error represents a bug, please attach the log file at /Users/mac/Library/Developer/Xcode/DerivedData/WebDriverAgent-bigkcildozwtjqcpfxzovxskdqza/Logs/Test/79C8FCCF-B531-40EC-85B8-59D11FA5B4FA/Session-WebDriverAgentRunner-2016-11-07_210124-LVSTWc.log}

>> xctest-client.js:160:14 [master] pid:3167 
Testing failed:

>> xctest-client.js:160:14 [master] pid:3167  Signing for "WebDriverAgentRunner" requires a development team. Select a development team in the project editor.
  Code signing is required for product type 'UI Testing Bundle' in SDK 'iOS 10.1'
** TEST FAILED **


The following build commands failed:
  Check dependencies
(1 failure)

>> xctest-client.js:168:14 [master] pid:3167 xctest client exit with code: 65, signal: null

可以看到上面的提示信息是需要选择开发帐号,路径也能在上面找到/usr/local/lib/node_modules/.app-inspector_npminstall/node_modules/.1.0.28@webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj

用xcode打开/usr/local/lib/node_modules/.app-inspector_npminstall/node_modules/.1.0.28@webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj，请对应上面的报错的日志中输出的路径打开,按照下面步骤进行修改,修改后再次运行命令即可

同样，进行脚本运行的时候也可能会出现错误，比如下面跑脚本报错,

mac:changshutest mac$ macaca run -d js/weixin.js --verbose
>> request.js:24:12 [master] pid:4046 get remote update info failed.
>> index.js:17:12 [master] pid:4047 webdriver server start with config:
 { port: 3456,
  verbose: false,
  always: true,
  window: true,
  ip: '192.168.31.164',
  host: 'mac.local',
  loaded_time: '2016-11-07 21:24:47' }
>> middlewares.js:17:10 [master] pid:4047 base middlewares attached
>> router.js:129:10 [master] pid:4047 router set
>> webdriver sdk launched
>> 

>> 

>>   macaca mobile sample

>> responseHandler.js:11:12 [master] pid:4047 Recieve HTTP Request from Client: method: POST url: /wd/hub/session, jsonBody: {"desiredCapabilities":{"autoAcceptAlerts":true,"platformName":"iOS","platformVersion":"9.3.4","deviceName":"iPhone6 plus","udid":"9daef85429600e88ea5442ba3d5fee548c02aeac","bundleId":"com.tencent.xin"}}
>> session.js:47:10 [master] pid:4047 Creating session, sessionId: 52c618f1-79c3-49bb-ac3d-ef78e3cb8581.
>> xctest-client.js:46:14 [master] pid:4047 project path: /usr/local/lib/node_modules/.macaca-ios_npminstall/node_modules/.1.0.28@webdriveragent/WebDriverAgent/WebDriverAgent.xcodeproj
>> macaca-ios.js:144:10 [master] pid:4047 {
  "bundleId": "com.tencent.xin",
  "platformName": "iOS",
  "platformVersion": "9.3.4"
}
>> xctest-client.js:215:12 [master] pid:4047 xcode version: 8.1
>> WebDriverAgent version: 1.0.28
>> xctest-client.js:160:14 [master] pid:4047 2016-11-07 21:24:49.160 xcodebuild[4058:91967]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/mac/Library/Developer/Xcode/DerivedData/WebDriverAgent-bagqeermmcadmcctywbyvbqewyeo/Logs/Test/8AA0F0C9-B54C-4C20-AFB4-175215CAC0C1/Session-WebDriverAgentRunner-2016-11-07_212449-z5pAjR.log

>> xctest-client.js:160:14 [master] pid:4047 2016-11-07 21:24:49.160 xcodebuild[4058:91966] [MT] IDETestOperationsObserverDebug: (D8B50A66-C013-4DE1-B715-2839FD6DF5D2) Beginning test session WebDriverAgentRunner-D8B50A66-C013-4DE1-B715-2839FD6DF5D2 at 2016-11-07 21:24:49.159 with Xcode 8B62 on target <DVTiOSDevice: 0x7fde1135db10> {
    deviceSerialNumber:         FK2P1TWEG5QY
    identifier:                 9daef85429600e88ea5442ba3d5fee548c02aeac
    deviceClass:                iPhone
    deviceName:                 李小蛟的 iPhone
    deviceIdentifier:           9daef85429600e88ea5442ba3d5fee548c02aeac
    productVersion:             10.1.1
    buildVersion:               14B100
    deviceSoftwareVersion:      10.1.1 (14B100)
    deviceArchitecture:         arm64
    deviceTotalCapacity:        58959704064
    deviceAvailableCapacity:    44848533504
    deviceIsTransient:          NO
    ignored:                    NO
    deviceIsBusy:               NO
    deviceIsActivated:          YES
    deviceActivationState:      Activated
    deviceType:                 <DVTDeviceType:0x7fde10f31c10 Xcode.DeviceType.iPhone>
    supportedDeviceFamilies:    (
    1
)
    applications:              (null)
    provisioningProfiles:      (null)
    activityProgress:          -2
    activityTitle:             
    hasInternalSupport:        NO
    isSupportedOS:             YES
    developerDiskMountError:   (null)
(null)
  } (10.1.1 (14B100))

>> xctest-client.js:160:14 [master] pid:4047 2016-11-07 21:24:51.911 xcodebuild[4058:91966] Error Domain=IDETestOperationsObserverErrorDomain Code=3 "Test operation was canceled. If you believe this error represents a bug, please attach the log file at /Users/mac/Library/Developer/Xcode/DerivedData/WebDriverAgent-bagqeermmcadmcctywbyvbqewyeo/Logs/Test/8AA0F0C9-B54C-4C20-AFB4-175215CAC0C1/Session-WebDriverAgentRunner-2016-11-07_212449-z5pAjR.log" UserInfo={NSLocalizedDescription=Test operation was canceled. If you believe this error represents a bug, please attach the log file at /Users/mac/Library/Developer/Xcode/DerivedData/WebDriverAgent-bagqeermmcadmcctywbyvbqewyeo/Logs/Test/8AA0F0C9-B54C-4C20-AFB4-175215CAC0C1/Session-WebDriverAgentRunner-2016-11-07_212449-z5pAjR.log}

>> xctest-client.js:160:14 [master] pid:4047 
Testing failed:

>> xctest-client.js:160:14 [master] pid:4047  Signing for "WebDriverAgentRunner" requires a development team. Select a development team in the project editor.
  Code signing is required for product type 'UI Testing Bundle' in SDK 'iOS 10.1'
** TEST FAILED **


The following build commands failed:
  Check dependencies
(1 failure)

>> xctest-client.js:168:14 [master] pid:4047 xctest client exit with code: 65, signal: null

一样从上面找到

同样按照上面的步骤进行操作，操作后再次运行即可

9.Product->build

mac:macaca-test-sample mac$ idevice_id -l
9daef85429600e88ea5442ba3d5fee548c02aeac

mac:macaca-test-sample mac$ inspector -u 9daef85429600e88ea5442ba3d5fee548c02aeac --verbose

小米论坛爬虫

2016-10-02T00:00:00+08:00

xiaomi_forum_spider

小米论坛爬虫

软件前提

需要安装python环境
pip install scrapy(python版本爬虫)
pip install MySQL-python(python版本mysql驱动)
mysql数据库

数据库表结构设计

小米论坛板块表forum

CREATE TABLE `forum` (
  `forum_plates_id` int(11) NOT NULL AUTO_INCREMENT COMMENT '小米论坛板块主键',
  `forum_plates_name` varchar(40) NOT NULL COMMENT '小米论坛板块名称',
  `forum_plates_number` int(11) NOT NULL COMMENT '小米论坛板块编号',
  `forum_plates_url` varchar(256) DEFAULT NULL COMMENT '小米论坛板块访问url',
  `forum_plates_desc` varchar(256) DEFAULT NULL COMMENT '小米论坛板块说明',
  `forum_plates_followed_person_numbers` int(11) NOT NULL COMMENT '板块关注人数',
  PRIMARY KEY (`forum_plates_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='小米论坛板块';

小米论坛主题表forum_theme

CREATE TABLE `forum_theme` (
  `theme_id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主题id',
  `theme_number` int(32) NOT NULL COMMENT '主题编号，理论上不相同',
  `forum_plate_number` int(11) NOT NULL COMMENT '论坛板块编号与forum_plate_number对应',
  `theme_title` varchar(256) DEFAULT NULL COMMENT '主题名称',
  `theme_creater` varchar(256) DEFAULT NULL COMMENT '主题创建者名称',
  `theme_creater_id` int(11) DEFAULT NULL COMMENT '主题创建者id',
  `theme_read_number` int(11) NOT NULL DEFAULT '0' COMMENT '主题阅读人数',
  `theme_reply_number` int(11) NOT NULL DEFAULT '0' COMMENT '主题评论人数',
  `theme_type` varchar(64) NOT NULL COMMENT '主题分类',
  `theme_create_time` varchar(256) DEFAULT NULL COMMENT '主题创建时间',
  `theme_create_device` varchar(256) DEFAULT NULL COMMENT '主题创建设备来源',
  `theme_content` longtext COMMENT '主题内容',
  PRIMARY KEY (`theme_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;

小米论坛主题评论表forum_theme_comment

CREATE TABLE `forum_theme_comment` (
  `comment_id` int(11) NOT NULL AUTO_INCREMENT,
  `theme_number` int(32) NOT NULL COMMENT '主题编号',
  `user_id` int(11) NOT NULL,
  `user_nick_name` varchar(256) NOT NULL COMMENT '用户昵称',
  `user_group` varchar(256) DEFAULT NULL COMMENT '用户组',
  `comment_content` longblob COMMENT '评论内容',
  `comment_time` datetime DEFAULT NULL COMMENT '评论时间',
  `comment_device_type` varchar(256) DEFAULT NULL COMMENT '评论设备来源',
  `comment_floor` int(11) DEFAULT NULL COMMENT '评论楼层',
  PRIMARY KEY (`comment_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;

小米论坛用户表user

CREATE TABLE `user` (
  `user_id` int(11) NOT NULL,
  `user_miid` int(11) NOT NULL,
  `user_group` varchar(256) DEFAULT NULL,
  `user_theme_number` int(11) DEFAULT NULL COMMENT '用户主题数',
  `user_reply_number` int(11) DEFAULT NULL COMMENT '用户回复主题数',
  `user_contribution_value` int(11) DEFAULT NULL COMMENT '贡献值',
  `user_signature` varchar(512) DEFAULT NULL COMMENT '用户个性签名',
  `latest_login_time` datetime DEFAULT NULL COMMENT '最后访问时间',
  `latest_participate_time` datetime DEFAULT NULL COMMENT '上次活动时间',
  `account_register_time` datetime DEFAULT NULL COMMENT '账号注册时间',
  `latest_publish_time` datetime DEFAULT NULL COMMENT '上次发表时间',
  `user_level` varchar(31) DEFAULT NULL COMMENT 'VIP等级',
  `user_vip_number` int(11) DEFAULT NULL COMMENT '第多少位VIP',
  `user_forum_score` int(11) DEFAULT '0' COMMENT '论坛积分',
  PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

程序启动

git clone https://github.com/CodeToSurvive1/xiaomi_forum_spider.git
cd xiaomi_forum_spider
scrapy crawl xiaomi-theme