朔州如何做百度的网站,网站数据库做好了 怎么做网页,怎样打开网站制作,网站策划岗位要求一、服务器准备 1、服务器配置 准备3台服务器#xff0c;建议最低配置如下#xff08;阿里云大约每台每月六百多#xff09;#xff1a; centos7.4 8c 16G 39.101.192.109 node001 39.101.179.3 node002 39.99.236.205 node003 为服务器开启外网端口访问 7180、8900 2…一、服务器准备 1、服务器配置 准备3台服务器建议最低配置如下阿里云大约每台每月六百多 centos7.4 8c 16G 39.101.192.109 node001 39.101.179.3 node002 39.99.236.205 node003 为服务器开启外网端口访问 7180、8900 2、下载安装包 把下面列出的共12个文件下载到一台文件服务器上 下载cdh文件到/var/www/html/cloudera-repos/目录 1https://archive.cloudera.com/cdh6/6.0.0/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel 2https://archive.cloudera.com/cdh6/6.0.0/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha256 3https://archive.cloudera.com/cdh6/6.0.0/parcels/manifest.json 下载repo支持文件到/var/www/html/cloudera-repos/repodata/目录 4https://archive.cloudera.com/cdh6/6.3.2/redhat7/yum/repodata/repomd.xml 5https://archive.cloudera.com/cdh6/6.3.2/redhat7/yum/repodata/3224703272392229e02b46c2ef95286e92cce148a07c04752bcfe98870bfd675-primary.sqlite.bz2 6https://archive.cloudera.com/cdh6/6.3.2/redhat7/yum/repodata/4f1d5a908a6b4fd79988109121695168a65ef0846f178bbf32dc99f92c7cef36-filelists.sqlite.bz2 下载cm文件到/var/www/html/cloudera-repos/RPMS/x86_64/目录 7https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm 8https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm 9https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.0.0-530873.el7.x86_64.rpm 10https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.0.0-530873.el7.x86_64.rpm 11https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0update141-1.x86_64.rpm 下载文件到/var/www/html/cloudera-repos/目录 12https://archive.cloudera.com/cm6/6.0.0/allkeys.asc 3、配置文件服务器 启动一个http服务器用于安装rpm安装包以及方便后面cdh安装 cd /var/www/html/ cp ./cloudera-repos/CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha256 ./cloudera-repos/CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha cp ./cloudera-repos/CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha256 ./cloudera-repos/CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha1 python -m SimpleHTTPServer 89001234 为所有需要升级的服务器配置本地repo库 vi /etc/yum.repos.d/cloudera-manager.repo1 [cloudera-manager] nameCloudera Manager 6.0.0 baseurlhttp://node001:8900/cloudera-repos/ gpgcheck0 enabled112345 测试配置是否生效 yum deplist cloudera-manager-agent1 二、环境准备 1、设置服务器超时时间1小时 /etc/profile中添加 export TMOUT3600 # 超时时间1小时1 让配置生效 source /etc/profile1 /etc/ssh/sshd_config中添加 ClientAliveInterval 60 # 每60秒检测一次客户端是否存在 ClientAliveCountMax 3 # 检测3次12 修改配置后重启sshd服务 systemctl restart sshd1 2、配置3节点免密登录 在每一台服务器中的/etc/hosts添加节点名字 172.26.53.202 node001 172.26.53.203 node002 172.26.53.204 node003123 在每一台服务器中生成rsa**一路按回车即可 ssh-****** -t rsa1 在每一台服务器执行如下命令让3台服务器可互相免密登录 ssh-copy-id -i ~/.ssh/id_rsa.pub rootnode001 ssh-copy-id -i ~/.ssh/id_rsa.pub rootnode002 ssh-copy-id -i ~/.ssh/id_rsa.pub rootnode003123 3、 修改transparent_hugepage参数(透明大页面压缩) echo never /sys/kernel/mm/transparent_hugepage/defrag echo never /sys/kernel/mm/transparent_hugepage/enabled echo echo never /sys/kernel/mm/transparent_hugepage/defrag /etc/rc.d/rc.local echo echo never /sys/kernel/mm/transparent_hugepage/enabled /etc/rc.d/rc.local1234 三、cdh安装 1、每台服务器安装java yum install oracle-j2sdk1.8-1.8.0update141-1.x86_64.rpm1 在/etc/profile中追加如下配置 JAVA_HOME/usr/java/jdk1.8.0_141-cloudera CLASSPATH.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar PATH$PATH:$JAVA_HOME/bin:$HOME/bin:$HOME/.local/bin123 让配置生效 source /etc/profile1 2、安装mysql 下载安装包安装mysql-server wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm rpm -ivh mysql-community-release-el7-5.noarch.rpm yum update yum install mysql-server1234 设置mysql大小写不敏感非常重要 修改/etc/my.conf文件中 [mysqld] 的内容 lower_case_table_names11 启动mysql服务并把mysql加入开机启动 systemctl start mysqld systemctl enable mysqld12 登录到mysql mysql -uroot -p1 为root用户授外网访问权限 GRANT ALL ON *.* TO root% IDENTIFIED BY ##你的密码##; flush privileges;12 4、安装 sudo yum clean all sudo yum upgrade cloudera-manager-daemons cloudera-manager-agent rpm -qa cloudera-manager-*123 5、初始化数据库 mysql -uroot -p1 # Cloudera Manager的数据库 CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON scm.* TO scm% IDENTIFIED BY scm123; # Activity Monitor的数据库 CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON amon.* TO amon% IDENTIFIED BY amon123; # Reports Manager的数据库 CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON rman.* TO rman% IDENTIFIED BY rman123; # Hue的数据库 CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON hue.* TO hue% IDENTIFIED BY hue123; # Metastore的数据库 CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON metastore.* TO metastore% IDENTIFIED BY metastore123; # Sentry的数据库 CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON sentry.* TO sentry% IDENTIFIED BY sentry123; # Cloudera Navigator Audit Server的数据库 CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON nav.* TO nav% IDENTIFIED BY nav123; # Cloudera Navigator Metadata Server的数据库 CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON navms.* TO navms% IDENTIFIED BY navms123; # Oozie的数据库 CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON oozie.* TO oozie% IDENTIFIED BY oozie123; # Hive的数据库 CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON hive.* TO hive% IDENTIFIED BY hive123; flush privileges;12345678910111213141516171819202122232425262728293031 6、使用scm初始化mysql sh /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm1 7、主节点启动server systemctl start cloudera-scm-server systemctl enable cloudera-scm-server tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log systemctl status cloudera-scm-server1234 8、子节点启动agent 配置server服务器地址 vim /etc/cloudera-scm-agent/config.ini1 server_hostnode0011 启动agent服务 systemctl start cloudera-scm-agent systemctl enable cloudera-scm-agent tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log systemctl status cloudera-scm-agent1234 9、网页访问cm http://39.101.192.109:7180/ 四、异常解决方案 1、SqlExceptionHelper - Table ‘scm.CM_VERSION’ doesn’t exist 如果为linux系统mysql修改为大小写不敏感修改/etc/my.conf文件中 [mysqld] 的内容 lower_case_table_names11 systemctl restart mysql1 删除scm数据库新建数据库重新执行scm_prepare_database.sh sh /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm1 2、cm-server[38046]: ERROR StatusLogger No log4j2 configuration file found. 看一下/var/log/cloudera-scm-server/cloudera-scm-server.log日志可能为其他包中报了其他的异常导致找不到log4j2的配置。例如找不到数据表或者数据表数据为空等。 3、scm.hosts not exist 或者表无数据 1如果为linux系统mysql修改为大小写不敏感删除scm数据库后新建数据库重新执行scm_prepare_database.sh 2主机名冲突 问题解决CDH6.3.1初始化scm_prepare_database数据库scm为空_金玉满堂5的博客-CSDN博客 3未设置主机名在hosts中未设置主机名与ip的映射关系 4、cm安装完成后无法访问 1如果是自建服务器可以关闭防火墙或者开放网络端口 2如果是从阿里云或腾讯云购买的服务器通过管理控制台开放网络端口 5、找不到哈希文件 sha1文件重命名为sha要保证这里的sha的哈希在mainfest.json中一致。 在界面中修改“更多选项”中的列表添加一个新的然后再删除相当于刷新一遍缓存。 6、找不到cloudera-repos/allkeys.asc 未下载这个文件在网上下载这个文件后放到/var/www/html/cloudera-repos/目录 7、cdh离线安装无法复制文件 sha256文件重命名为sha和sha1要保证这里的sha的哈希在mainfest.json中一致。 8、主机运行状态不良 / Cluster not found 安装CDH时候出现主机运行状态不良情况的解决_紧到长卜帅的博客-CSDN博客_主机运行状况不良 在cm界面中退回上一步 rm -f /opt/cloudera-manager/cm/lib/cloudera-scm-agent/cm_guid1 或者 rm -f /var/lib/cloudera-scm-agent/cm_guid systemctl restart cloudera-scm-agent12 9、CDH HDFSNameNode is not formatted. 未使用的集群使用hadoop namenode -format 已有数据的集群CDH 生产环境: NameNode is not formatted问题处理_Asher Boone的博客-CSDN博客 10、Starting cloudera-scm-server: [FAILED] systemctl status -l cloudera-scm-server1 查看具体的失败原因如果未找到可以在日志文件中查看原因 vi /var/log/cloudera-scm-server/cloudera-scm-server.log1 11、cloudera-scm-agent日志中有错误 rm -f /opt/cloudera-manager/cm-5.11.1/lib/cloudera-scm-agent/cm_guid1 或者 rm -f /var/lib/cloudera-scm-agent/cm_guid systemctl restart cloudera-scm-agent12 12、启动cloudera-scm-agent,cloudera-scm-server时说JDK不支持要使用oracle jdk 一般在安装前就会报这个错如果你已经安装了oracle jdk检查环境变量是否配置JAVA或者删除系统中的openjdk重新安装oracle jdk 13、无法发出查询未能连接到Host Monitor 未初始化scm数据导致的可能是由于在新建集群后中断安装导致的 解决办法为在CM界面删除集群停止cm服务并在后台停止服务 systemctl stop cloudera-scm-agent systemctl stop cloudera-scm-server12 重新从 第三章的第6小结开始 14、Hive启动失败报如下错误 Command failed to run because this role has an invalid configuration. Review and correct its configuration. First error: Enable Stored Notifications in Database is required by Sentry and must be enabled.1 需要在Sentry中设置hive_enable_db_notification这个选项打上勾。 15、Hive启动失败报如下错误 Command failed to run because server kudu has an invalid configuration. Review and correct its configuration. First error: Sentry authraizion in Kudu required that Hive is configured with Kudu.1 重启Hive后Resume。 16、全部安装好后impala一直在重启看日志显示 Could not connect to meta store using any of the URIs provided. Most recent failure:.......Commection refused Failed to connect to the MetaStore Server... Failed to connect to Hive MetaStore. Retrying.123 1重启Hive、MetaStore通过命令行连接Hive看是否正常 2检查hive连接的数据库数据库名可能为hive或者metastore看里面是否有29张表表中是否有数据 3如果库中没有表或没有数据则 17、用beeline连接hiveServer2时报下面的错 FAILED: InvalidConfigurationException hive.server2.authentication can’t be none in non-testing mode 修改hive-site.xml中的testing.mode为true propertynamesentry.hive.testing.mode/namevaluetrue/value /property1234 18、通过beeline连接hive后切换database显示如下 User Hive does not have privileges for SWITCHDATABASE1 新建角色赋权限给hive用户User hdfs does not have privileges for CREATEDATABASE_lvtula的博客-CSDN博客 19、Sentry启动HA的时候报错后台提示com.mysql.jdbc.Driver找不到 在HA的节点中看一下/share/lib/有没有jdbc的jar包。如果没有下载一个到该目录MySQL :: Download Connector/J 20、HDFS、YARN webui无法访问 1如果域名绑定的是内网的IP那这个WebUI默认绑定内网IP可以在HDFS / YARN 配置中搜索“通配符”把NameNode和DataNode使用通配符的选项勾选 2搜索webui关闭kerberos的配置 任何程序错误以及技术疑问或需要解答的请添加