porting guide (apache) - huawei cloud
TRANSCRIPT
Kunpeng BoostKit for Big Data
Porting Guide (Apache)
Issue 07
Date 2021-07-13
HUAWEI TECHNOLOGIES CO., LTD.
Copyright © Huawei Technologies Co., Ltd. 2021. All rights reserved.
No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions
and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.
The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. i
Contents
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 & openEuler 20.03)..................................... 11.1 Introduction............................................................................................................................................................................... 11.2 Environment Requirements................................................................................................................................................. 21.3 Configuring the Compilation Environment.................................................................................................................... 31.3.1 Installing Basic Libraries.................................................................................................................................................... 31.3.2 Installing OpenJDK.............................................................................................................................................................. 51.3.3 Installing Maven................................................................................................................................................................... 51.3.4 Installing CMake.................................................................................................................................................................. 61.3.5 Installing Protobuf............................................................................................................................................................... 71.4 Performing Porting Analysis................................................................................................................................................ 81.5 Compiling Dependencies...................................................................................................................................................... 81.6 Compiling Hadoop.................................................................................................................................................................. 91.7 Verification.............................................................................................................................................................................. 10
2 HBase 2.1.0 Porting Guide (CentOS 7.6 & openEuler 20.03)...................................... 122.1 Introduction............................................................................................................................................................................ 122.2 Environment Requirements............................................................................................................................................... 132.3 Configuring the Compilation Environment.................................................................................................................. 142.3.1 Installing Basic Libraries.................................................................................................................................................. 142.3.2 Installing OpenJDK............................................................................................................................................................162.3.3 Installing Maven................................................................................................................................................................ 162.3.4 Installing Protobuf.............................................................................................................................................................172.4 Performing Porting Analysis..............................................................................................................................................182.5 Compiling HBase................................................................................................................................................................... 182.6 Verifying HBase..................................................................................................................................................................... 192.7 Troubleshooting..................................................................................................................................................................... 192.7.1 mojo' findbugs' Fails to Be Loaded, Causing the Default findbugs-maven-plugin Fail to Be Executed............................................................................................................................................................................................................ 192.7.2 Failed to Locate protoc During the HBase Compilation Process...................................................................... 20
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler 20.03)......................................... 213.1 Introduction............................................................................................................................................................................ 213.2 Environment Requirements............................................................................................................................................... 223.3 Configuring the Compilation Environment.................................................................................................................. 233.3.1 Installing Basic Libraries.................................................................................................................................................. 23
Kunpeng BoostKit for Big DataPorting Guide (Apache) Contents
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. ii
3.3.2 Installing OpenJDK............................................................................................................................................................253.3.3 Installing Maven................................................................................................................................................................ 253.3.4 Installing Protobuf.............................................................................................................................................................263.4 Performing Porting Analysis..............................................................................................................................................273.5 Compiling Hive...................................................................................................................................................................... 273.6 Verifying Hive......................................................................................................................................................................... 293.7 Troubleshooting..................................................................................................................................................................... 29
4 Spark 2.3.0 Porting Guide (CentOS 7.6 & openEuler 20.03).......................................304.1 Introduction............................................................................................................................................................................ 304.2 Environment Requirements............................................................................................................................................... 314.3 Configuring the Compilation Environment.................................................................................................................. 324.3.1 Installing Basic Libraries.................................................................................................................................................. 324.3.2 Installing OpenJDK............................................................................................................................................................344.3.3 Installing Maven................................................................................................................................................................ 344.3.4 Installing the R Language...............................................................................................................................................354.4 Performing Porting Analysis..............................................................................................................................................364.5 Compiling Spark.................................................................................................................................................................... 364.6 Rectifying Faults.................................................................................................................................................................... 384.6.1 "Cannot find 'R_HOME'. Please specify 'R_HOME' or make sure R is properly installed" ReportedDuring the Spark Compilation.................................................................................................................................................384.6.2 "error: cannot compile a simple Fortran program" Reported During the R Language Compilation... 394.6.3 "configure: error: --with-x=yes (default) and X11 headers/libs are not available" Reported Duringthe R Language Compilation................................................................................................................................................... 394.6.4 "/usr/bin/install: cannot stat' NEWS.pdf': No such file or directory" Reported During the R LanguageCompilation.................................................................................................................................................................................... 40
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 & openEuler 20.03)..............................415.1 Introduction............................................................................................................................................................................ 415.2 Environment Requirements............................................................................................................................................... 415.3 Configuring the Compilation Environment.................................................................................................................. 425.3.1 Installing Basic Libraries.................................................................................................................................................. 425.3.2 Installing OpenJDK............................................................................................................................................................445.3.3 Installing Maven................................................................................................................................................................ 455.4 Analyzing Software Porting...............................................................................................................................................465.5 Compiling ZooKeeper.......................................................................................................................................................... 46
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler 20.03)........................................ 486.1 Introduction............................................................................................................................................................................ 486.2 Environment Requirements............................................................................................................................................... 486.3 Configuring the Compilation Environment.................................................................................................................. 496.3.1 Installing Basic Libraries.................................................................................................................................................. 496.3.2 Installing OpenJDK............................................................................................................................................................516.3.3 Installing Maven................................................................................................................................................................ 526.4 Analyzing Software Porting...............................................................................................................................................53
Kunpeng BoostKit for Big DataPorting Guide (Apache) Contents
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. iii
6.5 Compiling Flink...................................................................................................................................................................... 536.6 Troubleshooting..................................................................................................................................................................... 556.6.1 Flink Compilation Error................................................................................................................................................... 55
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler 20.03)........................................ 577.1 Introduction............................................................................................................................................................................ 577.2 Environment Requirements............................................................................................................................................... 577.3 Configuring the Compilation Environment.................................................................................................................. 587.3.1 Installing Basic Libraries.................................................................................................................................................. 587.3.2 Installing Maven................................................................................................................................................................ 607.3.3 Installing OpenJDK............................................................................................................................................................617.4 Performing Porting Analysis..............................................................................................................................................627.5 Compilation.............................................................................................................................................................................62
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 & openEuler 20.03).......................................668.1 Introduction............................................................................................................................................................................ 668.2 Environment Requirements............................................................................................................................................... 668.3 Configuring the Compilation Environment.................................................................................................................. 678.3.1 Installing Basic Libraries.................................................................................................................................................. 688.3.2 Installing OpenJDK............................................................................................................................................................708.3.3 Installing Gradle................................................................................................................................................................. 708.4 Performing Porting Analysis..............................................................................................................................................708.5 Compiling the Source Code............................................................................................................................................... 718.6 Verifying Kafka...................................................................................................................................................................... 72
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 & openEuler 20.03).......................................739.1 Introduction............................................................................................................................................................................ 739.2 Environment Requirements............................................................................................................................................... 739.3 Configuring the Compilation Environment.................................................................................................................. 749.3.1 Installing Basic Libraries.................................................................................................................................................. 759.3.2 Installing OpenJDK............................................................................................................................................................779.3.3 Installing Gradle................................................................................................................................................................. 779.4 Performing Porting Analysis..............................................................................................................................................779.5 Compiling the Source Code............................................................................................................................................... 789.6 Verification.............................................................................................................................................................................. 79
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 & openEuler 20.03)............................... 8010.1 Introduction.......................................................................................................................................................................... 8010.2 Environment Requirements.............................................................................................................................................8010.3 Configuring the Compilation Environment............................................................................................................... 8110.3.1 Installing Basic Libraries............................................................................................................................................... 8110.3.2 Installing OpenJDK......................................................................................................................................................... 8310.3.3 Installing Gradle.............................................................................................................................................................. 8410.4 Performing Porting Analysis........................................................................................................................................... 8410.5 Compiling the Source Code.............................................................................................................................................85
Kunpeng BoostKit for Big DataPorting Guide (Apache) Contents
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. iv
11 Druid 0.12.1 Porting Guide (CentOS 7.6 & openEuler 20.03)...................................8711.1 Introduction.......................................................................................................................................................................... 8711.2 Environment Requirements.............................................................................................................................................8711.3 Configuring the Compilation Environment............................................................................................................... 8811.3.1 Installing Basic Libraries............................................................................................................................................... 8811.3.2 Installing OpenJDK......................................................................................................................................................... 9011.3.3 Installing Maven.............................................................................................................................................................. 9111.3.4 Installing Protobuf.......................................................................................................................................................... 9211.4 Performing Porting Analysis........................................................................................................................................... 9311.5 Compiling Druid.................................................................................................................................................................. 9411.6 Troubleshooting.................................................................................................................................................................. 9511.6.1 Certificate Error Reported After git clone Is Executed...................................................................................... 9511.6.2 Failed to Verify the github.com Certificate Downloaded by Using wget....................................................9511.6.3 An Error Reported When Downloading the hadoop-lzo Dependency......................................................... 9511.6.4 An Error Reported When Downloading the libthrift Dependency................................................................. 96
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 & openEuler 20.03).................................9712.1 Introduction.......................................................................................................................................................................... 9712.2 Environment Requirements.............................................................................................................................................9712.3 Configuring the Compilation Environment\.............................................................................................................. 9812.3.1 Installing Basic Libraries............................................................................................................................................... 9812.3.2 Installing OpenJDK....................................................................................................................................................... 10012.3.3 Installing Maven........................................................................................................................................................... 10112.4 Analyzing Software Porting.......................................................................................................................................... 10212.5 Compiling Phoenix........................................................................................................................................................... 103
A Change History....................................................................................................................105
Kunpeng BoostKit for Big DataPorting Guide (Apache) Contents
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. v
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6& openEuler 20.03)
1.1 Introduction
1.2 Environment Requirements
1.3 Configuring the Compilation Environment
1.4 Performing Porting Analysis
1.5 Compiling Dependencies
1.6 Compiling Hadoop
1.7 Verification
1.1 Introduction
DescriptionHadoop is an open-source distributed storage and computing framework that iswidely used for massive data storage and processing. It can process data in areliable, efficient, and scalable manner.
For more information about Hadoop, visit https://hadoop.apache.org/docs/r3.1.1.
Programming language: Java
Brief description: distributed system infrastructure
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 1
1.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software requirementsItem Version
OpenJDK 1.8.0_252
Maven 3.5.4
Hadoop 3.1.1
CMake 3.12.4
Protobuf 2.5.0
CentOSItem Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEulerItem Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 2
1.3 Configuring the Compilation Environment
1.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 3
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 4
yum install -y wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
1.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
1.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 5
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
1.3.4 Installing CMakeThe CMake version must be 3.12 or later. This document uses CMake 3.12.4 as anexample.
CentOS
The built-in CMake version of CentOS is too early. To manually install CMake,perform the following steps:
Step 1 Download the CMake installation package.wget https://cmake.org/files/v3.12/cmake-3.12.4.tar.gz
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 6
Step 2 Decompress the installation package.tar -zxf cmake-3.12.4.tar.gz
Step 3 Compile and install CMake.cd cmake-3.12.4./bootstrapmake -j8make install
----End
openEuleryum install cmake
1.3.5 Installing Protobuf
CentOS
Step 1 Install Protobuf.yum install -y protobuf protobuf-devel
Step 2 Check whether Protobuf is installed successfully.protoc --version
The installation is successful if information similar to the following is displayed:
Step 3 Install Maven.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc
----End
openEuler
Step 1 Download and decompress the source code.wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gztar -zxf protobuf-2.5.0.tar.gz
Step 2 Move the decompressed directory to the /opt/tools/installed/ directory.mv protobuf-2.5.0 /opt/tools/installed/
Step 3 Go to the /opt/tools/installed/ directory.cd /opt/tools/installed
Step 4 Download the protoc.zip package and decompress it to obtain the protoc.patchfile whose storage path can be specified, for example, to /opt/tools/installed/.wget https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zipunzip protoc.zipcp ./protoc/protoc.patch ./protobuf-2.5.0/src/google/protobuf/stubs/
Step 5 Go to the protobuf-2.5.0/src/google/protobuf/stubs/ directory and install thepatch.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 7
cd protobuf-2.5.0/src/google/protobuf/stubs/patch -p1 < protoc.patch
Step 6 Go back to the root directory of protobuf-2.5.0, compile the file, and install it inthe default directory.cd /opt/tools/installed/protobuf-2.5.0./autogen.sh && ./configure CFLAGS='-fsigned-char' && make -j8 && make install
Step 7 Deploy Protoc in the local Maven repository.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/local/bin/protoc
----End
1.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Hadoop installation package andobtain information about the third-party dependencies to be ported. For details,see Kunpeng Porting Advisor Case Study. Table 1-1 lists the third-partydependencies to be ported.
Table 1-1 Third-party dependencies to be ported
Original JAR File SO File
lz4-1.2.0.jar liblz4-java.so
netty-all-4.0.52.Final.jar libnetty_transport_native_epoll_x86_64.so
leveldbjni-all-1.8.jar libleveldbjni.so
snappy-java-1.0.5.jar libsnappy.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
1.5 Compiling DependenciesStep 1 Use the Yum source to install the basic library.
yum install openssl-devel zlib-devel automake libtool
Step 2 (Optional) Compile Snappy.
Snappy is an optional compression library. You can add compilation options to theHadoop software package.
1. Download the Snappy dependency package.URL: https://github.com/google/snappy/releases/tag/1.1.7Decompress the package and go to the extracted folder.cd snappy-1.1.7
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 8
2. Modify the CMakeLists.txt file to enable the dynamic link library compilation.vim CMakeLists.txt
Change the value of this parameter from OFF to ON.option(BUILD_SHARED_LIBS "Build shared libraries(DLLs)." ON)
3. Create a compilation directory and perform compilation.mkdir buildcd buildcmake ../make install
----End
1.6 Compiling HadoopStep 1 Download the Hadoop source code package.
wget https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz
Step 2 Decompress the source code package.tar -zxvf hadoop-3.1.1-src.tar.gz
Step 3 Go to the Hadoop directory decompressed.cd hadoop-3.1.1-src
Step 4 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository in the first line of the repositories tag.
<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository>
In addition to the dependency repository source, add the plug-in repository source.The node levels of pluginRepositories and repositories are the same.
<pluginRepositories> <pluginRepository> <id>huaweicloud-plugin</id> <url>http://mirrors.huaweicloud.com/repository/maven</url> <snapshots> <enabled>true</enabled> </snapshots> </pluginRepository></pluginRepositories>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 9
Step 5 Run the compilation command.mvn package -DskipTests -Pdist,native -Dtar -Dmaven.javadoc.skip=true
Step 6 (Optional) Add the Snappy repository compilation command.mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib64 -Dbundle.snappy -Dmaven.javadoc.skip=true
NO TE
If you have followed 1.5 Compiling Dependencies to install the Snappy dependencypackage, run the compilation commands above.
After the compilation is successful, the tar.gz package is generated in the hadoop-dist/target/ directory of the source code.
Step 7 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
NO TE
The compiled hadoop-3.1.1.tar.gz must be scanned by using the Kunpeng Porting Advisorto ensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.
For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
1.7 VerificationStep 1 For details about how to install components, see the documents at the official
website. There are two installation modes: Single-node installation and clusterinstallation.● Single-node installation: https://hadoop.apache.org/docs/r3.1.1/hadoop-
project-dist/hadoop-common/SingleCluster.html● Cluster installation: https://hadoop.apache.org/docs/r3.1.1/hadoop-project-
dist/hadoop-common/ClusterSetup.html
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 10
Step 2 Use the test program of Hadoop to calculate the PI value../hadoop-3.1.1/bin/yarn jar hadoop-3.1.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar pi 5 10
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
1 Hadoop 3.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 11
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
2.1 Introduction
2.2 Environment Requirements
2.3 Configuring the Compilation Environment
2.4 Performing Porting Analysis
2.5 Compiling HBase
2.6 Verifying HBase
2.7 Troubleshooting
2.1 Introduction
HBase OverviewHBase (short for Hadoop Database) is a distributed storage system that iscolumn-based and scalable with high reliability and excellent performance. Withthe HBase technology, a large-scale storage cluster can be set up on low-cost PCservers, allowing random and quick access of massive data.
For more information about HBase, see https://hbase.apache.org.
Programming language: Java
Brief description: distributed database
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 12
2.2 Environment Requirements
Hardware Requirements
Item Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
HBase 2.1.0
Protobuf 2.5.0
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 13
2.3 Configuring the Compilation Environment
2.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 14
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 15
yum install -y wget vim openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
2.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
2.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 16
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
2.3.4 Installing ProtobufStep 1 Install Protobuf using the Yum source.
yum install -y protobuf protobuf-devel
Step 2 Specify the executable files of Protoc.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 17
2.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the HBase installation package andobtain information about the third-party dependencies to be ported. For details,see Kunpeng Porting Advisor Case Study. Table 2-1 lists the third-partydependencies to be ported.
Table 2-1 Third-party dependencies to be ported
Original JAR Package SO File
commons-crypto-1.0.0.jar libcommons-crypto.so
netty-all-4.0.23.Final.jar libnetty-transport-native-epoll.so
leveldbjni-all-1.8.jar libleveldbjni.so
snappy-java-1.0.5.jar libsnappyjava.so
hbase-shaded-netty-2.1.0.jar liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
2.5 Compiling HBaseStep 1 Download the HBase installation package.
wget http://archive.apache.org/dist/hbase/2.1.0/hbase-2.1.0-src.tar.gz
Step 2 Decompress the package.tar -zxf hbase-2.1.0-src.tar.gz
Step 3 Go to the directory generated after the decompression and add the HuaweiKunpeng repository to the pom.xml file.cd hbase-2.1.0
Step 4 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository in the first line of the repositories tag.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 18
Step 5 Perform compilation.mvn clean package -DskipTests assembly:single
After the compilation is successful, a tar.gz package is generated in the hbase-assembly/target/ directory of the source code.
Step 6 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
NO TE
The compiled HBase package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
2.6 Verifying HBaseDeploy an HBase cluster (see the HBase Cluster Deployment Guide (CentOS7.6) and verify HBase.
2.7 Troubleshooting
2.7.1 mojo' findbugs' Fails to Be Loaded, Causing the Defaultfindbugs-maven-plugin Fail to Be Executed
SymptomExecution of the default findbugs-maven-plugin fails because the mojo 'findbugs'cannot be loaded.
SolutionChange the findbugs-maven-plugin version to 3.0.4.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 19
2.7.2 Failed to Locate protoc During the HBase CompilationProcess
Symptomprotobuf 2.5.0 has been installed. However, a message is displayed during theHBase compilation process indicating that protoc is missing.
SolutionRun the following command to specify the protoc executable file installed:
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc
Kunpeng BoostKit for Big DataPorting Guide (Apache)
2 HBase 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 20
3 Hive 3.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
3.1 Introduction
3.2 Environment Requirements
3.3 Configuring the Compilation Environment
3.4 Performing Porting Analysis
3.5 Compiling Hive
3.6 Verifying Hive
3.7 Troubleshooting
3.1 Introduction
Hive OverviewHive is a data warehouse tool running on Hadoop. It maps structured data files toa database table, provides simple SQL search functions, and converts SQLstatements into MapReduce tasks. It features low learning cost. You can use SQL-like statements to quickly implement simple MapReduce statistics and do notneed to develop dedicated MapReduce applications. Therefore, Hive is verysuitable for statistical analysis of data warehouse.
For more information about Hive, visit https://hive.apache.org/.
Programming language: Java
Brief description: Hadoop-based data warehouse tool
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 21
3.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software RequirementsItem Version
OpenJDK 1.8.0_252
Maven 3.5.4
Hive 3.0.0
CentOSItem Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEulerItem Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 22
3.3 Configuring the Compilation Environment
3.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 23
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 24
yum install -y wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
3.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
3.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 25
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
3.3.4 Installing ProtobufStep 1 Install Protobuf using the Yum source.
yum install -y protobuf protobuf-devel
Step 2 Specify the executable files of Protoc.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 26
3.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Hive installation package. For details,see Kunpeng Porting Advisor Case Study. Table 3-1 lists the third-partydependencies to be ported.
Table 3-1 Third-party dependencies to be ported
Original JAR Package SO File
commons-crypto-1.0.0.jar libcommons-crypto.so
netty-all-4.1.17.Final.jar libnetty_transport_native_epoll_x86_64.so
jline-2.12.jar libjansi.so
server-metrics-0.2.8.jar libsigar.so
lz4-1.2.0.jar liblz4-java.so
hbase-shaded-netty-1.0.1.jar liborg-apache-hadoop-hbase-shaded-netty-transport-native-epoll.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
3.5 Compiling HiveStep 1 Download the Hive installation package.
wget https://github.com/apache/hive/archive/rel/release-3.0.0.tar.gz
Step 2 Decompress the installation package.mv release-3.0.0.tar.gz hive-rel-release-3.0.0.tar.gztar -zxf hive-rel-release-3.0.0.tar.gz
Step 3 Switch to the directory where the source code package is decompressed.cd hive-rel-release-3.0.0
Step 4 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository source at the first place under therepositories tag.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 27
vim jdbc/pom.xml
Add the Kunpeng Maven repository source.
<repositories><repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository></repositories>
vim druid-handler/pom.xml
Add the Kunpeng Maven repository source.
<repositories><repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository></repositories>
Step 5 Perform compilation.mvn clean install -Pdist -DskipTests
After the compilation is successful, the tar.gz package for Hive is generated in thepackaging/target directory of the source code.
Step 6 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 28
NO TE
The compiled Hive package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
3.6 Verifying HiveDeploy a Hive cluster (see the Hive Deployment Guide (CentOS 7.6)) and verifyHive.
3.7 Troubleshooting
Symptomprotobuf-2.5.0 has been installed. However, a message is displayed during the Hivecompilation process indicating that protoc is missing.
ProcedureRun the following command to specify the protoc executable file installed:
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc
Kunpeng BoostKit for Big DataPorting Guide (Apache)
3 Hive 3.0.0 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 29
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
4.1 Introduction
4.2 Environment Requirements
4.3 Configuring the Compilation Environment
4.4 Performing Porting Analysis
4.5 Compiling Spark
4.6 Rectifying Faults
4.1 Introduction
DescriptionSpark is a unified analysis engine used for large-scale data processing. It featuresscalability and memory-based computing and has become a unified platform forquick processing of lightweight big data. Spark can be used to build the data storeand running system for various applications, such as real-time stream processing,machine learning, and interactive query.
For more information about Spark, visit https://spark.apache.org/docs/latest/building-spark.html.
Programming language: Scala
Brief description: large-scale data computing engine
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 30
4.2 Environment Requirements
Hardware Requirements
Item Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software Requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
R 3.1.1
Spark 2.3.0
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 31
4.3 Configuring the Compilation Environment
4.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 32
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 33
yum install -y wget vim openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
4.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
4.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 34
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
4.3.4 Installing the R LanguageStep 1 Download the R language installation package.
wget http://cran.rstudio.com/src/base/R-3/R-3.1.1.tar.gz
Step 2 Decompress the package.tar -zxf R-3.1.1.tar.gz
Step 3 Install dependencies.yum -y install libXt-devel.aarch64 readline-devel.aarch64
Step 4 Go to the directory generated after the decompression.cd R-3.1.1
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 35
Step 5 Perform compilation../configure --enable-R-shlib --enable-R-static-lib --with-libpng --with-jpeglib --prefix=/opt/tools/installed/R-3.1.1make all -j10 && make install
NO TE
For details about how to rectify faults that occur during the compilation of the R language,see 4.6.2 "error: cannot compile a simple Fortran program" Reported During the RLanguage Compilation, 4.6.3 "configure: error: --with-x=yes (default) and X11headers/libs are not available" Reported During the R Language Compilation, and4.6.4 "/usr/bin/install: cannot stat' NEWS.pdf': No such file or directory" ReportedDuring the R Language Compilation.
Step 6 Configure the R language environment variables.export R_HOME=/opt/tools/installed/R-3.1.1export PATH=$R_HOME/bin:$PATH
----End
4.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Spark installation package byreferring to Kunpeng Porting Advisor Case Study. Table 4-1 lists the third-partydependencies to be ported.
Table 4-1 Software porting analysis result
Original JAR File SO File
commons-crypto-1.0.0.jar libcommons-crypto.so
jline-2.12.1.jar libjansi.so
leveldbjni-all-1.8.jar libleveldbjni.so
netty-all-4.1.17.Final.jar libnetty_transport_native_epoll_x86_64.so
scala-compiler-2.11.12.jar libjansi.so
snappy-java-1.1.2.6.jar libsnappyjava.so
lz4-java-1.4.0.jar liblz4-java.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
4.5 Compiling SparkStep 1 Download the Spark installation package.
wget https://github.com/apache/spark/archive/v2.3.0.tar.gz
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 36
Step 2 Decompress the package.tar -zxf v2.3.0.tar.gz
Step 3 Go to the directory generated after the decompression.cd spark-2.3.0
Step 4 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository in the first line of the repositories tag.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>
Add Huawei Maven source to the first line of the pluginRepository tag.
<pluginRepository> <id>huaweicloud-plugin</id> <url>https://mirrors.huaweicloud.com/repository/maven</url></pluginRepository>
Step 5 Perform compilation.dev/make-distribution.sh --tgz -Pyarn,hive,hive-thriftserver,sparkr
After the compilation is complete, a compiled package is generated in the sourcecode directory.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 37
NO TE
For details about how to rectify faults that occur during the Spark compilation, see 4.6.1"Cannot find 'R_HOME'. Please specify 'R_HOME' or make sure R is properly installed"Reported During the Spark Compilation.
Step 6 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
NO TE
The compiled spark-2.3.0-bin-2.6.5.tgz must be scanned by using the Kunpeng PortingAdvisor to ensure that no x86 .so or .jar packages are contained. If the compiled directorycontains x86 .so or .jar packages, the component functions may be affected.
For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
4.6 Rectifying Faults
4.6.1 "Cannot find 'R_HOME'. Please specify 'R_HOME' ormake sure R is properly installed" Reported During the SparkCompilation
Symptom
During the Spark compilation, "Cannot find 'R_HOME'. Please specify 'R_HOME' ormake sure R is properly installed" is displayed.
Cause Analysis
The R language support is enabled during Spark compilation. You need to compileand install the R language in the environment.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 38
Solution
Step 1 Compile and install the R language in the /opt/tools/installed directory, and setthe environment variables.
For details, see 4.3.4 Installing the R Language.
Step 2 Run the Spark compilation command again.
----End
4.6.2 "error: cannot compile a simple Fortran program"Reported During the R Language Compilation
Symptom
During the R language compilation, "error: cannot compile a simple Fortranprogram" is displayed.
Cause Analysis
The gfortran package does not exist in the system.
Solution
Run the yum command to install the gfortran package in the OS image.
yum -y install gcc-gfortran.aarch64
4.6.3 "configure: error: --with-x=yes (default) and X11headers/libs are not available" Reported During the RLanguage Compilation
Symptom
During the R language compilation, "configure: error: --with-x=yes (default) andX11 headers/libs are not available" is displayed.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 39
Cause Analysis--with-x=yes (use the X Window System) is enabled by default. Therefore, youneed to install the libXt-devel module.
SolutionRun the yum command to install the module in the OS image.
yum -y install libXt-devel.aarch64
4.6.4 "/usr/bin/install: cannot stat' NEWS.pdf': No such file ordirectory" Reported During the R Language Compilation
SymptomDuring the R language compilation, "/usr/bin/install: cannot stat' NEWS.pdf': Nosuch file or directory" is displayed.
Cause AnalysisThe NEWS.pdf file is not found in the source code directory R-3.1.1. Therefore, thefile cannot be copied when the make install command is executed.
SolutionThe NEWS text file exists in the doc directory. Therefore, copy the contents of theNEWS file in the doc directory to the NEWS.pdf file.
cat doc/NEWS > doc/NEWS.pdf
Kunpeng BoostKit for Big DataPorting Guide (Apache)
4 Spark 2.3.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 40
5 ZooKeeper 3.4.6 Porting Guide (CentOS7.6 & openEuler 20.03)
5.1 Introduction
5.2 Environment Requirements
5.3 Configuring the Compilation Environment
5.4 Analyzing Software Porting
5.5 Compiling ZooKeeper
5.1 IntroductionZooKeeper is a distributed and open-source coordination service for distributedapplications, an open-source implementation of Google Chubby, and an importantcomponent of Hadoop and HBase. ZooKeeper provides alignment services fordistributed applications, including configuration maintenance, domain naming,distributed synchronization, and group services.
For more information about ZooKeeper, visit https://zookeeper.apache.org/.
5.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 41
Software Requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
CMake 3.12.4
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
5.3 Configuring the Compilation Environment
5.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 42
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 43
command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Install dependencies using the Yum source.
yum install -y wget vim openssl-devel zlib-devel autoconf automake libtool ant svn make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
5.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 44
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
5.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 45
<name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
5.4 Analyzing Software PortingUse the Kunpeng Porting Advisor to scan the ZooKeeper installation package byreferring to Kunpeng Porting Advisor Case Study. No third-party dependencyneeds to be ported.
5.5 Compiling ZooKeeperStep 1 Download the zookeeper-release-3.4.6 source code package from GitHub and
decompress it.wget https://github.com/apache/zookeeper/archive/release-3.4.6.tar.gztar -zxf release-3.4.6.tar.gz
Step 2 Go to the directory decompressed and install the dependency.cd zookeeper-release-3.4.6yum install cppunit-devel -y
Step 3 The build.xml and ivysettings.xml files need to be modified due to network proxyproblems.
1. Manually download the ant-eclipse-1.0.bin.tar.bz2 file from http://downloads.sourceforge.net/project/ant-eclipse/ant-eclipse/1.0/ant-eclipse-1.0.bin.tar.bz2, copy it to the ${src.dir}/java/ path, and comment outthe content of download and copy operations in the build.xml file.
2. Change the value of repo.maven.org in ivysettings.xml to https://mirrors.huaweicloud.com/repository/maven/.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 46
3. Modify the repo URL in the build.xml file.sed -i 's/http:\/\/repo2/https:\/\/repo1/' build.xml
Step 4 Perform compilation.ant package tar
After the compilation is successful, the zookeeper-3.4.6.tar.gz package isgenerated in the ./build/ directory of the source code directory.
Step 5 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
NO TE
The compiled zookeeper-3.4.6.tar.gz must be scanned by using the Kunpeng PortingAdvisor to ensure that no x86 .so or .jar packages are contained. If the compiled directorycontains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
5 ZooKeeper 3.4.6 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 47
6 Flink 1.8.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
6.1 Introduction
6.2 Environment Requirements
6.3 Configuring the Compilation Environment
6.4 Analyzing Software Porting
6.5 Compiling Flink
6.6 Troubleshooting
6.1 IntroductionFlink is a distributed, high-performance, and high-availability open-source big dataprocessing framework. It features in-memory computing and stream processingand is used for stateful computing of unbounded and bounded data streams.
6.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 48
Software requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
Flink 1.8.1
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
6.3 Configuring the Compilation Environment
6.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 49
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 50
command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
yum install -y wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
6.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 51
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
6.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 52
<name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
6.4 Analyzing Software PortingUse the Kunpeng Porting Advisor to scan the Flink installation package and obtaininformation about the third-party dependencies to be ported. For details, seeKunpeng Porting Advisor Case Study. Table 6-1 lists the third-partydependencies to be ported.
Table 6-1 Third-party dependencies to be ported
Original JAR File SO File
jython-standalone-2.7.1.jar libjansi.so
scala-compiler-2.12.7.jar libjansi.so
flink-shaded-netty-4.1.32.Final-6.0.jar
liborg_apache_flink_shaded_netty4_netty_transport_native_epoll_x86_64.so
frocksdbjni-5.17.2-artisans-1.0.jar
librocksdbjni32.so, librocksdbjni64.so,librocksdbjnile.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
6.5 Compiling FlinkStep 1 Download the flink-release-1.8.1 source code package from GitHub and
decompress it.wget https://github.com/apache/flink/archive/release-1.8.1.tar.gztar -zxf release-1.8.1.tar.gz
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 53
Step 2 Go to the Flink directory decompressed.cd flink-release-1.8.1
Step 3 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository in the first line of the repositories tag.
<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>
Modify the flink-dist/pom.xml file.
vim flink-dist/pom.xml
Add the following code to line 653 to add the Kunpeng Maven repository source:
<repositories><repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository></repositories>
Modify the flink-libraries/pom.xml file.
vim flink-libraries/pom.xml
Add the following code to line 69 to add the Kunpeng Maven repository source:
<repositories><repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository></repositories>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 54
Step 4 Perform compilation.mvn clean install -DskipTests -Dhadoop.version=2.7.0 -Dscala-2.12 -Dpython2.12
After the compilation is complete, the directory flink-1.8.1 is generated in flink-release-1.8.1/flink-dist/target/flink-1.8.1-bin/.
Step 5 Use the Kunpeng Porting Advisor to scan the .tar package generated after thecompilation and ensure that the .tar package does not contain the x86 .so or .jarpackages.
NO TE
The compiled flink package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled package containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
6.6 Troubleshooting
6.6.1 Flink Compilation Error
SymptomThe following error message is reported during Flink compilation: Failed toexecute goal on project flink-avro-confluent-registry: Could not resolvedependencies for project org.apache.flink:flink-avro-confluent-registry:jar:1.8.1:Could not find artifact io.confluent:kafka-schema-registry-client:jar:3.3.1 in big(http://10.93.238.51/maven/)
Cause AnalysisThe dependent third-party .jar package kafka-schema-registry-client-3.3.1.jar ismissing.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 55
SolutionDownload the available kafka-schema-registry-client-3.3.1.jar package to theMaven repository or the */.m2/repository/io/confluent/kafka-schema-registry-client/3.3.1 directory of the local cache library, and compile Flink again.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
6 Flink 1.8.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 56
7 Flink 1.9.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
7.1 Introduction
7.2 Environment Requirements
7.3 Configuring the Compilation Environment
7.4 Performing Porting Analysis
7.5 Compilation
7.1 IntroductionFlink is a distributed, high-performance, and high-availability open-source big dataprocessing framework. It features in-memory computing and stream processingand is used for stateful computing of unbounded and bounded data streams.
7.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 57
Software Requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
Flink 1.9.1
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
7.3 Configuring the Compilation Environment
7.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 58
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 59
command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing DependenciesUse the Yum source to install dependencies.
yum install -y wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
7.3.2 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 60
The installation is successful if information similar to the following is displayed:
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
7.3.3 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 61
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
7.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Flink installation package and obtaininformation about the third-party dependencies to be ported. For details, see theKunpeng Porting Advisor Case Study. Table 7-1 lists the third-partydependencies to be ported.
Table 7-1 Third-party dependencies to be ported
Original JAR File SO File
scala-compiler-2.11.12.jar libjansi.so
flink-shaded-netty-4.1.32.Final-7.0.jar
liborg_apache_flink_shaded_netty4_netty_transport_native_epoll_x86_64.so
frocksdbjni-5.17.2-artisans-1.0.jar
librocksdbjni32.so,librocksdbjni64.so,librocksdbjnile.so
flink-shaded-netty-tcnative-dynamic-2.0.25.Final-7.0. jar
liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_64.so,liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_64_fedora.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
7.5 CompilationStep 1 Download the flink-release-1.9.1 source code package and decompress it.
wget https://github.com/apache/flink/archive/release-1.9.1.tar.gztar -zxf release-1.9.1.tar.gz
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 62
Step 2 Go to the directory generated after the decompression.cd flink-release-1.9.1
Step 3 Add the reference to the Kunpeng repository to the flink-release-1.9.1/pom.xmlfile.vim pom.xml
Add the Kunpeng Maven repository in the first line under repositories.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>
Step 4 Add the reference to the Kunpeng repository to the flink-release-1.9.1/flink-dist/pom.xml file.vim flink-dist/pom.xml
Add the Kunpeng Maven repository in the first line under repositories.
<repositories> <repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository></repositories>
Step 5 Add the reference of the Kunpeng repository to the flink-release-1.9.1/flink-state-backends/flink-statebackend-rocksdb/pom.xml file.vim flink-state-backends/flink-statebackend-rocksdb/pom.xml
Add the Kunpeng Maven repository in the first line under repositories.
<repositories> <repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository></repositories>
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 63
Step 6 Add the reference to the Kunpeng repository to the flink-runtime/pom.xml file.vim flink-runtime/pom.xml
Add the Kunpeng Maven repository in the first line under repositories.
<repositories> <repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository></repositories>
Step 7 Perform compilation in the flink-release-1.9.1 directory.mvn install -DskipTests -Dtar -Dhadoop.version=2.8.3
The following figure shows that the compilation is successful.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 64
After the compilation is complete, the directory flink-1.9.1 is generated in flink-release-1.9.1/flink-dist/target/flink-1.9.1-bin/.
Step 8 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
NO TE
The compiled Flink package must be scanned by using the Kunpeng Porting Advisor toensure that no x86 .so or .jar packages are contained. If the compiled directory containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see the Kunpeng PortingAdvisor Case Study.
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
7 Flink 1.9.1 Porting Guide (CentOS 7.6 & openEuler20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 65
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
8.1 Introduction
8.2 Environment Requirements
8.3 Configuring the Compilation Environment
8.4 Performing Porting Analysis
8.5 Compiling the Source Code
8.6 Verifying Kafka
8.1 Introduction
Kafka OverviewKafka is an open-source streaming platform developed by Apache SoftwareFoundation in Scala and Java. Kafka is a distributed publish-subscribe messagingsystem with high throughput. It can process all action flow data on the customer-related websites.
For more information about Kafka, visit https://kafka.apache.org/.
Category: application
8.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 66
Item Remarks
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software requirementsItem Version
OpenJDK 1.8.0_252
Maven 3.5.4
Gradle 4.10
Kafka 2.1.0
CentOSItem Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEulerItem Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
8.3 Configuring the Compilation Environment
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 67
8.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 68
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use Yum to install dependencies.
yum install -y wget unzip openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 69
8.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
8.3.3 Installing GradleStep 1 Download and decompress the Gradle source code package to a directory (for
example, /opt/tools/installed/).wget https://downloads.gradle.org/distributions/gradle-4.10-bin.zip --no-check-certificateunzip gradle-4.10-bin.zipmv gradle-4.10 /opt/tools/installed/
Step 2 Modify the gradle environment variables.vim /etc/profile
Add the following code at the end of the /etc/profile file:export GRADLE_HOME=/opt/tools/installed/gradle-4.10export PATH=$GRADLE_HOME/bin:$PATH
Step 3 Press Esc and run :wq to save the configuration and exit.
Step 4 Make the environment variables take effect.source /etc/profile
----End
8.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Kafka installation package andobtain information about the third-party dependencies to be ported. For details,
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 70
see Kunpeng Porting Advisor Case Study. Table 8-1 lists the third-partydependencies to be ported.
Table 8-1 Third-party dependencies to be ported
Original JAR File SO File
rocksdbjni-5.14.2.jar librocksdbjni32.so
rocksdbjni-5.14.2.jar librocksdbjni64.so
rocksdbjni-5.14.2.jar librocksdbjnile.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
8.5 Compiling the Source CodeStep 1 Download the kafka-2.1.0 source code package.
Download address: https://archive.apache.org/dist/kafka/2.1.0/kafka-2.1.0-src.tgz
Step 2 Decompress the source code package.tar -zxvf kafka-2.1.0-src.tgz
Step 3 Go to the directory where the source code is located.cd kafka-2.1.0-src
Step 4 Modify the build.gradle file.vim build.gradle
Add multiple Maven repositories under the repositories section of the file.allprojects { repositories { maven { url "https://mirrors.huaweicloud.com/kunpeng/maven" } maven { url "https://mirrors.huaweicloud.com/repository/maven" } maven { url "https://plugins.gradle.org/m2/" } mavenCentral() }
Step 5 Perform compilation.gradle -g /$UserHome/gradleRepository releaseTarGz -info
The compiled package is stored in the following directory:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 71
NO TE
If connection timeout 443 error occurs during compilation, run the compilation commandagain.
Step 6 Use the Kunpeng Porting Advisor to scan the .tar package generated after thecompilation and ensure that the .tar package does not contain the x86 .so or .jarpackages.
NO TE
The compiled kafka_2.11-2.1.0.tgz must be scanned by using the Kunpeng Porting Advisorto ensure that no x86 .so or .jar packages are contained. If the compiled package containsx86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
8.6 Verifying KafkaDeploy Kafka (see Kafka Deployment Guide (CentOS 7.6)) and performverification.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
8 Kafka 2.1.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 72
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
9.1 Introduction
9.2 Environment Requirements
9.3 Configuring the Compilation Environment
9.4 Performing Porting Analysis
9.5 Compiling the Source Code
9.6 Verification
9.1 Introduction
Kafka OverviewKafka is an open-source streaming platform developed by Apache SoftwareFoundation in Scala and Java. Kafka is a distributed publish-subscribe messagingsystem with high throughput. It can process all action flow data on the customer-related websites.
For more information about Kafka, visit https://kafka.apache.org/.
Category: application
9.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 73
Item Remarks
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software RequirementsItem Version
OpenJDK 1.8.0_252
Maven 3.5.4
Gradle 4.10
Kafka 2.2.0
CentOSItem Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEulerItem Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
9.3 Configuring the Compilation Environment
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 74
9.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 75
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
yum install -y wget vim openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 76
9.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
9.3.3 Installing GradleStep 1 Download and decompress the Gradle source code package to a directory (for
example, /opt/tools/installed/).wget https://downloads.gradle.org/distributions/gradle-4.10-bin.zip --no-check-certificateunzip gradle-4.10-bin.zipmv gradle-4.10 /opt/tools/installed/
Step 2 Modify the gradle environment variables.vim /etc/profile
Add the following code at the end of the /etc/profile file:export GRADLE_HOME=/opt/tools/installed/gradle-4.10export PATH=$GRADLE_HOME/bin:$PATH
Step 3 Press Esc and run :wq to save the configuration and exit.
Step 4 Make the environment variables take effect.source /etc/profile
----End
9.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Kafka installation package andobtain information about the third-party dependencies to be ported. For details,
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 77
see the Kunpeng Porting Advisor Case Study. Table 9-1 lists the third-partydependencies to be ported.
Table 9-1 Third-party dependencies to be ported
Original JAR Package SO File
rocksdbjni-5.15.10.jar librocksdbjni32.so
rocksdbjni-5.15.10.jar librocksdbjni64.so
rocksdbjni-5.15.10.jar librocksdbjnile.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
9.5 Compiling the Source CodeStep 1 Download the kafka-2.2.0 source code package.
URL: https://archive.apache.org/dist/kafka/2.2.0/kafka-2.2.0-src.tgz
Step 2 Decompress the source code package.tar -zxvf kafka-2.2.0-src.tgz
Step 3 Go to the directory where the source code is located.cd kafka-2.2.0-src
Step 4 Modify the build.gradle file.vim build.gradle
Add multiple Maven repositories under the repositories tag, as shown in thefollowing figure.allprojects { repositories { maven { url "https://mirrors.huaweicloud.com/kunpeng/maven" } maven { url "https://mirrors.huaweicloud.com/repository/maven" } maven { url "https://plugins.gradle.org/m2/" } mavenCentral() }
Step 5 Compile the source code.gradle -g /$UserHome/gradleRepository releaseTarGz -info
NO TE
gradleRepository is the local repository directory of Gradle. You need to manually createthis directory.mkdir -p /$UserHome/gradleRepository
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 78
The compiled package is stored in the following directory:
Step 6 Use the Kunpeng Code Scanner to scan the .tar package generated aftercompilation and ensure that the .tar package contains no x86 .so or .jar packages.
NO TE
Use the Kunpeng Code Scanner to scan the compiled kafka_2.12-2.2.0.tgz package andensure that the .so or .jar packages of x86 are not contained. If the compiled directorycontains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Code Scanner, see Kunpeng Porting AdvisorCase Study.
----End
9.6 VerificationDeploy Kafka (see the Kafka Deployment Guide (CentOS 7.6)) and verify theporting result.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 79
10 Kafka 0.10.1.1 Porting Guide(CentOS 7.6 & openEuler 20.03)
10.1 Introduction
10.2 Environment Requirements
10.3 Configuring the Compilation Environment
10.4 Performing Porting Analysis
10.5 Compiling the Source Code
10.1 Introduction
Kafka Overview
Kafka is an open-source streaming platform developed by Apache SoftwareFoundation in Scala and Java. Kafka is a distributed publish-subscribe messagingsystem with high throughput. It can process all action flow data on the customer-related websites.
For more information about Kafka, visit https://kafka.apache.org/.
Category: application
10.2 Environment Requirements
Hardware Requirements
Item Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 80
Item Remarks
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
Gradle 4.10
Kafka 0.10.1.1
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
10.3 Configuring the Compilation Environment
10.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 81
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 82
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use Yum to install dependencies.
yum install -y wget unzip openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
10.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 83
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
10.3.3 Installing GradleStep 1 Download and decompress the Gradle source code package to a directory (for
example, /opt/tools/installed/).wget https://downloads.gradle.org/distributions/gradle-4.10-bin.zip --no-check-certificateunzip gradle-4.10-bin.zipmv gradle-4.10 /opt/tools/installed/
Step 2 Modify the gradle environment variables.vim /etc/profile
Add the following code at the end of the /etc/profile file:export GRADLE_HOME=/opt/tools/installed/gradle-4.10export PATH=$GRADLE_HOME/bin:$PATH
Step 3 Press Esc and run :wq to save the configuration and exit.
Step 4 Make the environment variables take effect.source /etc/profile
----End
10.4 Performing Porting AnalysisUse the Kunpeng Porting Advisor to scan the Kafka installation package andobtain information about the third-party dependencies to be ported. For details,see Kunpeng Porting Advisor Case Study. Table 10-1 lists the third-partydependencies to be ported.
Table 10-1 Third-party dependencies to be ported
Original JAR File SO File
rocksdbjni-4.9.0.jar librocksdbjni32.so
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 84
Original JAR File SO File
rocksdbjni-4.9.0.jar librocksdbjni64.so
rocksdbjni-4.9.0.jar librocksdbjnile.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
10.5 Compiling the Source CodeStep 1 Download the kafka-0.10.1.1 source code package.
wget https://archive.apache.org/dist/kafka/0.10.1.1/kafka-0.10.1.1-src.tgz
Step 2 Decompress the source code package.tar -zxvf kafka-0.10.1.1-src.tgz
Step 3 Go to the directory where the source code is located.cd kafka-0.10.1.1-src
Step 4 Modify the build.gradle file.vim build.gradle
Add multiple Maven repositories under the repositories section of the file.allprojects { apply plugin: 'idea' apply plugin: 'eclipse' apply plugin: 'jacoco'
repositories { maven { url "https://mirrors.huaweicloud.com/kunpeng/maven" } maven { url "https://mirrors.huaweicloud.com/repository/maven" } maven { url "https://plugins.gradle.org/m2/" } mavenCentral() }
Step 5 Perform compilation.gradle -g /$UserHome/gradleRepository releaseTarGz -info
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 85
The compiled package is stored in the following directory:
NO TE
If connection timeout 443 error occurs during compilation, run the compilation commandagain.
Step 6 Use the Kunpeng Porting Advisor to scan the .tar package generated after thecompilation and ensure that the .tar package does not contain the x86 .so or .jarpackages.
NO TE
The compiled kafka_2.10-0.10.1.1.tgz must be scanned by using the Kunpeng PortingAdvisor to ensure that no x86 .so or .jar packages are contained. If the compiled packagecontains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 86
11 Druid 0.12.1 Porting Guide (CentOS7.6 & openEuler 20.03)
11.1 Introduction
11.2 Environment Requirements
11.3 Configuring the Compilation Environment
11.4 Performing Porting Analysis
11.5 Compiling Druid
11.6 Troubleshooting
11.1 IntroductionDruid is an efficient data query system that is used to aggregate and query a largeamount of time-series data. Data can be ingested in real time and can be queriedimmediately after being ingested to the Druid. In addition, data is almostimmutable. After a time-series event occurs and is ingested to Druid, an externalsystem can query the event.
For more information about Druid, visit https://github.com/apache/druid.
11.2 Environment Requirements
Hardware RequirementsItem Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Drive partition No requirement for drive partitions
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 87
Item Remarks
Network Accessible to the Internet
Software Requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
CMake 3.12.4
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
11.3 Configuring the Compilation Environment
11.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 88
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 89
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
yum install -y wget unzip vim
11.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 90
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
11.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 91
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
11.3.4 Installing Protobuf
CentOS
Step 1 Install Protobuf.yum install -y protobuf protobuf-devel
Step 2 Check whether Protobuf is installed successfully.protoc --version
The installation is successful if information similar to the following is displayed:
Step 3 Install Maven.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/bin/protoc
----End
openEuler
Step 1 Download and decompress the source code.wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gztar -zxf protobuf-2.5.0.tar.gz
Step 2 Move the decompressed directory to the /opt/tools/installed/ directory.mv protobuf-2.5.0 /opt/tools/installed/
Step 3 Go to the /opt/tools/installed/ directory.cd /opt/tools/installed
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 92
Step 4 Download the protoc.zip package and decompress it to obtain the protoc.patchfile whose storage path can be specified, for example, to /opt/tools/installed/.wget https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Patch/protoc.zipunzip protoc.zipcp ./protoc/protoc.patch ./protobuf-2.5.0/src/google/protobuf/stubs/
Step 5 Go to the protobuf-2.5.0/src/google/protobuf/stubs/ directory and install thepatch.cd protobuf-2.5.0/src/google/protobuf/stubs/patch -p1 < protoc.patch
Step 6 Go back to the root directory of protobuf-2.5.0, compile the file, and install it inthe default directory.cd /opt/tools/installed/protobuf-2.5.0./autogen.sh && ./configure CFLAGS='-fsigned-char' && make -j8 && make install
Step 7 Deploy Protoc in the local Maven repository.mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/local/bin/protoc
----End
11.4 Performing Porting AnalysisScan the Druid installation package by referring to Kunpeng Porting AdvisorCase Study and obtain the third-party dependencies to be ported. Table 11-1 liststhe third-party dependencies to be ported.
Table 11-1 Third-party dependencies to be ported
Original .jar Package .so File
java-util-0.12.1.jar libsigar.so
leveldbjni-all-1.8.jar libleveldbjni.so
lz4-1.3.0.jar liblz4-java.so
netty-all-4.0.23.Final.jar libnetty-transport-native-epoll.so
netty-all-4.0.52.Final.jar libnetty_transport_native_epoll_x86_64.so
netty-transport-native-epoll-4.0.52.Final-linux-x86_64.jar
libnetty_transport_native_epoll_x86_64.so
snappy-java-1.0.4.1.jar libsnappyjava.so
snappy-java-1.1.1.3.jar libsnappyjava.so
snappy-java-1.1.1.6.jar libsnappyjava.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 93
11.5 Compiling DruidStep 1 Download the source code and decompress it.
wget https://github.com/apache/druid/archive/druid-0.12.1.tar.gztar -xvf druid-0.12.1.tar.gzmv druid-druid-0.12.1 druid-0.12.1
Step 2 Switch to the directory where the Druid source code package is decompressed.cd druid-0.12.1
Step 3 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository source at the beginning of line 1295 in thefile.
<repositories> <repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url> </repository></repositories>
Step 4 Modify the java-util/pom.xml file.vim java-util/pom.xml
Change the download path of sigar-dist-1.6.5.132.zip from line 271 in the file.
<!--<url>https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/</url>--><url>http://porting-dependence.obs.cn-north-4.myhuaweicloud.com/ARTIFACTS/</url>
Step 5 Perform compilation.mvn package -DskipTests
Obtain the .tar.gz package generated in the distribution/target/ directory.
Step 6 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 94
NO TE
The compiled package druid-0.12.1-bin.tar.gz must be scanned by using the KunpengPorting Advisor to ensure that no x86 .so or .jar packages are contained. If the compileddirectory contains x86 .so or .jar packages, the component functions may be affected.
For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
11.6 Troubleshooting
11.6.1 Certificate Error Reported After git clone Is Executed
Symptom
In the compilation process, "fatal:unable to access 'https://*******': Peer's Certificateissuer is not recognized." is displayed.
Procedure
Before running the git clone command, run the following command:
git config --global http.sslVerify false
11.6.2 Failed to Verify the github.com Certificate Downloadedby Using wget
Symptom
During compilation, an error is reported, indicating that the certificate issued bygithub.com cannot be verified. The permission of the issuer cannot be verifiedlocally.
Procedure
To connect to github.com in an insecure manner, specify the --no-check-certificate parameter in the wget command.
11.6.3 An Error Reported When Downloading the hadoop-lzoDependency
Symptom
The following error is reported during compilation:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 95
[ERROR] Failed to execute goal on project druid-thrift-extensions: Could not resolve dependencies for project io.druid.extensions.contribto collect dependencies at com.twitter.elephantbird:elephant-bird-core:jar:4.8 -> com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19: Failedgplcompression:hadoop-lzo:jar:0.4.19: Could not transfer artifact com.hadoop.gplcompression:hadoop-lzo:pom:0.4.19 from/to twitter (http:/http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.pom. Return code is: 502, ReasonPhrase:badgateway.)
Procedure
Modify the local Maven repository file ./repository/com/twitter/elephantbird/elephant-bird/4.8/elephant-bird-4.8.pom.
The code after modification is as follows:
<repositories><repository><id>twitter</id><!--url>http://maven.twttr.com</url--><url>https://nexus.xebialabs.com/nexus/content/groups/public/</url></repository></repositories><pluginRepositories><pluginRepository><id>twitter</id><!--url>http://maven.twttr.com</url--><url>https://nexus.xebialabs.com/nexus/content/groups/public/</url></pluginRepository></pluginRepositories>
11.6.4 An Error Reported When Downloading the libthriftDependency
Symptom
The following error is reported during compilation:
[ERROR] Failed to execute goal com.twitter:scrooge-maven-plugin:4.11.0:testCompile (thrift-test-sources) on project druid-thrift-extensions: Execution thrift-test-sources of goal com.twitter:scrooge-maven-plugin:4.11.0:testCompile failed: Plugin com.twitter:scrooge-maven-plugin:4.11.0 or one of its dependencies could not be resolved: Failed to collect dependencies at com.twitter:scrooge-maven-plugin:jar:4.11.0 -> com.twitter:scrooge-generator_2.10:jar:4.11.0 -> org.apache.thrift:libthrift:jar:0.5.0-1: Failed to read artifact descriptor for org.apache.thrift:libthrift:jar:0.5.0-1: Could not transfer artifact org.apache.thrift:libthrift:pom:0.5.0-1 from/to maven.twttr.com
Procedure
Download maven-repo and replace the thrift directory of the local Mavenrepository with the downloaded org/apache/thrift directory.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
11 Druid 0.12.1 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 96
12 Phoenix 5.0.0 Porting Guide (CentOS7.6 & openEuler 20.03)
12.1 Introduction
12.2 Environment Requirements
12.3 Configuring the Compilation Environment\
12.4 Analyzing Software Porting
12.5 Compiling Phoenix
12.1 IntroductionPhoenix is an intermediate structured query language (SQL) layer built on HBase.It enables you to use the standard JDBC APIs instead of the regular HBase clientAPIs to create tables, insert data, and query your HBase data. Phoenix is written inJava and functions as the JDBC driver embedded in HBase. The Phoenix queryengine converts SQL queries into one or more HBase scans, and orchestrates andexecutes the scans to generate standard JDBC result sets. HBase APIs, collaborativeprocessors, and custom filters are directly used. For simple queries, theperformance is measured in milliseconds. For millions of rows, the performance ismeasured in seconds.
For more information about Phoenix, visit https://phoenix.apache.org/.
12.2 Environment Requirements
Hardware Requirements
Item Remarks
Server TaiShan server
CPU Huawei Kunpeng 920 processor orHuawei Kunpeng 916 processor
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 97
Item Remarks
Drive partition No requirement for drive partitions
Network Accessible to the Internet
Software Requirements
Item Version
OpenJDK 1.8.0_252
Maven 3.5.4
CMake 3.12.4
CentOS
Item Version
CentOS 7.6
OS kernel 4.14.0
GCC 4.8.5
openEuler
Item Version
openEuler 20.03 LTS SP1
OS Kernel 4.19.90
GCC 7.3.0
12.3 Configuring the Compilation Environment\
12.3.1 Installing Basic Libraries
Installing GCC
Step 1 Mount the OS image.mount YOUR_OS.iso /media -o loop
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 98
NO TE
Replace YOUR_OS.iso with the actual iso package name.
Step 2 Back up the repo file and clear the /etc/yum.repos.d/ directory.cp -r /etc/yum.repos.d /etc/yum.repos.d-bakrm /etc/yum.repos.d/*
NO TICE
Ensure that all repo files have been backed up. Enter y on the rm deletion page todelete the files.
Step 3 Modify the /etc/yum.repos.d/Local.repo file.vi /etc/yum.repos.d/Local.repo
Configure the local Yum source.[Local]name=Localbaseurl=file:///media/enabled=1gpgcheck=0
Step 4 Make the Yum source configuration take effect.yum clean allyum makecache
Step 5 Use the Yum source to install GCC-related software.yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64
Step 6 Resolve the -fsigned-char problem (by modifying the GCC).
1. Search for the directory where GCC is located. Generally, the directoryis /usr/bin/gcc.command -v gcc
2. Rename the original GCC file, for example, to gcc-impl.mv /usr/bin/gcc /usr/bin/gcc-impl
3. Create a new GCC file.vi /usr/bin/gcc
Add the following information to the file and save the file:#! /bin/sh/usr/bin/gcc-impl -fsigned-char "$@"
4. Add the execute permission for the GCC file.chmod +x /usr/bin/gcc
5. Check whether the GCC is available.gcc --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 99
Step 7 Resolve the -fsigned-char problem (by modifying G++).
1. Search for the directory where G++ is located. Generally, the directoryis /usr/bin/g++.command -v g++
2. Change the original G++ file name, for example, to g++-impl.mv /usr/bin/g++ /usr/bin/g++-impl
3. Create a new G++ file.vi /usr/bin/g++
Add the following information to the file and save the file:#! /bin/sh/usr/bin/g++-impl -fsigned-char "$@"
4. Add the execute permission for the G++ file.chmod +x /usr/bin/g++
5. Check whether G++ is available.g++ --version
– CentOS: The installation is successful if information similar to thefollowing is displayed:
– openEuler: The installation is successful if information similar to thefollowing is displayed:
----End
Installing Dependencies
Use the Yum source to install dependencies.
yum install -y wget vim openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel
12.3.2 Installing OpenJDKStep 1 Download and decompress the installation package to a directory (for
example, /opt/tools/installed/).wget https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gztar -zxf OpenJDK8U-jdk_aarch64_linux_hotspot_8u252b09.tar.gzmkdir -p /opt/tools/installed/mv jdk8u252-b09 /opt/tools/installed/
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 100
Step 2 Configure Java environment variables.vim /etc/profile
Add the following to the end of the file:export JAVA_HOME=/opt/tools/installed/jdk8u252-b09export PATH=$JAVA_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether OpenJDK is successfully installed.java -version
The installation is successful if information similar to the following is displayed:
----End
12.3.3 Installing MavenStep 1 Download the installation package and install Maven to a directory (for
example, /opt/tools/installed/).wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gztar -zxf apache-maven-3.5.4-bin.tar.gzmv apache-maven-3.5.4 /opt/tools/installed/
Step 2 Modify the Maven environment variables.vim /etc/profile
Add the following at the end of the /etc/profile file:export MAVEN_HOME=/opt/tools/installed/apache-maven-3.5.4export PATH=$MAVEN_HOME/bin:$PATH
Step 3 Make the environment variables take effect.source /etc/profile
Step 4 Check whether Maven is successfully installed.mvn -v
The installation is successful if information similar to the following is displayed:
Step 5 Modify the local repository path and remote repository in the Mavenconfiguration file.
Configuration file path: /opt/tools/installed/apache-maven-3.5.4/conf/settings.xml.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 101
NO TE
The default local repository directory is ~/.m2/. If you want to change the directory to aspecified one, modify the localRepository tag. You do not need to modify this parameterunless otherwise specified.
Add the following content to the <mirrors> tag to configure the remote repository(change the repository to the Maven repository that you have built. If the Mavenrepository does not exist, configure it based on the following example):
<mirror> <id>huaweimaven</id> <name>huawei maven</name> <url>https://mirrors.huaweicloud.com/repository/maven/</url> <mirrorOf>central</mirrorOf></mirror>
If the compilation environment cannot access Internet, add the following proxyconfiguration to settings.xml:
<proxies> <proxy> <id>optional</id> <active>true</active> <protocol>http</protocol> <username>Username</username> <password>Password</password> <host>Proxy server URL</host> <port>Proxy server port</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy></proxies>
----End
12.4 Analyzing Software PortingUse the Kunpeng Porting Advisor to scan the Phoenix installation package byreferring to Kunpeng Porting Advisor Case Study. Table 12-1 lists the third-partydependencies to be ported.
Table 12-1 Third-party dependencies to be ported
Original JAR File SO File
commons-crypto-1.0.0.jar libcommons-crypto.so
netty-all-4.0.23.Final.jar libnetty-transport-native-epoll_x86_64.so
hbase-shaded-netty-2.1.0.jar liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
jline-2.11.1.jar libjansi.so
netty-all-4.1.17.Final.jar libnetty-transport-native-epoll.so
leveldbjni-all-1.8.jar libleveldbjni.so
snappy-java-1.0.5.jar libsnappyjava.so
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 102
Original JAR File SO File
lz4-java-1.2.0.jar liblz4-java.so
scala-compiler-2.11.8.jar libjansi.so
NO TE
The purpose is to identify the .jar packages to be recompiled so that they can be ported tothe Kunpeng platform.
12.5 Compiling PhoenixStep 1 Download the Phoenix-5.0.0-HBase-2.0 source code package and decompress it.
wget https://github.com/apache/phoenix/archive/v5.0.0-HBase-2.0.tar.gzmv v5.0.0-HBase-2.0.tar.gz phoenix-5.0.0-HBase-2.0.tar.gztar -zxf phoenix-5.0.0-HBase-2.0.tar.gzcd phoenix-5.0.0-HBase-2.0
Step 2 Modify the pom.xml file.vim pom.xml
Add the Kunpeng Maven repository in the first line of the repositories tag.<repository> <id>kunpengmaven</id> <name>kunpeng maven</name> <url>https://mirrors.huaweicloud.com/kunpeng/maven</url></repository>
Step 3 Perform compilation.mvn clean package -DskipTests -Dmaven.javadoc.skip=true
After the compilation is successful, the phoenix-5.0.0-HBase-2.0.tar.gz package isgenerated in the /phoenix-assembly/target/ directory of the source codedirectory.
Step 4 Use the Kunpeng Porting Advisor to scan the package generated after thecompilation and ensure that the package does not contain x86 .so or .jarpackages.
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 103
NO TE
The compiled phoenix-5.0.0-HBase-2.0.tar.gz must be scanned by using the KunpengPorting Advisor to ensure that no x86 .so or .jar packages are contained. If the compileddirectory contains x86 .so or .jar packages, the component functions may be affected.For details about how to use the Kunpeng Porting Advisor, see Kunpeng Porting AdvisorCase Study.
----End
Kunpeng BoostKit for Big DataPorting Guide (Apache)
12 Phoenix 5.0.0 Porting Guide (CentOS 7.6 &openEuler 20.03)
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 104
A Change History
Date Description
2021-07-13 This issue is the seventh official release.Added the adaptation to openEuler 20.03 in the porting guides ofApache components.
2021-01-28 This issue is the sixth official release.Added the 9 Kafka 2.2.0 Porting Guide (CentOS 7.6 &openEuler 20.03).
2020-12-30 This issue is the fifth official release.Changed "Kunpeng Code Scanner" to "Kunpeng Porting Advisor".
2020-10-16 This issue is the fourth official release.Added the 10 Kafka 0.10.1.1 Porting Guide (CentOS 7.6 &openEuler 20.03).
2020-09-23 This issue is the third official release.● Deleted "Compiling Dependency Libraries".● Added the 7 Flink 1.9.1 Porting Guide (CentOS 7.6 &
openEuler 20.03).
Kunpeng BoostKit for Big DataPorting Guide (Apache) A Change History
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 105
Date Description
2020-05-23 This issue is the second official release.● Modified 1.6 Compiling Hadoop in the 1 Hadoop 3.1.1
Porting Guide (CentOS 7.6 & openEuler 20.03) and addednotes.
● Deleted "Troubleshooting" from the Hadoop 3.1.1 PortingGuide (CentOS 7.6).
● Deleted "Troubleshooting" from the 3 Hive 3.0.0 PortingGuide (CentOS 7.6 & openEuler 20.03).
● Deleted "Troubleshooting" from the 8 Kafka 2.1.0 PortingGuide (CentOS 7.6 & openEuler 20.03).
● Deleted "Reference" from the 11 Druid 0.12.1 Porting Guide(CentOS 7.6 & openEuler 20.03) and added the relatedinformation to 11.1 Introduction.
● Deleted "Reference" from the 12 Phoenix 5.0.0 Porting Guide(CentOS 7.6 & openEuler 20.03) and added the relatedinformation to 12.1 Introduction.
2020-03-20 This issue is the first official release.
Kunpeng BoostKit for Big DataPorting Guide (Apache) A Change History
Issue 07 (2021-07-13) Copyright © Huawei Technologies Co., Ltd. 106