一、概述
Livy是一個提供Rest接口和Spark集群交互的服務。它可以提交SPARK Job或者Spark一段代碼,同步或者異步的返回結果;也提供Sparkcontext的管理,通過Restful接口或RPC客戶端庫。Livy也簡化了與Spark與應用服務的交互,這允許通過web/mobile與Spark的使用交互。
官網：https://livy.incubator.Apache.org/
GitHub地址：https://github.com/apache/incubator-livy
關于apache Livy更多介紹也可以參考我這篇文章：Spark開源REST服務——Apache Livy（Spark 客戶端）
二、開始編排部署1）部署包準備
這里也提供上面編譯好的livy部署包，有需要的小伙伴可以自行下載：
鏈接：https://pan.baidu.com/s/1pPCbe0lUJ6ji8rvQYsVw9A?pwd=qn7i 提取碼：qn7i1）構建鏡像
Dockerfile
FROM myharbor.com/bigdata/centos:7.9.2009 RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone RUN export LANG=zh_CN.UTF-8 ### install tools RUN yum install -y vim tar wget curl less te.NET net-tools lsof RUN groupadd --system --gid=9999 admin && useradd --system -m /home/admin --uid=9999 --gid=admin admin RUN mkdir -p /opt/apache ADD apache-livy-0.8.0-incubating-SNAPSHOT-bin.zip /opt/apache/ ENV LIVY_HOME=/opt/apache/apache-livy RUN ln -s /opt/apache/apache-livy-0.8.0-incubating-SNAPSHOT-bin $LIVY_HOME ADD hadoop-3.3.2.tar.gz /opt/apache/ ENV HADOOP_HOME=/opt/apache/hadoop RUN ln -s /opt/apache/hadoop-3.3.2 $HADOOP_HOME ENV HADOOP_CONFIG_DIR=${HADOOP_HOME}/etc/hadoop ADD spark-3.3.0-bin-hadoop3.tar.gz /opt/apache/ ENV SPARK_HOME=/opt/apache/spark RUN ln -s /opt/apache/spark-3.3.0-bin-hadoop3 $SPARK_HOME ENV PATH=${LIVY_HOME}/bin:${HADOOP_HOME}/bin:${SPARK_HOME}/bin:$PATH RUN chown -R admin:admin /opt/apache WORKDIR $LIVY_HOME ENTRYPOINT ${LIVY_HOME}/bin/livy-server start;tail -f ${LIVY_HOME}/logs/livy-root-server.out
【注意】hadoop包里的core-site.xml，hdfs-site.xml，yarn-site.xml
開始構建鏡像
docker build -t myharbor.com/bigdata/livy:0.8.0 . --no-cache ### 參數解釋 # -t：指定鏡像名稱 # . ：當前目錄Dockerfile # -f：指定Dockerfile路徑 # --no-cache：不緩存 # 推送到harbor docker push myharbor.com/bigdata/livy:0.8.02）創建livy chart模板helm create livy3）修改yaml編排
- livy/values.yaml
replicaCount: 1 image: repository: myharbor.com/bigdata/livy pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart AppVersion. tag: "0.8.0" securityContext: runAsUser: 9999 runAsGroup: 9999 privileged: true service: type: NodePort port: 8998 nodePort: 31998
- livy/templates/configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: {{ include "livy.fullname" . }} labels: {{- include "livy.labels" . | nindent 4 }} data: livy.conf: |- livy.spark.master = yarn livy.spark.deploy-mode = client livy.environment = production livy.impersonation.enabled = true livy.server.csrf_protection.enabled = false livy.server.port = {{ .Values.service.port }} livy.server.session.timeout = 3600000 livy.server.recovery.mode = recovery livy.server.recovery.state-store = filesystem livy.server.recovery.state-store.url = /tmp/livy livy.repl.enable-hive-context = true livy-env.sh: |- export JAVA_HOME=/opt/apache/jdk1.8.0_212 export HADOOP_HOME=/opt/apache/hadoop export HADOOP_CONF_DIR=/opt/apache/hadoop/etc/hadoop export SPARK_HOME=/opt/apache/spark export SPARK_CONF_DIR=/opt/apache/spark/conf export LIVY_LOG_DIR=/opt/apache/livy/logs export LIVY_PID_DIR=/opt/apache/livy/pid-dir export LIVY_SERVER_JAVA_OPTS="-Xmx512m" spark-blacklist.conf: |- spark.master spark.submit.deployMode # Disallow overriding the location of Spark cached jars. spark.yarn.jar spark.yarn.jars spark.yarn.archive # Don't allow users to override the RSC timeout. livy.rsc.server.idle-timeout
- livy/templates/deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "livy.fullname" . }} labels: {{- include "livy.labels" . | nindent 4 }} spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} selector: matchLabels: {{- include "livy.selectorLabels" . | nindent 6 }} template: metadata: {{- with .Values.podAnnotations }} annotations: {{- toYaml . | nindent 8 }} {{- end }} labels: {{- include "livy.selectorLabels" . | nindent 8 }} spec: {{- with .Values.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} serviceAccountName: {{ include "livy.serviceAccountName" . }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} containers: - name: {{ .Chart.Name }} securityContext: {{- toYaml .Values.securityContext | nindent 12 }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: - name: http containerPort: 8998 protocol: TCP livenessProbe: httpGet: path: / port: http readinessProbe: httpGet: path: / port: http resources: {{- toYaml .Values.resources | nindent 12 }} {{- with .Values.securityContext }} securityContext: runAsUser: {{ .runAsUser }} runAsGroup: {{ .runAsGroup }} privileged: {{ .privileged }} {{- end }} volumeMounts: - name: {{ .Release.Name }}-livy-conf mountPath: /opt/apache/livy/conf/livy.conf subPath: livy.conf - name: {{ .Release.Name }}-livy-env mountPath: /opt/apache/livy/conf/livy-env.sh subPath: livy-env.sh - name: {{ .Release.Name }}-spark-blacklist-conf mountPath: /opt/apache/livy/conf/spark-blacklist.conf subPath: spark-blacklist.conf {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} volumes: - name: {{ .Release.Name }}-livy-conf configMap: name: {{ include "livy.fullname" . }} - name: {{ .Release.Name }}-livy-env configMap: name: {{ include "livy.fullname" . }} - name: {{ .Release.Name }}-spark-blacklist-conf configMap: name: {{ include "livy.fullname" . }}4）開始部署helm install livy ./livy -n livy --create-namespace
NOTES
NOTES: 1. Get the application URL by running these commands: export NODE_PORT=$(kubectl get --namespace livy -o jsonpath="{.spec.ports[0].nodePort}" services livy) export NODE_IP=$(kubectl get nodes --namespace livy -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT
kubectl get pods,svc -n livy -owide
web地址：http://192.168.182.110:31998/ui
5）測試驗證curl -s -XPOST -d '{"file":"hdfs://myhdfs/tmp/spark-examples_2.12-3.3.0.jar","className":"org.apache.spark.examples.SparkPi","name":"SparkPi-test"}' -H "Content-Type: application/json" http://local-168-182-110:31998/batches|Python/ target=_blank class=infotextkey>Python -m json.tool
6）卸載helm uninstall livy -n livy
git地址：https://gitee.com/hadoop-bigdata/livy-on-k8s
Apache Livy on k8s 編排部署講解就先到這里了，有疑問的小伙伴歡迎給我留言，后續會持續更新【云原生+大數據】相關的文章，請小伙伴耐心等待~
