Skip to content

README.md

Latest commit

 

History

History
170 lines (136 loc) · 6.17 KB

README.md

File metadata and controls

170 lines (136 loc) · 6.17 KB

Example Applications

CSI Ephemeral Volume Example

# replace <bucket-name> with your pre-provisioned GCS bucket name
GCS_BUCKET_NAME=your-bucket-name
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/ephemeral/deployment.yaml
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/ephemeral/deployment-non-root.yaml
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/ephemeral/deployment-two-vols.yaml

# install a Deployment using CSI Ephemeral Inline volume
kubectl apply -f ./examples/ephemeral/deployment.yaml
kubectl apply -f ./examples/ephemeral/deployment-non-root.yaml
kubectl apply -f ./examples/ephemeral/deployment-two-vols.yaml

# clean up
kubectl delete -f ./examples/ephemeral/deployment.yaml
kubectl delete -f ./examples/ephemeral/deployment-non-root.yaml
kubectl delete -f ./examples/ephemeral/deployment-two-vols.yaml

Static Provisioning Example

# replace <bucket-name> with your pre-provisioned GCS bucket name
GCS_BUCKET_NAME=your-bucket-name
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/static/pv-pvc-deployment.yaml
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/static/pv-pvc-deploymen-non-root.yaml

# install PV/PVC and a Deployment
kubectl apply -f ./examples/static/pv-pvc-deployment.yaml
kubectl apply -f ./examples/static/pv-pvc-deploymen-non-root.yaml

# clean up
# the PV deletion will not delete your GCS bucket
kubectl delete -f ./examples/static/pv-pvc-deployment.yaml
kubectl delete -f ./examples/static/pv-pvc-deploymen-non-root.yaml

Batch Job Example

# replace <bucket-name> with your pre-provisioned GCS bucket name
GCS_BUCKET_NAME=your-bucket-name
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/batch-job/job.yaml

# install a Job using CSI Ephemeral Inline volume
kubectl apply -f ./examples/batch-job/job.yaml

# clean up
kubectl delete -f ./examples/batch-job/job.yaml

Jupyter Notebook Example

# replace <bucket-name> with your pre-provisioned GCS bucket name
GCS_BUCKET_NAME=your-bucket-name
sed -i "s/<bucket-name>/$GCS_BUCKET_NAME/g" ./examples/jupyter/jupyter-notebook-server.yaml

# install a Jupyter Notebook server using CSI Ephemeral Inline volume
kubectl apply -f ./examples/jupyter/jupyter-notebook-server.yaml

# access the Jupyter Notebook via http://localhost:8888
kubectl port-forward jupyter-notebook-server 8888:8888

# clean up
kubectl delete -f ./examples/jupyter/jupyter-notebook-server.yaml

Multi-NIC Example

If running on a VM instance (k8s node) with multiple NICs, the multiNICIndex volume attribute can be used to attach a mount point to a particular NIC. If the instance NICs are aligned to NUMA nodes, then the index corresponds to the NUMA node, otherwise the index is that of gve and virtio_net devices given by ip link.

To use multiple NICs, the pod must run in host network. If the gke-gcsfuse/enable-numa-pinning=true annotation is used, then the gcs fuse daemon will be pinned to the corresponding NUMA node.

As an example, the following command on GKE creates an n2 instance with 2 NUMA nodes and two NICs. There is no actual NUMA alignment for the NICs, but this is an inexpensive way to test the behavior of machines like TPU v7x which do have NUMA aligned NICs.

gcloud container node-pools create ${NODE_POOL} \
  --machine-type n2-standard-64 --num-nodes 1 \
  --enable-gvnic \
  --additional-node-network=network=additional,subnetwork=alt

It assumes you have created the network and subnetwork with commands like the following (the exact ranges used will depend on your specifics). The router commands are necessary to expose the network externally, so it can reach GCS endpoints. The parameters used are not critical and depend on your setup as well. The network and subnet names are only used for node pool creation and are not critcal for your pod configuration.

gcloud compute networks create additional
gcloud compute networks subnests create alt \
  --range=10.144.16.0/20 \
  --network=additional \
  --secondary-range=alt=10.144.32.0/20
  --region=${CLUSTER_LOCATION}
gcloud compute routers create additional \
  --network=additional \
  --region=${CLUSTER_LOCATION}
gcloud compute routers nats create additional-nat \
  --router=additional \
  --region=${CLUSTER_LOCATION} \
  --auto-allocate-nat-external-ips \
  --nat-all-subnet-ip-ranges

Then deploy a pod which uses the second NIC from a node in your new node pool. Ensure that you have set up permissions to your GCS bucket as described elsewhere.

cat ./multi-nic/index.yaml | \
  sed s/BUCKET/${BUCKET}/ | \
  sed s/PROJECT/${PROJECT}/ | \
  sed s/LOCATION/${CLUSTER_LOCATION}/ | \
  sed s/CLUSTER/${CLUSTER}/ | \
  sed s/NODE-POOL/${NODE_POOL}/ | \
  kubectl apply -f -

Multi-NIC on Cloud TPU

https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus#deploy-multi-container shows how to run multi-container jobs on TPU platforms such a Ironwood (TPU7x) which have multiple NUMA nodes. The multi-nic/cloudtpu.yaml example shows how to use multi-NIC GCS Fuse in this setting. The cloud tpu driver on GKE injects a WORKLOAD_NIC_PREFERRED_NUMA environment variable into containers which gives the NUMA node that the container has been assigned to. Each container mounts the GCS fuse volumes indexed to all NUMA nodes. The WORKLOAD_NIC_PREFERRED_NUMA environment variable is then used to select the proper mount point for processes running in that container.

To see the complete example, run the following on a cluster that has been set up both for multicontainer and multi-NIC as described above.

cat ./multi-nic/cloudtpu.yaml | \
  sed s/BUCKET/${BUCKET}/ | \
  kubectl apply -f -