README.md 9.99 KB
Newer Older
1
2
3
4
5
6
7
8
9
# Kubernetes Cluster Federation (KubeFed)

## Documentation

Below some useful links for the main tools of a Kubernetes federation:

- https://github.com/kubernetes-sigs/kubefed
- https://github.com/kubernetes-sigs/external-dns
- https://github.com/kubernetes-sigs/kubefed/blob/master/docs/ingressdns-with-externaldns.md
10
11
12
13
14
15
16
17


## Environment configuration

Before starting it is necessary create several Kubernetes clusters to make the federation. One cluster will be the Host (Host Cluster), in which will be install the KubeFed (Control Plane), and the remaining clusters will be the Members (Member Cluster).
To create the clusters on Openstack, it is possible to use the follow Juju bundle: https://jaas.ai/u/csd-garr/kubernetes-openstack-integrator/bundle.

Once all the clusters for federation have been created, it is useful to collect all cluster access configurations (.kube/config) in a single config file. In this way you can easily switch from one cluster to another.
Marco Lorini's avatar
Marco Lorini committed
18
Below is an example:
19

20
```bash
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://<ip>:443
  name: k8s-cluster-host
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://<ip>:443
  name: k8s-cluster-member1
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://<ip>:443
  name: k8s-cluster-member2
contexts:
- context:
    cluster: k8s-cluster-host
    user: admin-host
  name: k8s-context-host
- context:
    cluster: k8s-cluster-member1
    user: admin-member1
  name: k8s-context-member1
- context:
    cluster: k8s-cluster-member2
    user: admin-member2
  name: k8s-context-member2
current-context: k8s-context-host
kind: Config
preferences: {}
users:
- name: admin-host
  user:
    password: 8Rvl5qW...
    username: admin
- name: admin-member1
  user:
    password: 0JefxZq...
    username: admin
- name: admin-member2
  user:
    password: hJLiVLI...
    username: admin

```

In this way you have access to all the clusters by the context switch, below are the commands for the context switch:

list contexts:
70
```bash
71
72
73
74
$ kubectl config get-contexts
```

switch context:
75
```bash
76
77
$ kubectl config use-context <context-name>
```
78
79
80
81
82

## Installation

### KubeFed client (kubefedctl)

Marco Lorini's avatar
Marco Lorini committed
83
To install the KubeFed client run the following commands in your local machine:
84

85
```bash
86
87
88
89
90
91
92
93
94
95
96
97
98
$ VERSION=0.2.0-alpha.1
$ OS=linux
$ ARCH=amd64
$ curl -LO https://github.com/kubernetes-sigs/kubefed/releases/download/v${VERSION}/kubefedctl-${VERSION}-${OS}-${ARCH}.tgz
$ tar -zxvf kubefedctl-*.tgz
$ chmod u+x kubefedctl
$ sudo mv kubefedctl /usr/local/bin/ # make sure the location is in the PATH
```

**N.B.**: the versions we tested are: 0.1.0-rc6 and 0.2.0-alpha.1, you can try the latest version.

### KubeFed server (Control Plane)

Marco Lorini's avatar
Marco Lorini committed
99
To install the Controll Plane, switch to the host context and configuring the RBAC role for Tiller (in this case we use Helm v2):
100

101
```bash
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
$ cat << EOF | kubectl apply -f -
	apiVersion: v1
	kind: ServiceAccount
	metadata:
	  name: tiller
	  namespace: kube-system
	---
	apiVersion: rbac.authorization.k8s.io/v1
	kind: ClusterRoleBinding
	metadata:
	  name: tiller
	roleRef:
	  apiGroup: rbac.authorization.k8s.io
	  kind: ClusterRole
	  name: cluster-admin
	subjects:
	  - kind: ServiceAccount
	    name: tiller
	    namespace: kube-system
	EOF
```

helm init command:
125
126

```bash
127
128
129
130
131
$ helm init --service-account tiller
```

Now it is possible install the Control Plane on the Host Cluster:

132
```bash
133
134
135
136
137
138
139
$ helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
$ helm install kubefed-charts/kubefed --name kubefed --version=0.2.0-alpha.1 --namespace kube-federation-system
```

**N.B.**: use the same version for client and server.

**N.B.**: in this way the Control Plane can find Federated resources in each namespace, to limit this action in a specific namespace add the parameter `--set global.scope=Namespaced`.
140
141
142
143
144
145


## Create the federation

To create a Kubernetes Federation it is necessary federate Member Clusters. Use the KubeFed `join` command:

146
```bash
147
148
149
150
151
152
$ kubefedctl join fed-cluster-member1 --cluster-context k8s-context-member1 --host-cluster-context k8s-context-host --v=2
$ kubefedctl join fed-cluster-member2 --cluster-context k8s-context-member2 --host-cluster-context k8s-context-host --v=2
```

Remove cluster to the federation, `unjoin` command:

153
```bash
154
155
156
157
$ kubefedctl unjoin fed-cluster-member1 --cluster-context k8s-context-member1 --host-cluster-context k8s-context-host --v=2
$ kubefedctl unjoin fed-cluster-member2 --cluster-context k8s-context-member2 --host-cluster-context k8s-context-host --v=2
```

Marco Lorini's avatar
Marco Lorini committed
158
Command to show federated clusters:
159

160
```bash
161
162
$ kubectl -n kube-federation-system get kubefedclusters
```
163

164

Marco Lorini's avatar
Marco Lorini committed
165
## Enable API
166

Marco Lorini's avatar
Marco Lorini committed
167
To enable a type of resource to be federated it is necessary to use the enable command of kubefedctl:
168

169
```bash
170
171
172
173
174
$ kubefedctl enable <api-resources-name.api-group> --kubefed-namespace kube-federation-system
```

To know the list of resourses:

175
```bash
176
177
178
179
180
$ kubectl api-resources -o wide
```

Some examples:

181
```bash
182
183
184
185
186
187
188
$ kubefedctl enable deployments.apps --kubefed-namespace kube-federation-system
$ kubefedctl enable services --kubefed-namespace kube-federation-system
$ kubefedctl enable ingresses.extensions --kubefed-namespace kube-federation-system
```

Command to disable federated API:

189
```bash
190
191
192
193
$ kubefedctl disable services --kubefed-namespace kube-federation-system
```

**N.B.**: Do not federate the “kind: pod” resource. In this case, federating a whole namespace will federate pods and deployments at the same time. Deployments will create replicasets which in turn create pods. It will result a duplication of the pods resources.
194
195


196
## Federate an application
197
198
199

Below the procedure to deploy and federate an application and enable ExternalDNS.

200
201
202
**N.B.**: all commands must be run on Host Cluster. 

### Create namespace
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248

The first step is create a namespace in the host cluster:

command line:

```bash
$ kubectl create namespace <namespace-name>
```

or create it by yaml file (resource/namespace.yaml):

```yaml
# namespace.yaml file
apiVersion: v1
kind: Namespace
metadata:
  name: fed-namespace
```

```bash
$ kubectl create -f resource/namespace.yaml
```

### Federate namespace

Once the namespace is created in the Host Cluster, it can be federate on the Member Cluster:

```yaml
# federated_namespace.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: fed-namespace
  namespace: fed-namespace
spec:
  placement:
    clusters:
    - name: member-cluster-1
    - name: member-cluster-2
```

```bash
$ kubectl --context=<host-cluster-context> create -f resource/federated_namespace.yaml
```

**N.B.**: the option `--context` is not necessary but make sure that the right context is selected.
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343

### Create RBAC for ExternalDNS

Now it is possible to deploy ExternalDNS in the federated namespace. Create the RBAC for the ExternalDNS:

```yaml
# rbac_externaldns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: fed-namespace
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
  namespace: fed-namespace
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
  namespace: fed-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: fed-namespace
```

```bash
$ kubectl --context=<host-cluster-context> create -f resource/rbac_externaldns.yaml
```

### Deploy ExternalDNS for PowerDNS

Create the ExternalDNS deployment and configure it for PowerDNS (in the our case):

```yaml
# externaldns.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: fed-namespace
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      # Only use if you're also using RBAC
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:latest
        args:
        - --source=crd # or service or ingress
        - --crd-source-apiversion=multiclusterdns.kubefed.io/v1alpha1
        - --crd-source-kind=DNSEndpoint
        - --provider=pdns
        - --pdns-server=http://<ip>:<port>
        - --pdns-api-key=<api-key>
        - --registry=txt
        - --txt-prefix=cname
        - --domain-filter=<domain> # will make ExternalDNS see only the zones matching provided domain; omit to process all available zones in PowerDNS 
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization

```

```bash
$ kubectl --context=<host-cluster-context> create -f resource/externaldns.yaml
```
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388

### Federate deployment

Now create the federated deployment (for this example we used a simple "Hello World" image):

```yaml
# federated_deployment.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: fed-helloworld
  namespace: fed-namespace
spec:
  template:
    metadata:
      name: helloworld
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: helloworld
      template:
        metadata:
          labels:
            app: helloworld
        spec:
          containers:
          - image: docker.io/csdgarr/hello-world:v1
            name: helloworld
            imagePullPolicy: IfNotPresent
  placement:
    clusters:
    - name: member-cluster-1
    - name: member-cluster-2
  overrides:
    - clusterName: member-cluster-2
      clusterOverrides:
      - path: "/spec/replicas"
        value: 3

```

```bash
$ kubectl --context=<host-cluster-context> create -f resource/federated_deployment.yaml
```