AWS EKS for Fargate, with eksctl

Guestbook

AWS EKS, with eksctl

Second try with AWS EKS on Fargate. This time with eksctl.

Create EKS cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
𝜆 eksctl create cluster --name sandpit --version 1.14 --region us-east-2 --fargate
[ℹ]  eksctl version 0.11.1
[ℹ]  using region us-east-2
[ℹ]  setting availability zones to [us-east-2b us-east-2a us-east-2c]
[ℹ]  subnets for us-east-2b - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for us-east-2a - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for us-east-2c - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "sandpit" in "us-east-2" region with Fargate profile
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-2 --cluster=sandpit'
[ℹ]  CloudWatch logging will not be enabled for cluster "sandpit" in "us-east-2"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=us-east-2 --cluster=sandpit'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "sandpit" in "us-east-2"
[ℹ]  1 task: { create cluster control plane "sandpit" }
[ℹ]  building cluster stack "eksctl-sandpit-cluster"
[ℹ]  deploying stack "eksctl-sandpit-cluster"
[✔]  all EKS cluster resources for "sandpit" have been created
[✔]  saved kubeconfig as "/Users/terrence/.kube/config"
[ℹ]  creating Fargate profile "fp-default" on EKS cluster "sandpit"
[ℹ]  created Fargate profile "fp-default" on EKS cluster "sandpit"
[ℹ]  "coredns" is now schedulable onto Fargate
[ℹ]  "coredns" is now scheduled onto Fargate
[ℹ]  "coredns" pods are now scheduled onto Fargate
[ℹ]  kubectl command should work with "/Users/terrence/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "sandpit" in "us-east-2" region is ready

AWS EKS on Fargate, with eksctl - Cluster

Create and add EKS mansged node group:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
𝜆 eksctl create nodegroup --cluster sandpit --name workers --node-type t3a.medium --ssh-access --ssh-public-key aws-us-key --managed
[ℹ]  eksctl version 0.11.1
[ℹ]  using region us-east-2
[ℹ]  will use version 1.14 for new nodegroup(s) based on control plane version
[ℹ]  using EC2 key pair %!!(MISSING)q(*string=<nil>)
[ℹ]  1 nodegroup (workers) was included (based on the include/exclude rules)
[ℹ]  will create a CloudFormation stack for each of 1 managed nodegroups in cluster "sandpit"
[ℹ]  1 task: { 1 task: { create managed nodegroup "workers" } }
[ℹ]  building managed nodegroup stack "eksctl-sandpit-nodegroup-workers"
[ℹ]  deploying stack "eksctl-sandpit-nodegroup-workers"
[✔]  created 0 nodegroup(s) in cluster "sandpit"
[ℹ]  nodegroup "workers" has 2 node(s)
[ℹ]  node "ip-192-168-47-175.us-east-2.compute.internal" is ready
[ℹ]  node "ip-192-168-87-98.us-east-2.compute.internal" is ready
[ℹ]  waiting for at least 2 node(s) to become ready in "workers"
[ℹ]  nodegroup "workers" has 2 node(s)
[ℹ]  node "ip-192-168-47-175.us-east-2.compute.internal" is ready
[ℹ]  node "ip-192-168-87-98.us-east-2.compute.internal" is ready
[✔]  created 1 managed nodegroup(s) in cluster "sandpit"
[ℹ]  checking security group configuration for all nodegroups
[ℹ]  all nodegroups have up-to-date configuration

Kubernetes Dashboard

Install Kubernetes Dashboard in Kubernetes cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
𝜆 kubectl get services  --all-namespaces
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default                kubernetes                  ClusterIP   10.100.0.1       <none>        443/TCP         70m
kube-system            kube-dns                    ClusterIP   10.100.0.10      <none>        53/UDP,53/TCP   70m
kube-system            metrics-server              ClusterIP   10.100.142.106   <none>        443/TCP         14m
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.100.91.78     <none>        8000/TCP        11m
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.100.75.0      <none>        443/TCP         11m

𝜆 kubectl get pods  --all-namespaces
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
kube-system            aws-node-cnzrv                               1/1     Running   0          40m
kube-system            aws-node-m9tjp                               1/1     Running   0          40m
kube-system            coredns-7f5cccffc-h44mz                      1/1     Running   0          65m
kube-system            coredns-7f5cccffc-hmx7g                      1/1     Running   0          65m
kube-system            kube-proxy-7kn62                             1/1     Running   0          40m
kube-system            kube-proxy-g57ph                             1/1     Running   0          40m
kube-system            metrics-server-7fcf9cc98b-ftl4k              1/1     Running   0          14m
kubernetes-dashboard   dashboard-metrics-scraper-677768c755-mxlmc   1/1     Running   0          11m
kubernetes-dashboard   kubernetes-dashboard-995fd6fb4-xqcj5         1/1     Running   0          11m

Connect Kubernetes Dashboard via proxy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
𝜆 cat .kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/terrence/.minikube/ca.crt
    server: https://192.168.99.100:8443
  name: minikube
- cluster:
    certificate-authority-data: LS0tLS1CRUd ... tLS0tLQo=
    server: https://0559DE89F43B8766B56C7FD066C6C50F.yl4.us-east-2.eks.amazonaws.com
  name: sandpit.us-east-2.eksctl.io
contexts:
- context:
    cluster: sandpit.us-east-2.eksctl.io
    user: ADMMiaoT@sandpit.us-east-2.eksctl.io
  name: ADMMiaoT@sandpit.us-east-2.eksctl.io
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: ADMMiaoT@sandpit.us-east-2.eksctl.io
kind: Config
preferences: {}
users:
- name: ADMMiaoT@sandpit.us-east-2.eksctl.io
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - sandpit
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: paradise-dev
- name: minikube
  user:
    client-certificate: /Users/terrence/.minikube/client.crt
    client-key: /Users/terrence/.minikube/client.key

𝜆 kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}'
eks-admin-token-s2gf5

𝜆 kubectl -n kube-system describe secret eks-admin-token-s2gf5
Name:         eks-admin-token-s2gf5
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: eks-admin
              kubernetes.io/service-account.uid: fa3cf514-18bc-11ea-bbdd-0a4cd5e8dc70

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIs ... hpY8upQlA2q40g

𝜆 kubectl proxy

Visit URL http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#!/login

Choose Token, paste the token output from the previous command into the Token field, and choose SIGN IN.

AWS EKS on Fargate, with eksctl - Nodes

With AWS managed nodes, on Node EC2 Instance:

AWS EKS on Fargate, with eksctl - EC2 Instance

First Docker application

Deploy first Docker application react-typescript, from Docker Hub https://hub.docker.com/r/jtech/react-typescript, into Kubernetes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
𝜆 kubectl run react-typescript --image=jtech/react-typescript --port 3000
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/react-typescript created

𝜆 kubectl describe deployments
Name:                   react-typescript
Namespace:              default
CreationTimestamp:      Mon, 09 Dec 2019 14:56:09 +1100
Labels:                 run=react-typescript
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               run=react-typescript
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  run=react-typescript
  Containers:
   react-typescript:
    Image:        jtech/react-typescript
    Port:         3000/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   react-typescript-867c948446 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  71s   deployment-controller  Scaled up replica set react-typescript-867c948446 to 1

𝜆 kubectl describe pods react-typescript
Name:               react-typescript-867c948446-qtvrp
Namespace:          default
Priority:           2000001000
PriorityClassName:  system-node-critical
Node:               fargate-ip-192-168-183-250.us-east-2.compute.internal/192.168.183.250
Start Time:         Mon, 09 Dec 2019 14:56:59 +1100
Labels:             eks.amazonaws.com/fargate-profile=fp-default
                    pod-template-hash=867c948446
                    run=react-typescript
Annotations:        kubernetes.io/psp: eks.privileged
Status:             Running
IP:                 192.168.183.250
Controlled By:      ReplicaSet/react-typescript-867c948446
Containers:
  react-typescript:
    Container ID:   containerd://2ea5f1ea4fb731eb844f0e267581e9e188d29ab7a639b7b8ca50c83cfb12b4c3
    Image:          jtech/react-typescript
    Image ID:       docker.io/jtech/react-typescript@sha256:0951fe4d9a24390123c7aa23612c8cdf1d8191a9d8e7d3cbc8bb4d8d763e0ce5
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 09 Dec 2019 14:57:28 +1100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-knpqq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-knpqq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-knpqq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason   Age   From                                                            Message
  ----    ------   ----  ----                                                            -------
  Normal  Pulling  76s   kubelet, fargate-ip-192-168-183-250.us-east-2.compute.internal  Pulling image "jtech/react-typescript"
  Normal  Pulled   49s   kubelet, fargate-ip-192-168-183-250.us-east-2.compute.internal  Successfully pulled image "jtech/react-typescript"
  Normal  Created  49s   kubelet, fargate-ip-192-168-183-250.us-east-2.compute.internal  Created container react-typescript
  Normal  Started  49s   kubelet, fargate-ip-192-168-183-250.us-east-2.compute.internal  Started container react-typescript

Expose service:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
𝜆 kubectl expose deployment react-typescript --type="NodePort"
service/react-typescript exposed

𝜆 kubectl describe services react-typescript
Name:                     react-typescript
Namespace:                default
Labels:                   run=react-typescript
Annotations:              <none>
Selector:                 run=react-typescript
Type:                     NodePort
IP:                       10.100.54.37
Port:                     <unset>  3000/TCP
TargetPort:               3000/TCP
NodePort:                 <unset>  31799/TCP
Endpoints:                192.168.183.250:3000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

𝜆 kubectl get services  --all-namespaces
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
default                kubernetes                  ClusterIP   10.100.0.1       <none>        443/TCP          46h
default                react-typescript            NodePort    10.100.54.37     <none>        3000:31799/TCP   4m55s
kube-system            kube-dns                    ClusterIP   10.100.0.10      <none>        53/UDP,53/TCP    46h
kube-system            metrics-server              ClusterIP   10.100.142.106   <none>        443/TCP          45h
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.100.91.78     <none>        8000/TCP         45h
kubernetes-dashboard   kubernetes-dashboard        ClusterIP   10.100.75.0      <none>        443/TCP          45h

Run kubectl proxy and connect to react-typscript application on URL: http://localhost:8001/api/v1/namespaces/default/services/http:react-typescript:3000/proxy/

AWS EKS on Fargate, with eksctl - React Typescript

References

AWS EKS for Fargate

Guestbook

AWS EKS

After AWS EKS for Fargate annouced in Re:Invent 2019 - Amazon EKS on AWS Fargate Now Generally Available, I have a quick spin.

General configuration:

AWS EKS on Fargate - Configuration

AWS EKS on Fargate - Configuration

Fargate profile configuration:

AWS EKS on Fargate - Profile

Fargate roles:

AWS EKS on Fargate - Roles

CustomEKSRole role has AmazonEKSClusterPolicy and AmazonEKSServicePolicy.

CustomEKSFargatePodExecutionRole role has AmazonEKSFargatePodExecutionRolePolicy, and Trust relationships:

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "eks-fargate-pods.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

CustomEKSWorkerNodeRole role has AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly, and Trust relationships:

1
2
3
4
5
6
7
8
9
10
11
12
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Namespace for Fargate profile Pod Selectors is default.

Subnets for Fargate, including private subnets (subnet without Internet Gateway):

AWS EKS on Fargate - Subnets

References

Why React?

Guestbook

A new pedigree of Frontend Development.

The following are the preferred technical choices for any new projects that do not have a “hammer whats already there“ constraint that have been chosen for consistency.

Programming Language: TypeScript 

Alternatives to avoid:

  • Raw JS / Babel: Types can prevent a significant amount of bugs and provide a great means of documenting code that can be validated by a compiler. Therefore we will not dabble in untyped programming languages for Frontend development.
  • Other TypeSafe alternatives:
    • Dart / Elm / Blazor - typed compile to JavaScript languages: Browsers operate on JavaScript using a language that is too far removed from JS means people will need to learn two language and fix bugs at two levels.
    • Flow - A facebook alternative to TypeScript. Flow was a decent experiment into JavaScript type safety by the Facebook team, but news from inside Facebook (chats at conferences) is that it is not actively getting maintained. Therefore it should be avoided.

UI Library: React

Alternatives to avoid:

  • Angular: Angular goes through significant breaking changes as can be witnessed by the changelog. Also Angular is a full framework (as opposed to a library), but people rarely use it as a framework as there will be inevitable external dependencies for alternative (even better) community tools e.g. rxjs for state, axios for api etc.
  • Vue: Vue is the third most popular choice. However it not backed by a major enterprise (Facebook for React, Google for Angular)
  • Svelte: Same reasons as Vue.

State Management: MobX

Alternatives to avoid:

  • Redux: Redux is definitely the more popular choice in the React community. However most people will use it with some additional middleware library e.g. redux-thunk or redux-saga leading to multiple ways of doing Frontend. Also, Redux is significantly more verbose (because of always new object creation) and difficult to optimise (because of no internal knowledge of what changed).

Styles: CSS in JS

Prefer Emotion or TypeStyle over others.

Alternatives to avoid:

  • CSS preprocessor (e.g. sass / css-modules). For CSS in JS there is an excellent presentation that you should checkout. Key reasons are clear code co-location and management practices for CSS, the same way React does it for Html (with JSX).

End to End Testing : Cypress

Cypress for E2E testing.

Alternatives to avoid:

  • Selenium based solutions (e.g. Protractor / Nightwatch). Selenium operates over a wire transfer protocol that makes it impossible to provide many of the debugging features provided by Cypress. Cypress operates in process with the tests making automatic retries with greater flake resistance and a significantly improved debugging experience.

HTTP Requests

Prefer axios due to is built in TypeScript definitions, great browser support and community guidance.

Alternatives to avoid:

  • fetch: Needs polyfill and needs explicit calls for json de-serialization.

A PC person's guide to Mac

Guestbook

Prelude

This is a re-posted article I published on May 19th, 2012, long long time ago on the currently already died Google Plus.

Bought a lowest entry level Macbook Pro 13” i5 refurbished during the week. This is the first ever Apple product I bought. I’m not a fan boy of sleeky, popular iPhone. I keep poo-pooing iPad. But because of the admiration of MBP’s impeccable design and high quality, moreover as a tribute to Steve Jobs, I join the cohort of yuppies, with a silver MPB on my laptop.

A PC person's guide to Mac

Macbook Pro package comes without Mac OS X Lion installation DVD. You have to re-install OS from internet.

A PC person's guide to Mac

A PC person's guide to Mac

Impeccable design and high quality product, even the refurbished one.

A PC person's guide to Mac

A PC person's guide to Mac

However, as a typical PC person living a life of hacker. There should no boundary, no limitation, no rules can’t be broken. In order to maximise all the potentials inside this piece of hardware, and get my investment back, I also ordered 16GB DDR3 RAM, 240GB SATA3 SSD this week from two computer parts shops, at the best price I can find on the market. After some tweak to get rid of the bottleneck in the overall performance, finally this baby is muscled up.

A PC person's guide to Mac

A PC person's guide to Mac

A PC person's guide to Mac

A PC person's guide to Mac

A PC person's guide to Mac

A PC person's guide to Mac

Refurbished MBP, $1189.00 paid on credit card with free shipping. 16GB Patriot DDR3 RAM for Mac, $150 on cash. Corsair Force SATA3 240GB SSD, $295 on cash. The happiness and achievement to get something done after a long time planning, researching, waiting, torture and agony, priceless. There is something money can’t buy. For everything else, there is a Mastercard. And you have to work harder to earn your dole and make this happen.

A PC person's guide to Mac

A PC person's guide to Mac

Totally under 1700 dollars investment will keep me happy for a long time. I reckon it will be good money spent on. If you see the used to be memory hogging applications like M$ Office and Eclipse , suddenly they all jump on your face when you click and open them, the blissful feeling is hardly to describe in a single word, especially you need reward and motivate yourself if you were kept in the shity work. Lifespan? Certainly not. Longevity? Probably not. However, what else you can do if you don’t hack it?

Need to change my PC mindset finally. Keep reminding me “Don’t panic, you are using a Mac.”

Run ssh and scp with AWS Session Manager

Guestbook

AWS Session Manager

New AWS Systems Manager, including Session Manager is another step enhance security on Cloud. Here are step by step how to set up.

NOTE: There is NO need to require to have a Public IP on EC2 instance, and have network inbound rule setup with opened SSH port 22, and VPN connection.

  • You have ec2-user account on AWS EC2 instance. On localhost:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
𝜆 cat .aws/config
[default]
output = json
region = ap-southeast-2

[session]
output = json
region = ap-southeast-2

𝜆 cat .aws/credentials
[default]
aws_access_key_id = ASIANPOWERHOUSEBLAHBLAH
aws_secret_access_key = aGLMountainLionPIEXL0UK0TunalNB61Kt+GuavaVm4tAD

[session]
aws_access_key_id = ASIANPOWERHOUSEBLAHBLAH
aws_secret_access_key = aGLMountainLionPIEXL0UK0TunalNB61Kt+GuavaVm4tAD
aws_session_token = FwoGZXIvYXdzEB8aDDuzeYcE5QDFo0 ... z2h/Px7nUMEWaZOZZw==
aws_expiration=2019-11-22T20:33:18.000Z

𝜆 ssh -i .ssh/aws-key.pem -l ec2-user ec2-3-121-69-96.ap-southeast-2.compute.amazonaws.com
Last login: Fri Nov 22 01:17:33 2019 from 155.144.114.41

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

NOTE: Need ROOT access key pair ASIANPOWERHOUSEBLAHBLAH above setup in session profile to run aws sts command.

NOTE: IAM role for EC2 instance need to have AmazonSSMManagedInstanceCore policy. So create a customised role CustomAmazonSSMManagedInstanceCore in AWS IAM including AmazonSSMManagedInstanceCore policy, and bind this IAM role with EC2 instance, also with security group and key pair.

AWS Session Manager - Managed instances

AWS Session Manager - Agent auto update

  • Add customised RunAs users via “Run Command”

To elevated SSM_pwr_user, a customised user to allow login EC2 instance, with command:

1
2
𝜆 useradd -g wheel SSM_pwr_user
𝜆 echo "SSM_pwr_user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/SSM_pwr_user

AWS System Manager - Run a command

Updated above solution and add a DIFFERENT user ssm-user, if SSM throws error “Unable to start shell: failed to start pty since RunAs user ssm-user does not exist“:

1
2
𝜆 useradd -g wheel ssm-user
𝜆 echo "ssm-user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ssm-agent-users
  • Update SSH config file on localhost to proxy commands through the AWS Session Manager for any EC2 instance id
1
2
3
4
5
𝜆 cat .ssh/config

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
  • Generate session token
1
2
3
4
5
6
7
8
9
10
11
12
𝜆 date
Thu 21 Nov 2019 04:33:03 UTC

𝜆 aws sts get-session-token --duration-seconds 129600 --profile session
{
    "Credentials": {
        "SecretAccessKey": "I7brpY8XWDWYwwyUdp5PLq7cxpskuMSHyBtPjPNE",
        "SessionToken": "FwoGZXIvYXdzENL//////////wEaDKKdWTVmCrwKRiMbOSKCAbkQr ... YiNntciJszsZVRypXz1HTfa3gbcKoNXHon8=",
        "Expiration": "2019-11-22T16:33:12Z",
        "AccessKeyId": "ASIASZPQ3TMDVJVIGM7H"
    }
}

If AWS user ec2-user has MFA enabled, generate session token like this:

1
2
𝜆 aws sts get-session-token --duration-seconds 129600 --profile session \
    --serial-number arn:aws:iam::910218657901234:mfa/root-account-mfa-device --token-code 251556
  • Add session token in AWS credentials file on localhost, and test
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
𝜆 cat .aws/credentials
[default]
aws_access_key_id = ASIANPOWERHOUSEBLAHBLAH
aws_secret_access_key = aGLMountainLionPIEXL0UK0TunalNB61Kt+GuavaVm4tAD

[session]
aws_access_key_id = ASIASZPQ3TMDVJVIGM7H
aws_secret_access_key = I7brpY8XWDWYwwyUdp5PLq7cxpskuMSHyBtPjPNE
aws_session_token = FwoGZXIvYXdzENL//////////wEaDKKdWTVmCrwKRiMbOSKCAbkQr ... YiNntciJszsZVRypXz1HTfa3gbcKoNXHon8=
aws_expiration=2019-11-22T20:33:18.000Z

𝜆 aws ssm start-session --target i-e2f189dashfdf65weqfwda2 --profile session

Starting session with SessionId: ec2-user-094c34172bdc6fc22
sh-4.2$ whoami
SSM_pwr_user
  • Run Session Manager commands
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
𝜆 aws ssm send-command --instance-ids i-e2f189dashfdf65weqfwda2 --document-name AWS-RunShellScript --comment "IP config" --parameters commands=ifconfig --output text \
    --profile session
COMMAND	39e80533-376e-46fa-bb11-8daf040fe80f	IP config	0	0	AWS-RunShellScript		0	1574377262.9	50	0			1574370062.9		Pending	Pending	1
CLOUDWATCHOUTPUTCONFIG		False
INSTANCEIDS	i-e2f189dashfdf65weqfwda2
NOTIFICATIONCONFIG
COMMANDS	ifconfig

𝜆 aws ssm list-command-invocations --command-id 39e80533-376e-46fa-bb11-8daf040fe80f --details --profile session
{
    "CommandInvocations": [
        {
            "Comment": "IP config",
            "Status": "Success",
            "CommandPlugins": [
                {
                    "Status": "Success",
                    "ResponseStartDateTime": 1574370063.609,
                    "StandardErrorUrl": "",
                    "OutputS3BucketName": "",
                    "OutputS3Region": "ap-southeast-2",
                    "OutputS3KeyPrefix": "",
                    "ResponseCode": 0,
                    "Output": "eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001\n        inet 172.31.9.155  netmask 255.255.240.0  broadcast 172.31.15.255\n        inet6 fe80::69:c9ff:fe76:91ae  prefixlen 64  scopeid 0x20<link>\n        ether 02:69:c9:76:91:ae  txqueuelen 1000  (Ethernet)\n        RX packets 60339  bytes 17254584 (16.4 MiB)\n        RX errors 0  dropped 0  overruns 0  frame 0\n        TX packets 56971  bytes 13112880 (12.5 MiB)\n        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0\n\nlo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536\n        inet 127.0.0.1  netmask 255.0.0.0\n        inet6 ::1  prefixlen 128  scopeid 0x10<host>\n        loop  txqueuelen 1000  (Local Loopback)\n        RX packets 0  bytes 0 (0.0 B)\n        RX errors 0  dropped 0  overruns 0  frame 0\n        TX packets 0  bytes 0 (0.0 B)\n        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0\n\n",
                    "ResponseFinishDateTime": 1574370063.634,
                    "StatusDetails": "Success",
                    "StandardOutputUrl": "",
                    "Name": "aws:runShellScript"
                }
            ],
            "ServiceRole": "",
            "CloudWatchOutputConfig": {
                "CloudWatchLogGroupName": "",
                "CloudWatchOutputEnabled": false
            },
            "InstanceId": "i-e2f189dashfdf65weqfwda2",
            "DocumentName": "AWS-RunShellScript",
            "NotificationConfig": {
                "NotificationArn": "",
                "NotificationEvents": [],
                "NotificationType": ""
            },
            "DocumentVersion": "",
            "StatusDetails": "Success",
            "StandardOutputUrl": "",
            "StandardErrorUrl": "",
            "InstanceName": "",
            "CommandId": "39e80533-376e-46fa-bb11-8daf040fe80f",
            "RequestedDateTime": 1574370062.995
        }
    ]
}
  • Open a connection forwarding session to a remote port
1
2
3
4
5
𝜆 aws ssm start-session --target i-e2f189dashfdf65weqfwda2 --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"portNumber":["3306"],"localPortNumber":["13306"],"host":["remote-mysql-host-name"]}' \
    --profile session

𝜆 mysql -h 127.0.0.1 --port 13306 -u admin -p

  • Monitor AWS Session Manager log /var/log/amazon/ssm/amazon-ssm-agent.log on EC2 instance

  • Test ssh command with session token

1
2
3
4
5
6
7
8
𝜆 AWS_PROFILE=session ssh -i .ssh/aws-key.pem -l ec2-user i-e2f189dashfdf65weqfwda2
Last login: Fri Nov 22 00:19:32 2019 from localhost

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
  • Test scp command with session token
1
2
𝜆 AWS_PROFILE=session scp -i .ssh/aws-key.pem /tmp/stack-overflow.log ec2-user@i-e2f189dashfdf65weqfwda2:/tmp
stack-overflow.log                                          100%   67KB 437.0KB/s   00:00

Without using scp, transferring files directly is not possible with the AWS Session Manager. You should use S3 bucket and the AWS CLI to exchange data.

  • Test ssh tunnel
1
𝜆 AWS_PROFILE=session ssh -i .ssh/aws-key.pem -L 443:www.google.com:443 -l ec2-user@i-e2f189dashfdf65weqfwda2

OKTA

Set up integrated OKTA authentication with session. Have you OKTA AWS CLI installed at first, and configure it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
𝜆 cat ~/.okta-aws
[default]
base-url = hello.paradise.org

## The remaining parameters are optional.
## You will be prompted for them, if they're not included here.
username = terrence.miao@paradise.org

# Current choices are: GOOGLE or OKTA
factor = OKTA

# AWS role name (match one of the options prompted for by "Please select the AWS role" when this parameter is not specified
role =

# Found in Okta's configuration for your AWS account.
app-link = https://hello.paradise.org/home/amazon_aws/0oa1ch3l6/272

# duration in seconds to request a session token for, make sure your accounts (both AWS itself and the associated okta application) allow for large durations. default: 3600
duration = 28800

Create session:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
𝜆 bin/okta/okta --okta-profile default --force --profile session
Enter password:
Multi-factor Authentication required.
Pick a factor:
[ 0 ] Okta Verify App: SmartPhone_Android: ONEPLUS A5010
[ 1 ] token:software:totp( OKTA ) : terrence.miao@paradise.org
Selection: 0
1: arn:aws:iam::994385754915:role/federation/DeveloperPowerUser
2: arn:aws:iam::354184710243:role/federation/DeveloperPowerUser
3: arn:aws:iam::855034721059:role/federation/DeveloperPowerUser
4: arn:aws:iam::944986439712:role/federation/DeveloperPowerUser
5: arn:aws:iam::944986439712:role/federation/Operations
6: arn:aws:iam::015951833499:role/federation/DeveloperPowerUser
Please select the AWS role: 3
Session token expires on: 2019-12-05 06:30:18+00:00

Azure

Set up integrated Azure authentication with session. Have you AWS Azure CLI installed at first, and configure it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
𝜆 cat ~/.aws/config
[profile session]
output=json
region=ap-southeast-2
azure_tenant_id=paradise.org
azure_app_id_uri=dac7d48b-7f45-47bb-bdae-d3c4f8351839
azure_default_username=terrence.miao@paradise.org
azure_default_role_arn=arn:aws:iam::994385754915:role/federation/DeveloperPowerUser
azure_default_duration_hours=12
azure_default_remember_me=false

[profile session-cicd]
output=json
region=ap-southeast-2
role_arn=arn:aws:iam::994385754915:role/ap-sc-iam-cicd-ap-southeast-2
source_profile=session

Create session:

1
2
3
4
5
6
7
8
9
10
11
𝜆 bin/aws-azure-cli/aws-azure-cli --profile session --mode gui
Logging in with profile 'session'...
Using AWS SAML endpoint https://signin.aws.amazon.com/saml
Looking in /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
Found browser in /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
Please complete the login in the opened window
? Role: arn:aws:iam::994385754915:role/federation/DeveloperPowerUser
? Session Duration Hours (up to 12): 12
Assuming role arn:aws:iam::994385754915:role/federation/DeveloperPowerUser
Requesting session duration 43200s
Session expires Sun Feb 19 2023 06:46:43 GMT+1100 (Australian Eastern Daylight Time)

Login EC2 instance, overwritting default ap-southeast-2 region:

1
2
𝜆 aws ssm start-session --target i-04ee902e33625c4f3 --profile session --region us-east-2 --debug
Starting session with SessionId: terrence.miao@paradise.org-05411f2e2d5a58b0b

Attach a new key pair to EC2 instance by:

1
2
𝜆 ssh-keygen -y -f .ssh/aws-key.pem
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCL ... asasf2qcvASEFWF

Find out EC2 Instance ID by querying Instance Name:

1
2
3
4
5
𝜆 aws ec2 describe-instances --profile session \
    --filter "Name=tag:Name,Values=EC2Stack-Apptier/EC2" \
    --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" \
    --output text
i-04ee902e33625c4f3

References

A simple approach to export / import AWS DynamoDB table items

Guestbook

Try to export items from AWS Test environment DynamoDB tables into Production. A simple Bash SHELL script, with a few commands and AWS CLI could do the work.

  • In AWS DynamoDB console, export selected items into .csv file like this:
1
2
3
4
5
6
7
𝜆 cat dws-tile-publisher-ptest-tile.csv
"id (S)","available_to (SS)","benefit_type (S)","category_id (S)","created_date (S)","created_user (S)","description (S)","image_bucket (S)","image_key (S)","last_modified_date (S)","last_modified_user (S)","status (S)","sub_category_id (S)","title (S)","valid_from (S)","valid_to (S)","claim_url (S)","discount_code (S)"
"950c529b-d6ae-472b-b44a-510ec201c167","{ ""0b1b9ed4-e171-4eaf-9d20-1a870ab7cc7c"", ""2df9daf4-da50-4ff9-9322-969d02c178f4"", ""b6fdcdc7-ffe6-4007-ae56-f5ab545768ec"", ""bc1c7285-ac47-4d06-b4c6-6f3780cbfb3f"" } ","DISCOUNT","8d27fb2d-c5f8-487a-a869-2ffd0e476eb6","2019-04-23T01:02:18.123","Angelo.Woods@test.npe.paradise.org","Testing descriptions","dws-tile-images-ptest","Samsung-Galaxy-10-Leak.jpg","2019-04-23T01:02:18.125","Angelo.Woods@test.npe.paradise.org","ARCHIVED","6936a6ed-fa97-4439-9129-c43e288011b3","Samsung Galaxy 10","2019-03-24T01:00","2021-02-25T23:00","http://www.google.com",

...

"095cc9b4-fd64-4479-8817-0b35b8ddcbc2","{ ""0b1b9ed4-e171-4eaf-9d20-1a870ab7cc7c"", ""2df9daf4-da50-4ff9-9322-969d02c178f4"", ""b6fdcdc7-ffe6-4007-ae56-f5ab545768ec"", ""bc1c7285-ac47-4d06-b4c6-6f3780cbfb3f"" } ","DISCOUNT","1f028ecb-4115-4378-b00e-3fcd4cbdf54d","2019-04-23T03:50:31.226","Angelo.Woods@test.npe.paradise.org","<div>Employees now have access to an exclusive member offers portal through the Samsung Employee Purchase program. Get up to 35% off the RRP across a wide range of products including mobile phones, tablets, televisions, refrigerators, washing machines, monitors and more.</div><div><br /></div><div><br /><br /></div>","dws-tile-images-ptest","samsung.jpg","2019-04-23T03:50:31.228","Angelo.Woods@test.npe.paradise.org","PUBLISHED","ce07c19b-dd87-4890-9ad3-6acfe0998496","Samsung","2019-04-17T01:00","2025-01-01T23:00","https://shop.samsung.com/au/multistore/auepp/austpost_au/","Simply go to link provided to access the site. Browse our categories, add your products to cart and checkout. Happy Shopping! "
  • Generate a key:value hashtable used later. Key is “id” column, value is “image_key” colomn:
1
2
3
4
5
6
𝜆 awk -F"\",\"" '{print $1 ":" $9 "\""}' dws-tile-publisher-test-tile.csv | sed 's/jpg/json/' | sed 's/png/json/' | sed -n '1d;p'
"950c529b-d6ae-472b-b44a-510ec201c167:Samsung-Galaxy-10-Leak.json"

...

"095cc9b4-fd64-4479-8817-0b35b8ddcbc2:samsung.json"
  • Run the bash script and retrieve item from AWS DyanmoDB table, tranform it and output into a JSON file for import into AWS Production production table:
  • A generated JSON file looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
𝜆 cat Samsung-Galaxy-10-Leak.json
{
    "status": {
        "S": "PUBLISHED"
    },
    "valid_from": {
        "S": "2019-03-24T01:00"
    },
    "sub_category_id": {
        "S": "6936a6ed-fa97-4439-9129-c43e288011b3"
    },
    "description": {
        "S": "Testing descriptions"
    },
    "image_key": {
        "S": "Samsung-Galaxy-10-Leak.jpg"
    },
    "claim_url": {
        "S": "http://www.google.com"
    },
    "last_modified_date": {
        "S": "2019-04-23T01:02:18.125"
    },
    "valid_to": {
        "S": "2021-02-25T23:00"
    },
    "created_user": {
        "S": "Noel.French@paradise.org"
    },
    "benefit_type": {
        "S": "DISCOUNT"
    },
    "last_modified_user": {
        "S": "Noel.French@paradise.org"
    },
    "created_date": {
        "S": "2019-04-23T01:02:18.123"
    },
    "title": {
        "S": "Samsung Galaxy 10"
    },
    "category_id": {
        "S": "8d27fb2d-c5f8-487a-a869-2ffd0e476eb6"
    },
    "image_bucket": {
        "S": "dws-tile-images-prod"
    },
    "id": {
        "S": "950c529b-d6ae-472b-b44a-510ec201c167"
    },
    "available_to": {
        "SS": [
            "0b1b9ed4-e171-4eaf-9d20-1a870ab7cc7c",
            "2df9daf4-da50-4ff9-9322-969d02c178f4",
            "b6fdcdc7-ffe6-4007-ae56-f5ab545768ec",
            "bc1c7285-ac47-4d06-b4c6-6f3780cbfb3f"
        ]
    }
}
  • Now run the bash script and import JSON file into AWS Production DynamoDB table:

Keep running

Guestbook

许多杰出的人物都是长跑爱好者，比如阿兰·图灵。

图灵的头衔有很多，传奇性的数学家、逻辑学家、现代计算机之父。但很少人知道他也是一位出色的跑者，甚至差点站上了奥运会的舞台。

图灵从中学就开始跑步，但直到 30 多岁才正式进行长跑训练。

跑道上的图灵“大器晚成”，出色的运动天赋丝毫不逊色于他的头脑。

14 岁那年，他在谢伯恩寄宿高中就读的第一日，由于大罢工，当地的公共交通设施全部停运。图灵没有等到交通恢复，就从南安普敦一口气骑行 96 公里到达位于多塞特郡西北的学校上学，引发全校轰动。

1931 年，当他进入剑桥国王学院学习时，他曾多次在学校和伊利镇之间进行折返跑，一个来回就差不多 50 公里。

1947 年，在莱斯特郡拉夫堡（Loughborough）大学体育场举行的英国业余田径协会马拉松锦标赛上，图灵跑出了他在马拉松赛中的个人最好成绩 2 小时 46 分 03 秒，在那场比赛中名列第五。

在 1948 年伦敦的奥运会马拉松比赛上，当时冠军的成绩是 2 小时 34 分 52 秒。图灵仅比金牌得主慢了 11 分钟。而据英国《泰晤士报》报道，图灵的最好成绩在当年（1947年）足以排名世界前三。

Alan Turning running

When Alan Turing was asked why he punished himself so much in training. He said “I have such a stressful job that the only way I can get it out of my mind is by running hard; its the only way I can get some release.”

奥斯卡最佳改编剧本电影《模仿游戏》The Imitation Game 是根据他的传记《Alan Turing: The Enigma》拍摄的。

The Imitation Game

Movie scene: Alan Turning running

苹果公司的商标一度被误认为是源于图灵自杀时咬下的苹果，但该图案的设计师和苹果公司都否认这一说法。

Stephen Fry said on the BBC show QI XL in 2011 that his friend Steve Jobs said of the Turing tale, “It isn’t true, but God we wish it were!”

乔布斯被英国广播公司电视节目《QI》主持人史蒂芬·弗莱问到此事时说：“这（LOGO 向图灵致敬）不是真的，但是上帝啊，我们希望它是真的。”

"Apple logo"

Keep running.

Add final and javadoc for auto-generated getters and setters in IntelliJ

Guestbook

Default final modifier

To satisfy Checkstyle setup. In IntelliJ (version 2018.3.2) Preferences | Editor > Code Style > Java, select “Code Generation” tab, check “Make generated parameters final”

Final Modifier

javadoc for auto-generated getters and setters

Create customised Getter Template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/**
* Gets $field.name.
*
* @return value of $field.name
*/
#if($field.modifierStatic)
static ##
#end
$field.type ##
#set($name = $StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project))))
#if ($field.boolean && $field.primitive)
    is##
#else
    get##
#end
${name}() {
    return $field.name;
}

Getter Template

Create customised Setter Template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/**
* Sets $field.name.
*
* @param $field.name value of $field.name
*/
#set($paramName = $helper.getParamName($field, $project))
#if($field.modifierStatic)
static ##
#end
void set$StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project)))($field.type $paramName) {
#if ($field.name == $paramName)
    #if (!$field.modifierStatic)
    this.##
    #else
        $classname.##
    #end
#end
$field.name = $paramName;
}

Setter Template

Use the customised Getter and Setter templates when automatically generate getters and setters:

auto-generated getters setters

Year 2018 in the Rear View Mirror

Guestbook

2018 是一个让人敬畏恐惧“逢八必变”的年份。2018 是过去十年中最糟糕的一年，但也许是未来十年里最好的一年。

2018 年过去了，我很怀念它。

Year 2018 - 01

图 1：2018 年是在过去十年中唯一的各项投资都为负增长的一年。

Year 2018 - 02

图2：美国股市进入下半年一直跌跌不休。但在 Boxing Day 却创造了单日涨幅的纪录。

Year 2018 - 03

图 3：黎曼 Zeta 函数，用于计算素数分布。2018 年多人声称证明了百年未解决的黎曼猜想，但最终都是闹剧收场。黎曼猜想依然有如最为壮美，巍峨，险峻的奇山，矗立在人类的智力巅峰之上。

Year 2018 - 04

图 4：The cost of living，city by city。生活有多贵？中国六城市入选全球生活成本最昂贵的 top 25 list.

Year 2018 - 05

图 5：肯尼亚的 Eliud Kipchoge 在 2018 年柏林马拉松上创造的 2 小时 1 分 39 秒的马拉松世界纪录。人类仍然在试图突破 2 小时完成马拉松的运动极限。

Year 2018 - 06

图 6：中国·上海

Year 2018 - 07

图 7：Chinese President Xi Jinping,centre, arrives with Premier Li Keqiang, left, for the opening session of the Chinese People’s Political Consultative Conference in Beijing’s Great Hall of the People on March 3, 2018.

Year 2018 - 08

图 8：Chinese President Xi Jinping Speech.

Year 2018 - 09

图 9：在 2018 年 Pantone Colour of the year，Ultra Violet 紫外光后，潘通子给出 2019 年度流行色 Living Coral 珊瑚橙。

Year 2018 - 10

图 10：2018 年度汉字，由 “穷” 和 “丑” 合体所组成，读音是 “qiou”（音同糗），即又穷又丑的意思。不少网友表示，这字应该念 “wo”（我）。

Year 2018 - 11

图 11：电影《甲方乙方》的结局。杨立新走到雪地里，抬起头，和每个普通人都一样，要继续面对风雪。屋子里面是红灯笼，亮堂堂的。外面已经是冷色调了。外面就是现实的生活。

“我很怀念它”，经历越多，这句话带来的感触就越厚重。

时间，梦想和爱，都是如此。

Step by step run Kafka on Mac

Guestbook 
1
2
3
4
5
6
terrence@igloo /usr/local/kafka
15:51:25 𝜆 diff config/server.properties config/server.properties.orig
31c31
< listeners=PLAINTEXT://localhost:9092
---
> #listeners=PLAINTEXT://:9092
  • Start Zookeeper
1
igloo:kafka root# bin/zookeeper-server-start.sh config/zookeeper.properties
  • Start Kafka
1
igloo:kafka root# bin/kafka-server-start.sh config/server.properties
  • Create a new Kafka topic named “test”
1
igloo:kafka root# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
  • Initialise Producer in Console
1
2
3
4
5
terrence@igloo /usr/local/kafka_2.12-2.1.0
16:04:19 𝜆 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>Hello World
>Wow
>
  • Initialize Consumer in Console to receive the messages
1
2
3
4
terrence@igloo /usr/local/kafka_2.12-2.1.0
16:06:11 𝜆 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Hello World
Wow

Run Kafka Manager, an UI tool for managing Kafka.

1
2
terrence@igloo ~/bin/kafka-manager-1.3.3.18
16:16:47 𝜆 env ZK_HOSTS="localhost:2181" bin/kafka-manager

Then visit http://localhost:9000

Create a Kafka cluster at first:

Kafka Cluster

Visit “test” topic in Kafka cluster:

Kafka Manager