Try to export items from AWS Test environment DynamoDB tables into Production. A simple Bash SHELL script, with a few commands and AWS CLI could do the work.

• In AWS DynamoDB console, export selected items into .csv file like this:

1
2
3
4
5
6
7

𝜆 cat dws-tile-publisher-ptest-tile.csv
"id (S)","available_to (SS)","benefit_type (S)","category_id (S)","created_date (S)","created_user (S)","description (S)","image_bucket (S)","image_key (S)","last_modified_date (S)","last_modified_user (S)","status (S)","sub_category_id (S)","title (S)","valid_from (S)","valid_to (S)","claim_url (S)","discount_code (S)"
"950c529b-d6ae-472b-b44a-510ec201c167","{ ""0b1b9ed4-e171-4eaf-9d20-1a870ab7cc7c"", ""2df9daf4-da50-4ff9-9322-969d02c178f4"", ""b6fdcdc7-ffe6-4007-ae56-f5ab545768ec"", ""bc1c7285-ac47-4d06-b4c6-6f3780cbfb3f"" } ","DISCOUNT","8d27fb2d-c5f8-487a-a869-2ffd0e476eb6","2019-04-23T01:02:18.123","Angelo.Woods@test.npe.paradise.org","Testing descriptions","dws-tile-images-ptest","Samsung-Galaxy-10-Leak.jpg","2019-04-23T01:02:18.125","Angelo.Woods@test.npe.paradise.org","ARCHIVED","6936a6ed-fa97-4439-9129-c43e288011b3","Samsung Galaxy 10","2019-03-24T01:00","2021-02-25T23:00","http://www.google.com",

...

"095cc9b4-fd64-4479-8817-0b35b8ddcbc2","{ ""0b1b9ed4-e171-4eaf-9d20-1a870ab7cc7c"", ""2df9daf4-da50-4ff9-9322-969d02c178f4"", ""b6fdcdc7-ffe6-4007-ae56-f5ab545768ec"", ""bc1c7285-ac47-4d06-b4c6-6f3780cbfb3f"" } ","DISCOUNT","1f028ecb-4115-4378-b00e-3fcd4cbdf54d","2019-04-23T03:50:31.226","Angelo.Woods@test.npe.paradise.org","<div>Employees now have access to an exclusive member offers portal through the Samsung Employee Purchase program. Get up to 35% off the RRP across a wide range of products including mobile phones, tablets, televisions, refrigerators, washing machines, monitors and more.</div><div><br /></div><div><br /><br /></div>","dws-tile-images-ptest","samsung.jpg","2019-04-23T03:50:31.228","Angelo.Woods@test.npe.paradise.org","PUBLISHED","ce07c19b-dd87-4890-9ad3-6acfe0998496","Samsung","2019-04-17T01:00","2025-01-01T23:00","https://shop.samsung.com/au/multistore/auepp/austpost_au/","Simply go to link provided to access the site. Browse our categories, add your products to cart and checkout. Happy Shopping! "

• Generate a key:value hashtable used later. Key is “id” column, value is “image_key” colomn:

1
2
3
4
5
6

𝜆 awk -F"\",\"" '{print $1 ":" $9 "\""}' dws-tile-publisher-test-tile.csv | sed 's/jpg/json/' | sed 's/png/json/' | sed -n '1d;p'
"950c529b-d6ae-472b-b44a-510ec201c167:Samsung-Galaxy-10-Leak.json"

...

"095cc9b4-fd64-4479-8817-0b35b8ddcbc2:samsung.json"

• Run the bash script and retrieve item from AWS DyanmoDB table, tranform it and output into a JSON file for import into AWS Production production table:

• A generated JSON file looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

𝜆 cat Samsung-Galaxy-10-Leak.json
{
    "status": {
        "S": "PUBLISHED"
    },
    "valid_from": {
        "S": "2019-03-24T01:00"
    },
    "sub_category_id": {
        "S": "6936a6ed-fa97-4439-9129-c43e288011b3"
    },
    "description": {
        "S": "Testing descriptions"
    },
    "image_key": {
        "S": "Samsung-Galaxy-10-Leak.jpg"
    },
    "claim_url": {
        "S": "http://www.google.com"
    },
    "last_modified_date": {
        "S": "2019-04-23T01:02:18.125"
    },
    "valid_to": {
        "S": "2021-02-25T23:00"
    },
    "created_user": {
        "S": "Noel.French@paradise.org"
    },
    "benefit_type": {
        "S": "DISCOUNT"
    },
    "last_modified_user": {
        "S": "Noel.French@paradise.org"
    },
    "created_date": {
        "S": "2019-04-23T01:02:18.123"
    },
    "title": {
        "S": "Samsung Galaxy 10"
    },
    "category_id": {
        "S": "8d27fb2d-c5f8-487a-a869-2ffd0e476eb6"
    },
    "image_bucket": {
        "S": "dws-tile-images-prod"
    },
    "id": {
        "S": "950c529b-d6ae-472b-b44a-510ec201c167"
    },
    "available_to": {
        "SS": [
            "0b1b9ed4-e171-4eaf-9d20-1a870ab7cc7c",
            "2df9daf4-da50-4ff9-9322-969d02c178f4",
            "b6fdcdc7-ffe6-4007-ae56-f5ab545768ec",
            "bc1c7285-ac47-4d06-b4c6-6f3780cbfb3f"
        ]
    }
}

• Now run the bash script and import JSON file into AWS Production DynamoDB table:

许多杰出的人物都是长跑爱好者，比如阿兰·图灵。

图灵的头衔有很多，传奇性的数学家、逻辑学家、现代计算机之父。但很少人知道他也是一位出色的跑者，甚至差点站上了奥运会的舞台。

图灵从中学就开始跑步，但直到 30 多岁才正式进行长跑训练。

跑道上的图灵“大器晚成”，出色的运动天赋丝毫不逊色于他的头脑。

14 岁那年，他在谢伯恩寄宿高中就读的第一日，由于大罢工，当地的公共交通设施全部停运。图灵没有等到交通恢复，就从南安普敦一口气骑行 96 公里到达位于多塞特郡西北的学校上学，引发全校轰动。

1931 年，当他进入剑桥国王学院学习时，他曾多次在学校和伊利镇之间进行折返跑，一个来回就差不多 50 公里。

1947 年，在莱斯特郡拉夫堡（Loughborough）大学体育场举行的英国业余田径协会马拉松锦标赛上，图灵跑出了他在马拉松赛中的个人最好成绩 2 小时 46 分 03 秒，在那场比赛中名列第五。

在 1948 年伦敦的奥运会马拉松比赛上，当时冠军的成绩是 2 小时 34 分 52 秒。图灵仅比金牌得主慢了 11 分钟。而据英国《泰晤士报》报道，图灵的最好成绩在当年（1947年）足以排名世界前三。

When Alan Turing was asked why he punished himself so much in training. He said “I have such a stressful job that the only way I can get it out of my mind is by running hard; its the only way I can get some release.”

奥斯卡最佳改编剧本电影《模仿游戏》The Imitation Game 是根据他的传记《Alan Turing: The Enigma》拍摄的。

苹果公司的商标一度被误认为是源于图灵自杀时咬下的苹果，但该图案的设计师和苹果公司都否认这一说法。

Stephen Fry said on the BBC show QI XL in 2011 that his friend Steve Jobs said of the Turing tale, “It isn’t true, but God we wish it were!”

乔布斯被英国广播公司电视节目《QI》主持人史蒂芬·弗莱问到此事时说：“这（LOGO 向图灵致敬）不是真的，但是上帝啊，我们希望它是真的。”

Keep running.

Default final modifier

To satisfy Checkstyle setup. In IntelliJ (version 2018.3.2) Preferences | Editor > Code Style > Java, select “Code Generation” tab, check “Make generated parameters final”

javadoc for auto-generated getters and setters

Create customised Getter Template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

/**
* Gets $field.name.
*
* @return value of $field.name
*/
#if($field.modifierStatic)
static ##
#end
$field.type ##
#set($name = $StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project))))
#if ($field.boolean && $field.primitive)
    is##
#else
    get##
#end
${name}() {
    return $field.name;
}

Create customised Setter Template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

/**
* Sets $field.name.
*
* @param $field.name value of $field.name
*/
#set($paramName = $helper.getParamName($field, $project))
#if($field.modifierStatic)
static ##
#end
void set$StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project)))($field.type $paramName) {
#if ($field.name == $paramName)
    #if (!$field.modifierStatic)
    this.##
    #else
        $classname.##
    #end
#end
$field.name = $paramName;
}

Use the customised Getter and Setter templates when automatically generate getters and setters:

2018 是一个让人敬畏恐惧“逢八必变”的年份。2018 是过去十年中最糟糕的一年，但也许是未来十年里最好的一年。

2018 年过去了，我很怀念它。

图 1：2018 年是在过去十年中唯一的各项投资都为负增长的一年。

图2：美国股市进入下半年一直跌跌不休。但在 Boxing Day 却创造了单日涨幅的纪录。

图 3：黎曼 Zeta 函数，用于计算素数分布。2018 年多人声称证明了百年未解决的黎曼猜想，但最终都是闹剧收场。黎曼猜想依然有如最为壮美，巍峨，险峻的奇山，矗立在人类的智力巅峰之上。

图 4：The cost of living，city by city。生活有多贵？中国六城市入选全球生活成本最昂贵的 top 25 list.

图 5：肯尼亚的 Eliud Kipchoge 在 2018 年柏林马拉松上创造的 2 小时 1 分 39 秒的马拉松世界纪录。人类仍然在试图突破 2 小时完成马拉松的运动极限。

图 6：中国·上海

图 7：Chinese President Xi Jinping,centre, arrives with Premier Li Keqiang, left, for the opening session of the Chinese People’s Political Consultative Conference in Beijing’s Great Hall of the People on March 3, 2018.

图 8：Chinese President Xi Jinping Speech.

图 9：在 2018 年 Pantone Colour of the year，Ultra Violet 紫外光后，潘通子给出 2019 年度流行色 Living Coral 珊瑚橙。

图 10：2018 年度汉字，由 “穷” 和 “丑” 合体所组成，读音是 “qiou”（音同糗），即又穷又丑的意思。不少网友表示，这字应该念 “wo”（我）。

图 11：电影《甲方乙方》的结局。杨立新走到雪地里，抬起头，和每个普通人都一样，要继续面对风雪。屋子里面是红灯笼，亮堂堂的。外面已经是冷色调了。外面就是现实的生活。

“我很怀念它”，经历越多，这句话带来的感触就越厚重。

时间，梦想和爱，都是如此。

• Download Kafka binary file from https://kafka.apache.org/downloads and install it

• Set up Kafka listen on PLAINTEXT://localhost:9092

1
2
3
4
5
6

terrence@igloo /usr/local/kafka
15:51:25 𝜆 diff config/server.properties config/server.properties.orig
31c31
< listeners=PLAINTEXT://localhost:9092
---
> #listeners=PLAINTEXT://:9092

• Start Zookeeper

1

igloo:kafka root# bin/zookeeper-server-start.sh config/zookeeper.properties

• Start Kafka

1

igloo:kafka root# bin/kafka-server-start.sh config/server.properties

• Create a new Kafka topic named “test”

1

igloo:kafka root# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

• Initialise Producer in Console

1
2
3
4
5

terrence@igloo /usr/local/kafka_2.12-2.1.0
16:04:19 𝜆 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>Hello World
>Wow
>

• Initialize Consumer in Console to receive the messages

1
2
3
4

terrence@igloo /usr/local/kafka_2.12-2.1.0
16:06:11 𝜆 bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Hello World
Wow

Run Kafka Manager, an UI tool for managing Kafka.

1
2

terrence@igloo ~/bin/kafka-manager-1.3.3.18
16:16:47 𝜆 env ZK_HOSTS="localhost:2181" bin/kafka-manager

Then visit http://localhost:9000

Create a Kafka cluster at first:

Visit “test” topic in Kafka cluster:

In Apache HTTPClient 4.x, PoolingHttpClientConnectionManager and BasicHttpClientConnectionManager can be used as Connection Managers for HTTP Client. By default, above two Connection Managers load cacerts trust store, from $JRE_HOME/lib/security directory, not programmatically specified.

Since Apache Camel starts supporting HTTP4 Component, camel-http4 is using Apache HTTPClient 4.x. The following is a solution how to customise PoolingHttpClientConnectionManager and SSLContext to use specified KeyStore and TrustStore.

In application.properties file defined environment specified KeyStore and TrustStore:

1
2
3
4
5
6

...
keystore.location=classpath:certificates/client-certs.jks
keystore.password=client123
truststore.location=classpath:certificates/trusted-cacerts.jks
truststore.password=changeit
...

Without Connection Managers like PoolingHttpClientConnectionManager and BasicHttpClientConnectionManager in Apache HTTPClient 4.x, Apache Camel Route can be written in this way, binding programmatically specified KeyStore and TrustStore.

References

• Apache Camel HTTP4 Component, http://camel.apache.org/http4.html
• Java SSL with Multiple KeyStores, http://codyaray.com/2013/04/java-ssl-with-multiple-keystores
• Multiple Keystores in Apache Camel/HttpClient, http://codyaray.com/2013/05/multiple-keystores-in-apache-camelhttpclient

Inspirational Thought from Jeff Bezos.

最新的苹果发布会是一次团结的大会，求实的大会，胜利的大会。在以苹果总书记库克同志为核心的领导下，Apple Design 拨乱反正，继往开来，重拾经典。

Macbook Air 得以继续，Mac Mini 得到重生。iPad Pro 在经过不断的探索、失败后，终于拿出一款震撼心灵的设计，以及一款与时共进的 Apple Pencil。

新 iPad Pro 的机身背面内置了102 块磁铁，保证了 flat 扁平的这款 Apple Pencil 可牢固吸附在机身上，同时也能给 Apple Pencil 进行无线充电。这个设计优雅，实用，点赞必须。Furthermore, 边框塑性换成了方正的造型，重回 iPhone 5 方正边框的精典设计。从屏幕到外形，从外设到连接性，新 iPad Pro 的所有功能都是朝着高生产力笔记本电脑的 ecosystem 前进。

今天墨尔本赛马节，全维多利亚州人民在春天的草地上撒野。信主也准备潇洒一把。

登上了苹果网站，发现新 Apple Pencil 提供个人定制激光刻字服务。输入洋文 “Best there ever will be“ （后无来者的意思），苹果嫌弃太长。换成国语 “民族英雄 小平赠“，苹果又 handle 不了。无奈只好取消 $2,349.00 新 iPad Pro 和 $199.00 新 Apple Pencil 的订单 。

有钱花不出去，郁闷啊 … …

AWS DynamoDB is a good persistence solution for a specific solution. However, it’s not for a growing and changing application that could need new indexes and queries for its ever expanding features at any time. At this moment it is where the flexibility and speed of a relational database really shined through.

Now, a DynamoDB table with Consumers Preferences data urgently needs to update:

• RENAME one EXISTING attribute
• ADD a new attribute
• SET a value in the new attribute for ALL EXISTING items in DynamoDB’s new table

AWS lambda function plays handy here to migrate data, and sync newly inserted, modified / updated, deleted items between existing and new DynamoDB tables.

• Create new DynamoDB table

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

$ aws dynamodb create-table \
    --table-name userpreferences-ptest-02-USER_PREFERENCESV2 \
    --attribute-definitions AttributeName=id,AttributeType=S AttributeName=preferenceType,AttributeType=S \
    --key-schema AttributeName=id,KeyType=HASH  AttributeName=preferenceType,KeyType=RANGE \
    --provisioned-throughput ReadCapacityUnits=100,WriteCapacityUnits=100

{
    "TableDescription": {
        "TableArn": "arn:aws:dynamodb:ap-southeast-2:123456789012:table/userpreferences-ptest-02-USER_PREFERENCESV2",
        "AttributeDefinitions": [
            {
                "AttributeName": "id",
                "AttributeType": "S"
            },
            {
                "AttributeName": "preferenceType",
                "AttributeType": "S"
            }
        ],
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "WriteCapacityUnits": 100,
            "ReadCapacityUnits": 100
        },
        "TableSizeBytes": 0,
        "TableName": "userpreferences-ptest-02-USER_PREFERENCESV2",
        "TableStatus": "CREATING",
        "TableId": "d116efdc-1234-5678-90ab-011de3e124fe",
        "KeySchema": [
            {
                "KeyType": "HASH",
                "AttributeName": "id"
            },
            {
                "KeyType": "RANGE",
                "AttributeName": "preferenceType"
            }
        ],
        "ItemCount": 0,
        "CreationDateTime": 1540273906.1059999
    }
}

Migrate Data

• Create Migrate Data Lambda function

Increase Memory and Runtime Timeout https://docs.aws.amazon.com/lambda/latest/dg/limits.html in case of execution pre-maturely ended without finishing the migration. Furthermore, in case of overcharging DynamoDB due to its limits https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html add time delay in Lambda function.

migrate.js

• Add AWS DynamoDB Lambda execution role

userpreferences-ptest-02-migrateRole

• Create a simple test event that can kick off function

1
2
3
4
5

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

Trigger run the function. Should see the data migrated from existing DynamoDB table into new table.

Log can be found at AWS CloudWatch Log Groups /aws/lambda/userpreferences-ptest-02-migrate

Sync Data

• Enable Stream on the existing DynamoDB table

• Add a new trigger for DynamoDB table

• Create a new Lambda function linked to trigger

sync.js

• Trigger Testing

AWS Lambda built-in test can test trigger:

• Logging

Lambda function log can be found on AWS CloudWatch Log Groups /aws/lambda/userpreferences-ptest-02-sync

Counter Data

This Lambda function can count the number of items in DynamoDB table.

counter.js

Async call, callback and Non-blocking, it’s very hard implement so in every applications. In addition, reject promises or async functions, don’t handle them with a catch, NodeJS will raise a warning. In a large complex applications with lots of async, having a single unhandled promise or await function terminate NodeJS, or have to handle them with try and catch in every place (spaghetti code again?) would be very bad.

An example of AWS DynamoDB error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

2018-11-14T02:20:50.742Z	715f18fb-e7b3-11e8-b5c4-d75f9089dd50	Error thrown: { ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:48:27)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
message: 'The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.',
code: 'ProvisionedThroughputExceededException',
time: 2018-11-14T02:20:50.687Z,
requestId: 'C38MODOISAJEGTVPI2ISOPFGDBVV4KQNSO5AEMVJF66Q9ASUAAJG',
statusCode: 400,
retryable: true }

AWS example doesn’t throw the error in the catch block, it returns error instead, so any errors end up in the catch block. And return promises early and use Promise.all() method.

References

• DynamoDB: Changing table schema, https://www.abhayachauhan.com/2018/01/dynamodb-changing-table-schema/
• Tutorial: Processing New Items in a DynamoDB Table, https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.Lambda.Tutorial.html
• How to escape async/await hell, https://medium.freecodecamp.org/avoiding-the-async-await-hell-c77a0fb71c4c
• Node.js 8.10 runtime now available in AWS Lambda, https://aws.amazon.com/blogs/compute/node-js-8-10-runtime-now-available-in-aws-lambda/