The longest official place name in Australia

Guestbook

Mamungkukumpurangkuntjunya Hill, is a hill in South Australia. The name means “where the devil urinates” in the regional Pitjantjatjara language. The name is the longest official place name in Australia. The hill is located approximately 108.8 km west north-west of Marla.

Mamungkukumpurangkuntjunya Hill

House Prices Down Under

Guestbook

A look at historical house prices in Sydney and Melbourne from 1880 to 2010.

Historical house prices in Sydney and Melbourne from 1880 to 2010

Melbourne Median vs Average Wage 1965 - 2010.

Melbourne Median vs Average Wage 1965 - 2010

It must be remembered that house prices are determined by the demand and supply of credit, not the demand for and supply of housing.

房价不与国民收入，社会福利放在一起进行讨论，无异于光天化日里耍流氓。

AlphaGo movie

Guestbook

AlphaGo is an intriguing movie, full of striking imagery.

Proud of witnessing DeepMind and Google are making history …

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

AlphaGo movie

RESTful calls to create Index and Mappings in ElasticSearch

Guestbook

There is a simple way, using RESTful client to create Index and Mappings in ElasticSearch, for example, with ElasticSearch Head plugin.

Create Index in ElasticSearch with Settings JSON file. URL: http://localhost:9200/orders/, Method: PUT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "analysis": {
        "filter": {
          "partial_matching_filter": {
            "type": "edge_ngram",
            "min_gram": 1,
            "max_gram": 40
          }
        },
        "normalizer": {
           "lowercase_normalizer": {
             "type": "custom",
             "filter": ["lowercase"]
           }
         },        
        "analyzer": {
          "partial_matcher": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": [
              "lowercase",
              "partial_matching_filter"
            ]
          }
        }
      }
    }
  }
}

Create Mappings in ElasticSearch with Mappings JSON file. URL: http://localhost:9200/orders/_mapping/order, Method: PUT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
{
  "dynamic": "false",
  "_all": { "enabled": false},
  "properties": {
    "customer_id": {
      "type": "keyword"
    },
    "encryption_key_id": {
      "type": "keyword",
      "index": "false"
    },
    "metadata_tags": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "category": {
      "type": "keyword"
    },
    "order_creation_date": {
      "type": "date",
      "format": "date_optional_time"
    },
    "order_reference": {
      "type": "text"
    },
    "order_id": {
      "type": "keyword",
      "normalizer": "lowercase_normalizer"
    },
    "to_name": {
      "type": "text",
      "analyzer": "partial_matcher",
      "search_analyzer": "standard"
    },
    "to_suburb": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "to_state": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "to_postcode": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "to_email": {
      "type": "text",
      "analyzer": "whitespace"
    }
  }
}

To add a new fields to ElasticSearch, while dynamic mapping is off:

1
2
3
{
  "dynamic": "false"
}

Run with additional fields JSON file, with URL: http://localhost:9200/orders/_mapping/order, Method: PUT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
{
  "properties": {
    "from_name": {
      "type": "text",
      "analyzer": "partial_matcher",
      "search_analyzer": "standard"
    },
    "from_business_name": {
      "type": "text",
      "analyzer": "partial_matcher",
      "search_analyzer": "standard"
    },
    "from_suburb": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "from_state": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "from_postcode": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "from_country": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "from_email": {
      "type": "text",
      "analyzer": "whitespace"
    },
    "from_phone": {
      "type": "text",
      "analyzer": "whitespace"
    }
  }
}

An ElasticSearch query example, with URL: http://localhost:9200/orders/_search, Method: POST

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
{
 "query": {
   "bool": {
     "must": [
       {"match": {"customer_id": "1234567890"} },
       {"simple_query_string": {
            "query": "unlabelled+unmanifested",
            "fields": ["metadata_tags"]
       } },
       {
         "bool": {
           "should": [
             {
               "multi_match": {
                 "query": "TerrenceMiao",
                 "fields": [
                   "name", "business_name", "order_id", "shipment_id", "sender_references", "article_ids"
                 ]
               }
             },
             {"match": {"phone": "abracadbravvv5/vv70qBu+/ve+/ve6EuA7vv70g77+9Su+/ve+/vX5y77+95JuV"} },
             {
               "multi_match": {
                 "query": "gdgdggddvvv75/vv70qBu+/ve+/ve6EuA7vv70g77+9Su+/ve+/vX5y77+95JuV",
                 "fields": [
                   "suburb", "state", "country", "postcode", "email"
                 ]
               }
             }
           ]
         }
       },
       {"simple_query_string": {
         "query": "DESPATCH",
         "fields": ["movement_type"]
       } }
     ]
   }
 },
 "sort": [
   {"order_creation_date": {"order": "desc", "unmapped_type": "date"} },
   {"shipment_creation_date": {"order": "desc"} }
 ],
 "from": "0","size": "1"
}

An example query with date/time type in a range:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "query": {
    "range": {
      "shipment_creation_date": {
        "gte": "2018-04-19T15:30:00",
        "lte": "now",
        "time_zone": "+10:00"

      }
    }
  },
  "from": 0,
  "size": 10,
  "sort": [],
  "aggs": {}
}

AWS access key id and secret access key

Guestbook

There are some secret of AWS credentials, i.e., its Access Key ID and Secret Access Key, which let you connect AWS services withouth authentication.

In a quintessential project environment, you have dev, test, prod environment setup in AWS. Your local AWS_PROFILE is something like this so you can switch to different targeting end services while run your application on localhost:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
𝜆 cat ~/.aws/credentials
[default]
aws_access_key_id=dev-aws_access_key_id
aws_secret_access_key=dev-aws_secret_access_key

[dev]
aws_access_key_id=dev-aws_access_key_id
aws_secret_access_key=dev-aws_secret_access_key

[test]
aws_access_key_id=test-aws_access_key_id
aws_secret_access_key=test-aws_secret_access_key

[prod]
aws_access_key_id=prod-aws_access_key_id
aws_secret_access_key=prod-aws_secret_access_key

When test Jest client in Test environment, a AWS ElasticSearch client library, error thrown:

1
2
3
2018-02-27 15:18:19 INFO  org.paradise.search.routes.ElasticSearchRoute  - Body: io.searchbox.core.Index@4804b850[uri=orders/order/4SMK1UmbtqIAAAFhmYAFt9V7,method=PUT] message: searchIndexRoute - updating search index
2018-02-27 15:18:19 INFO  org.apache.camel.processor.interceptor.Tracer  - >>> (searchIndexRoute) org.paradise.search.routes.ElasticSearchRoute$$Lambda$137/819330075@119b837 --> org.paradise.search.routes.ElasticSearchRoute$$Lambda$138/2035788375@2acacdf <<< Pattern:InOnly, BodyType:io.searchbox.core.Index
2018-02-27 15:18:19 ERROR org.paradise.search.routes.ElasticSearchRoute  - Error while updating search index. 403 Forbidden {"Message":"User: arn:aws:iam::123456789012:user/svcbamboo is not authorized to perform: es:ESHttpPut on resource: paradise-esv5-test-esd"} at 'orders/order/4SMK1UmbtqIAAAFhmYAFt9V7'

It turns out that WRONG Dev AWS_PROFILE applied in Test environment.

Replacing “aws_access_key_id” and “aws_secret_access_key” from the default profile with “aws_access_key_id” and “aws_secret_access_key” from test profile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
𝜆 cat ~/.aws/credentials
[default]
aws_access_key_id=test-aws_access_key_id
aws_secret_access_key=test-aws_secret_access_key

[dev]
aws_access_key_id=dev-aws_access_key_id
aws_secret_access_key=dev-aws_secret_access_key

[test]
aws_access_key_id=test-aws_access_key_id
aws_secret_access_key=test-aws_secret_access_key

[prod]
aws_access_key_id=prod-aws_access_key_id
aws_secret_access_key=prod-aws_secret_access_key

Rerun the test and in log:

1
2
3
2018-02-27 15:23:54 INFO  org.paradise.search.routes.ElasticSearchRoute  - Body: io.searchbox.core.Index@62b18125[uri=orders/order/RVwK1UIBXmoAAAFhdJkFo9WA,method=PUT] message: searchIndexRoute - updating search index
2018-02-27 15:23:54 INFO  org.apache.camel.processor.interceptor.Tracer  - >>> (searchIndexRoute) org.paradise.search.routes.ElasticSearchRoute$$Lambda$137/1716909005@a809a62 --> org.paradise.search.routes.ElasticSearchRoute$$Lambda$138/612681832@4b2713b1 <<< Pattern:InOnly, BodyType:io.searchbox.core.Index
2018-02-27 15:23:55 INFO  org.paradise.search.routes.ElasticSearchRoute  - Finished updating search index at 'orders/order/RVwK1UIBXmoAAAFhdJkFo9WA'.

AWS ElasticSearch and Kibana proxy setup

Guestbook

There are several ways to access Amazon AWS ElasticSearch and Kibana services, which are HTTP based, without inject into HTTP request headers with authentication key …

AWS ES Proxy

Install a proxy application - AWS ES/Kibana Proxy. Download it from https://www.npmjs.com/package/aws-es-kibana and install:

1
$ npm install -g aws-es-kibana

Run AWS ES/Kibana Proxy then connect to ElasticSearch and Kibana services on AWS:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ cat ~/.aws/config
[default]
output = json
region = ap-southeast-2

[profile sandy-test]
output = json
region = ap-southeast-2

$ cat ~/.aws/credentials
[default]
aws_access_key_id=Academy
aws_secret_access_key=Dingly Ding

[ap-test]
aws_access_key_id=Abracadbra
aws_secret_access_key=Somewhere Over The Rainbow

$ AWS_PROFILE=ap-test aws-es-kibana search-paradise-esv5-test-01-esd-blah23dlaoed81nz890adle4.ap-southeast-2.es.amazonaws.com
__________       _________    _________________    ________                            ______
___    |_ |     / /_  ___/    ___  ____/_  ___/    ___  __ \________________  ______  ____  /
__  /| |_ | /| / /_____ \     __  __/  _____ \     __  /_/ /_  ___/  __ \_  |/_/_  / / /_  /
_  ___ |_ |/ |/ / ____/ /     _  /___  ____/ /     _  ____/_  /   / /_/ /_>  < _  /_/ / /_/
/_/  |_|___/|__/  /____/      /_____/  /____/      /_/     /_/    \____//_/|_| _\__, / (_)
                                                                               /____/
AWS ES cluster available at http://127.0.0.1:9200
Kibana available at http://127.0.0.1:9200/_plugin/kibana/

With Fish Shell:

1
2
3
4
5
6
7
8
9
𝜆 env AWS_PROFILE=ap-test aws-es-kibana search-paradise-esv5-test-01-esd-blah23dlaoed81nz890adle4.ap-southeast-2.es.amazonaws.com
__________       _________    _________________    ________                            ______
___    |_ |     / /_  ___/    ___  ____/_  ___/    ___  __ \________________  ______  ____  /
__  /| |_ | /| / /_____ \     __  __/  _____ \     __  /_/ /_  ___/  __ \_  |/_/_  / / /_  /
_  ___ |_ |/ |/ / ____/ /     _  /___  ____/ /     _  ____/_  /   / /_/ /_>  < _  /_/ / /_/
/_/  |_|___/|__/  /____/      /_____/  /____/      /_/     /_/    \____//_/|_| _\__, / (_)
                                                                               /____/
AWS ES cluster available at http://127.0.0.1:9200
Kibana available at http://127.0.0.1:9200/_plugin/kibana/

SSH Tunnel

Set up SSH Tunnel for AWS ElasticSearch https://search-paradise-esv5-test-01-esd-blah23dlaoed81nz890adle4.ap-southeast-2.es.amazonaws.com from 443 port to localhost 9200:

1
𝜆 ssh -L 9200:search-paradise-esv5-test-01-esd-blah23dlaoed81nz890adle4.ap-southeast-2.es.amazonaws.com:443 -l ec2-user aws-jump-box

Then access AWS ElasticSearch at: https://localhost:9200, AWS Kibana at: https://localhost:9200/_plugin/kibana

ElasticSearch Head

With plugin ElasticSearch Head, to query ElasticSearch, using URL and index “orders-search-box” e.g. https://localhost:9200/orders-search-test/, and context path “_search”