GitHub storage for online resume. Also to demonstrate some simple CD from here to webpage.
- Ensure you have an SSH key in GitHub
- Clone the repo
- Make sure that you have set the repo up as SSH:
- git config --local remote.origin.url git@github.com:
- (Anything else to do? ssh config?)
- Make changes
- git push
- Site will be automatically deployed
Yes you can just load index.html into your browser, but it is better to try and mimic it being served from a webserver instead. Do this with the following command:
docker run --name raf-resume -v ${PWD}:/usr/share/nginx/html:ro -p 8000:80 nginx:1.23-alpine
Note that the docker container does not seem to respond to Ctrl-C inputs. So you can run it with a -d flag (detached) and then just stop it with docker container stop...
There is a dockerfile which creates a simple docker image with mkdocs installed. Once this is created (say with the name: mkdimage), the blog can be built in docker as follows. First change to the mkdocs folder (with the YML file in it). Then run:
docker run --rm -v ${PWD}:/opt/project --entrypoint mkdocs mkdimage build
This will output the resulting build into a directory called 'site', which you can then use to run the previously mentioned nginx docker image.
Template: https://html5up.net/ Content: Mine Hosting: AWS S3
The website is designed to be hosted on S3, as designed and deployed by myself. This is in order to serve a) as a living exercise for others and b) as a demonstration of my AWS knowledge.
The infrastructure stack is expressed as Cloudformation in the aws-cf folder. The current stack name is: raf-tech-website-stack. To create it:
aws --profile <your cred profile> --region <target region> cloudformation create-stack --stack-name raf-tech-website-stack --template-body file://aws-cf/raf-tech-website.yaml --parameters file://filename.json
Note that you need to provide:
- WebsiteName
- IAM Roles and Policies
- S3
- CodePipeline, CodeBuild
- Lambda
- Dynamo
Set many attributes - best to do in a file with:
dynamodb put-item --table-name PageTrackTable --item file://totals.json
Increment an attribute:
dynamodb update-item --table-name PageTrackTable --key '{ "UserPages": {"S": "Richard"}, "SortKey": {"S": "PAGES"} }' --update-expression "ADD #page :incr" --expression-attribute-names '{ "#page": "blog/articles/2021-11-16-understanding-scrum/"}' --expression-attribute-values '{":incr": {"N": "1" }}' --return-values UPDATED_NEW
Add to a list and create new list if not present:
dynamodb update-item --table-name PageTrackTable --update-expression "SET #key = list_append(if_not_exists(#key, :empty), :val)" --expression-attribute-names '{"#key": "blog/books/2022-12-book-review-2022/"}' --expression-attribute-values '{ ":val": { "L": [...] }, ":empty": { "L": [] } }' --key '{"UserPages": {"S": "Richard"}, "SortKey": {"S": "VISITS"}}'
Remove an attribute:
dynamodb update-item --table-name PageTrackTable --key '{ "UserPages": {"S": "Richard"}, "SortKey": {"S": "PAGES"} }' --update-expression "REMOVE #attr" --expression-attribute-names '{ "#attr": "<attrname>"}'
Retrieve an attribute:
dynamodb get-item --table-name PageTrackTable --key '{ "UserPages": { "S": "Richard"}, "SortKey": {"S": "VISITS"}}' --projection-expression "#page" --expression-attribute-names '{ "#page": "blog/articles/2023-02-aws-cli-advanced-filtering/" }'
Get all records partially matching a key
dynamodb query --table-name PageTrackTable --key-condition-expression "UserPages = :pk AND begins_with(SortKey, :sk)" --expression-attribute-values '{ ":pk": { "S": "Richard#INDEX" }, ":sk": { "S": "Richard#blog/articles" } }' --return-consumed-capacity TOTAL
Get count of records matching a key
dynamodb query --table-name PageTrackTable --key-condition-expression "UserPages = :pk" --expression-attribute-values '{ ":pk": { "S": "Richard#blog/articles/2022-10-On-Technical-Debt/" } }' --select COUNT --return-consumed-capacity TOTAL
Get a single item match
dynamodb get-item --table-name PageTrackTable --key '{ "UserPages": { "S": "Richard#blog/articles/2022-10-On-Technical-Debt/" }, "SortKey": {"S": "VISITS#..."} }'
Get all the stored query names from athena
aws athena list-named-queries --query "NamedQueryIds | join(' ', @)" | tr -d '"' | xargs aws athena batch-get-named-query --query "NamedQueries[].[Name,Description || 'None'] | [*].join(':', @)" --named-query-ids | sed 's/:/:\t/'
Note the sed command is used to insert tabs for readability. I'm not sure how to do this in JMES.
- List all sort key results from dynamo response:
jq '.Items[].SortKey.S' - Convert DynamoDB list into regular array:
jq '[.Item.L[] | .N]' - Convert DynamoDB map of lists into arrays:
jq '.Item | with_entries(.value |= [.L[]?.N]) - Replace a list with its length:
jq '.Item.List.L |= length' - Replace all lists with lengths in an array:
jq '.Items |= [.[] | .List.L |= length]
In order to use the JQ scripts, use the command of the form:
jq -r -f <script-file>
-r will use 'raw' format which is compatible with piping the command to aws.
The form for launching a query from athena is:
aws athena list-named-queries --query "NamedQueryIds | join(' ', @)" | tr -d '"' \
| xargs aws athena batch-get-named-query --query "NamedQueries[?Name=='<name_of_query>'].QueryString | [0]" --named-query-ids \
| sed 's/\\n/ /g' \
| xargs aws athena start-query-execution --work-group primary --query-execution-context Catalog=AwsDataCatalog,Database=ddd_s3_access_logs_db --query-string
This can probably be put into a script. The sed command replaces the newlines with spaces.
Retrieving and processing results:
aws athena get-query-results --query-execution-id 123-ABC | jq -r -f <script-file> \
| xargs -n 1 -I data aws dynamodb batch-write-item --request-items 'data'
The -n 1 is only needed if there is a lot of data grouped into lines, for example the daily page visits which groups each page into a new batch request line. Queries which don't do this can be passed in as a single request (that is, up to 25 requests in the batch request)