Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
d8d0986
update dependencies and pin ruby version
autermann Oct 4, 2022
5a3daf1
store a link to datagarrison page in sensors
autermann Oct 4, 2022
0a0164c
fix ontology for minimum temperature values
autermann Oct 4, 2022
e49c50e
fix property access in envvironment_canada provider
autermann Oct 4, 2022
9b12c4e
fix datastream selection
autermann Oct 4, 2022
612f964
add cronjob
autermann Oct 20, 2022
768fdf7
add coordinats to environment canada stations
autermann Oct 20, 2022
e850e5d
add docker-compose.yaml
autermann Nov 15, 2022
12a83d8
add commnets and readme
autermann Jan 25, 2023
5f88828
use transload set metadata instead of sed
autermann Jan 25, 2023
a3cc73d
Update README.md
autermann Feb 6, 2023
95762b5
Fix whitespace errors
EHJ-52n Feb 10, 2023
d5c6e14
Sort env in crontab for better readability
EHJ-52n Feb 10, 2023
e3b1a65
add support for basic auth protected STA instances
EHJ-52n Feb 10, 2023
1bba540
ignore docker-compose during docker build
autermann Feb 15, 2023
0fc39f6
ontology updates
autermann Feb 15, 2023
735940a
split put metadata from cronjob
autermann Feb 15, 2023
212366b
Merge branch 'deployment/manitoba' of https://github.com/52North/data…
autermann Feb 15, 2023
b7b9fdc
adjust update-sensors to STA_URL ending with a /
autermann Feb 15, 2023
fc986d5
add missing sensor names/descriptions
autermann Feb 15, 2023
1aed23f
add .env
autermann Feb 15, 2023
5b9be13
add script to start get/put metadata services sequentially
autermann Feb 15, 2023
b113824
do not print credentials
autermann Feb 15, 2023
b871325
guard against empty SENSOR_ID
autermann Feb 15, 2023
26d1c29
Minor changes
EHJ-52n Feb 15, 2023
7373b12
Fix for https://github.com/docker/compose-cli/issues/1856
EHJ-52n Feb 15, 2023
be71fb1
Markdown lint
EHJ-52n Feb 15, 2023
ca4a8b2
do not include empty env variables in crontab
autermann Mar 6, 2023
c40e993
escape environment variables
autermann Mar 6, 2023
154a555
update ontology
autermann Mar 6, 2023
b6979c7
Merge branch 'uom' of https://github.com/52North/data-transloader int…
autermann Mar 6, 2023
b19d999
show more details on data garrison page
autermann Mar 10, 2023
d66625f
Enable fine grained scheduling of data upload jobs
EHJ-52n Apr 18, 2023
430c930
Fix metadata error between MMF stations
EHJ-52n Apr 18, 2023
30df4d1
Updates
EHJ-52n Apr 18, 2023
4f18592
Update definitions of observed properties
EHJ-52n May 22, 2023
ff4ef32
Update README.markdown
EHJ-52n Oct 29, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@
/spec
/test
/tmp
docker-compose.yaml
docker-compose.*.yaml
.dockerignore
2 changes: 2 additions & 0 deletions .env
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
STA_USER=
STA_PASSWORD=
6 changes: 3 additions & 3 deletions DOCKER.markdown
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,23 @@ There is currently no official Docker image on Docker Hub. Please build a Docker
## Building the Image

```terminal
$ docker build -t data-transloader:latest .
docker build -t data-transloader:latest .
```

## Running a Container

To test that the container runs:

```terminal
$ docker run -it --rm data-transloader:latest
docker run -it --rm data-transloader:latest
```

This should print the "help" information for the tool. We use `--rm` to delete the container after running, as it is an ephemeral one-time run.

To run the tool in actual usage, a Docker volume should be used to store the data and metadata. ([A bind mount][binds] to the host filesystem may also be used.)

```terminal
$ docker volume create etl-data
docker volume create etl-data
```

The tool can then be ran with the usual parameters; be sure to mount the volume so that the metadata is re-used.
Expand Down
7 changes: 6 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Ruby 2.6 is the recommended version
FROM ruby:2.6
FROM ruby:2.6.6

RUN set -ex \
&& apt-get update \
&& apt-get install -y --no-install-recommends curl jq \
&& rm -rf /var/lib/apt/lists/*

# Set the language to ensure encodings are handled properly.
# Otherwise the transloader tool will have errors parsing.
Expand Down
2 changes: 1 addition & 1 deletion Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ gem "deep_merge", "~> 1.2"
gem "nokogiri", "~> 1.10"
gem "parallel", "~> 1.19"
gem "pg", "~> 1.2"
gem "rdf-turtle", "~> 3.0"
gem "rdf-turtle", "~> 3.2"
gem "semantic_logger", "~> 4.5"
gem "spreadsheet", "~> 1.2"

Expand Down
33 changes: 16 additions & 17 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -14,27 +14,26 @@ GEM
zeitwerk (~> 2.3)
addressable (2.7.0)
public_suffix (>= 2.0.2, < 5.0)
amazing_print (1.3.0)
amazing_print (1.4.0)
coderay (1.1.3)
concurrent-ruby (1.1.8)
crack (0.4.5)
rexml
deep_merge (1.2.1)
diff-lcs (1.4.4)
ebnf (2.1.3)
amazing_print (~> 1.2)
ebnf (2.3.0)
amazing_print (~> 1.4)
htmlentities (~> 4.3)
rdf (~> 3.1)
rdf (~> 3.2)
scanf (~> 1.0)
sxp (~> 1.1)
unicode-types (~> 1.6)
hamster (3.0.0)
concurrent-ruby (~> 1.0)
sxp (~> 1.2)
unicode-types (~> 1.7)
hashdiff (1.0.1)
htmlentities (4.3.4)
i18n (1.8.10)
concurrent-ruby (~> 1.0)
link_header (0.0.8)
matrix (0.4.2)
method_source (1.0.0)
mini_portile2 (2.5.3)
minitest (5.14.4)
Expand All @@ -48,12 +47,11 @@ GEM
method_source (~> 1.0)
public_suffix (4.0.6)
racc (1.5.2)
rdf (3.1.14)
hamster (~> 3.0)
rdf (3.2.3)
link_header (~> 0.0, >= 0.0.8)
rdf-turtle (3.1.3)
ebnf (~> 2.1)
rdf (~> 3.1, >= 3.1.8)
rdf-turtle (3.2.0)
ebnf (~> 2.3)
rdf (~> 3.2)
rexml (3.2.5)
rspec (3.10.0)
rspec-core (~> 3.10.0)
Expand All @@ -75,12 +73,13 @@ GEM
concurrent-ruby (~> 1.0)
spreadsheet (1.2.9)
ruby-ole
sxp (1.1.0)
rdf (~> 3.1)
sxp (1.2.1)
matrix
rdf (~> 3.2)
timecop (0.9.4)
tzinfo (2.0.4)
concurrent-ruby (~> 1.0)
unicode-types (1.6.0)
unicode-types (1.7.0)
vcr (5.1.0)
webmock (3.13.0)
addressable (>= 2.3.6)
Expand All @@ -98,7 +97,7 @@ DEPENDENCIES
parallel (~> 1.19)
pg (~> 1.2)
pry (~> 0.12)
rdf-turtle (~> 3.0)
rdf-turtle (~> 3.2)
rspec (~> 3.9)
ruby-prof (~> 1.4)
semantic_logger (~> 4.5)
Expand Down
4 changes: 4 additions & 0 deletions README.markdown
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
**The latest state of the 52North UoM branch of the data transloader is migrated to https://cwincloud.cc.umanitoba.ca/ccadi/transloader (a private repository!) and will be maintained there.**
**Hence, this public fork of the data-transloader by 52N will be archived.**


# Arctic Sensor Web Data Transloader

[![Build Status](https://travis-ci.org/GeoSensorWebLab/data-transloader.svg?branch=master)](https://travis-ci.org/GeoSensorWebLab/data-transloader)
Expand Down
Loading