This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
uda:index [2017/11/19 14:33] Correia Lopes [3. Building the UDA] |
uda:index [2020/06/03 10:39] (current) Correia Lopes [UPORTO DATA ARCHIVE (UDA)] |
||
---|---|---|---|
Line 2: | Line 2: | ||
Perdigão' | Perdigão' | ||
+ | |||
+ | * **most recent info at the [[https:// | ||
+ | * [2020.05.23] TDS password removed | ||
+ | * [2020.05.23] cron jobs removed and the mirroring process stopped {{closing_history.tgz|DTU account closing status}} | ||
===== - Using the UDA ===== | ===== - Using the UDA ===== | ||
- | The UPORTO Data Archive (UDA) can be accessed using the THREDDS Data Server (TDS) at https:// | + | The UPORTO Data Archive (UDA) may be [[https:// |
+ | |||
+ | WindsP App users may [[https:// | ||
+ | |||
+ | {{uporto_data_archive_20171120_.pdf|}} | {{ : | ||
- | WindsP App users can also explore the UPORTO Data Archive (UDA) at https:// | ||
===== - UDA contents ===== | ===== - UDA contents ===== | ||
+ | |||
+ | ^ Export | ||
+ | | DLR | 1.2 TiB | 2.32553 GiB | | ||
+ | | DTU | 2.0 TiB | 0.000537872 GiB | | ||
+ | | INEGI | 301 MiB | 0 GiB | | ||
+ | | UCAR | 1.6 TiB | 6.24173 GiB | | ||
+ | | WINDFORS | ||
+ | |||
+ | Summary on 20-03-2018 00: | ||
+ | For more info see the [[http:// | ||
< | < | ||
Line 31: | Line 48: | ||
│ | │ | ||
│ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
│ | │ | ||
│ | │ | ||
│ | │ | ||
- | │ | + | │ |
- | │ | + | │ |
- | │ | + | │ |
- | │ | + | │ |
+ | │ | ||
+ | │ | ||
+ | │ | ||
├── dtu | ├── dtu | ||
│ | │ | ||
Line 44: | Line 69: | ||
│ | │ | ||
│ | │ | ||
+ | │ | ||
│ | │ | ||
│ | │ | ||
Line 53: | Line 79: | ||
│ | │ | ||
│ | │ | ||
- | │ └── PointCloud | + | │ ├── PerdigaoTurbineTopView.pdf |
+ | │ ├── PointCloud | ||
+ | │ | ||
├── ucar | ├── ucar | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
│ | │ | ||
│ | │ | ||
│ | │ | ||
- | │ | + | │ |
- | │ | + | │ │ |
- | │ | + | │ |
- | │ | + | │ |
- | │ | + | │ |
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
└── windfors | └── windfors | ||
├── 2017 | ├── 2017 | ||
Line 70: | Line 114: | ||
└── cross | └── cross | ||
└── 2017 | └── 2017 | ||
+ | | ||
+ | 441 directories, | ||
</ | </ | ||
===== - Building the UDA ===== | ===== - Building the UDA ===== | ||
- | Each institution, | + | Each institution, |
Available exports: | Available exports: | ||
Line 90: | Line 136: | ||
===== - Upload DTU data ===== | ===== - Upload DTU data ===== | ||
- | UPORTO uses the UDA export '' | + | UPORTO |
+ | |||
+ | First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job: | ||
+ | ''/ | ||
+ | |||
+ | Later the '' | ||
+ | |||
+ | <code bash crons> | ||
+ | # DTU data sync to UDA, At minute 31 past every 4th hour | ||
+ | #31 */4 * * * / | ||
+ | 31 */4 * * * / | ||
+ | </ | ||
< | < | ||
- | ssh nejoco@login.neweuropeanwindatlas.eu | + | cat ~nejoco/ |
+ | archive/ | ||
+ | data/ | ||
</ | </ | ||
- | First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using the cron command: | + | ===== - Upload UCAR data ===== |
- | '' | + | |
- | Later the '' | + | UCAR uses the UDA export |
+ | ===== - Upload DLR data ===== | ||
+ | |||
+ | DLR uses the UDA export '' | ||
+ | |||
+ | ===== - Upload INEGI data ===== | ||
+ | |||
+ | INEGI uses the UDA export '' | ||
+ | |||
+ | ===== - Upload WindsForS data ===== | ||
+ | |||
+ | WindsForS uses the UDA export '' | ||
+ | |||
+ | ===== - Mirror UCAR ftp site ===== | ||
+ | |||
+ | Preliminary data at the ftp site (ARL, Notre dame, ...) uploaded with: | ||
< | < | ||
- | cat ~nejoco/sync-excluded-list | + | #! /bin/sh |
- | archive/ | + | dir=arl |
- | data/DLT_WindScanner/ | + | source=ftp:// |
+ | destination=/ | ||
+ | nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 & | ||
</ | </ | ||
+ | |||
+ | Afterwards it is verified by running in the ftp site: | ||
+ | < | ||
+ | #! /bin/sh | ||
+ | dir=arl | ||
+ | cd < | ||
+ | export RSYNC_PASSWORD=t****YLa**** | ||
+ | archive=ucar@windsptds.fe.up.pt:: | ||
+ | rsync -avz --delete --dry-run $dir $archive | ||
+ | </ | ||
+ | |||
+ | ===== - Mirror UDA to DTU ===== | ||
+ | |||
+ | The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: '' | ||
<code bash crons> | <code bash crons> | ||
- | # DTU data sync to UDA, "At minute 31 past every 4th hour." | + | # UDA archive to DTU, At midnight every day |
- | 31 */4 * * * / | + | |
- | # | + | |
- | # UDA archive to DTU, "At midnight every day." | + | |
0 0 * * * / | 0 0 * * * / | ||
</ | </ | ||
Line 129: | Line 215: | ||
# the UDA readonly password | # the UDA readonly password | ||
- | export RSYNC_PASSWORD=t286rFTrktHn | + | export RSYNC_PASSWORD=-password- |
# catalogues to sync | # catalogues to sync | ||
Line 162: | Line 248: | ||
</ | </ | ||
- | ===== - Collecting UCAR data ===== | + | The DTU NEWA directory / |
- | + | ||
- | UCAR uses the UDA export '' | + | |
- | + | ||
- | DLR | + | |
- | + | ||
- | INEGI | + | |
- | + | ||
- | WindsForS | + | |
- | + | ||
- | ===== - Mirror UDA to DTU ===== | + | |
- | + | ||
- | The mirror of UPORTO Data Archive (UDA) to the NEWA storage is now complete. | + | |
- | + | ||
- | To manually update the archive, you may run the shell script (see the attached log of today' | + | |
- | ~nejoco/ | + | |
- | + | ||
- | We are automatically syncing every 24 hours using the cron command: | + | |
- | '' | + | |
- | + | ||
- | + | ||
- | The DTU NEWA directory / | + | |
< | < | ||
Line 213: | Line 278: | ||
===== - Current status ===== | ===== - Current status ===== | ||
- | ^ Data | + | <note warning> |
- | | **ALR** | + | |
- | | **CU** | + | {{ : |
- | | **DLR** | + | {{ : |
- | | **DTU** | + | |
- | | **ENERCON** | + | |
- | | **IPMA** | + | |
- | | **ISFS** | + | |
- | | **ISS** | + | |
- | | **Leosphere** | + | |
- | | **NCAS** | + | |
- | | **ND** | + | |
- | | **OU** | + | |
- | | **WindForS** | + | |
--- // | --- // | ||