This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
uda:index [2017/11/19 00:57] Correia Lopes [5. Current status] |
uda:index [2020/06/03 10:39] (current) Correia Lopes [UPORTO DATA ARCHIVE (UDA)] |
||
---|---|---|---|
Line 2: | Line 2: | ||
Perdigão' | Perdigão' | ||
+ | |||
+ | * **most recent info at the [[https:// | ||
+ | * [2020.05.23] TDS password removed | ||
+ | * [2020.05.23] cron jobs removed and the mirroring process stopped {{closing_history.tgz|DTU account closing status}} | ||
===== - Using the UDA ===== | ===== - Using the UDA ===== | ||
- | The UDA can be accessed using the TDS at | + | The UPORTO Data Archive (UDA) may be [[https:// |
- | The UDA can be explored using the WindsP App at https:// | + | WindsP App users may [[https:// |
- | As a WindsP user you get credentials to the Web App. | + | {{uporto_data_archive_20171120_.pdf|}} | {{ : |
- | When you request access to data that is in the Data Archive (outside WindsP and embargoed during 12 months) you must provide the TDS credentials: | ||
- | perdigao / Bl@yer! (the same as in the UCAR ftp site). | ||
- | Each institution also owns credentials to their catalogue in our Data Server, to upload and maintain their data. | + | ===== - UDA contents ===== |
+ | ^ Export | ||
+ | | DLR | 1.2 TiB | 2.32553 GiB | | ||
+ | | DTU | 2.0 TiB | 0.000537872 GiB | | ||
+ | | INEGI | 301 MiB | 0 GiB | | ||
+ | | UCAR | 1.6 TiB | 6.24173 GiB | | ||
+ | | WINDFORS | ||
- | ===== - Collecting DTU data ===== | + | Summary on 20-03-2018 00:00. |
+ | For more info see the [[http:// | ||
- | UPORTO uses the UDA export '' | + | < |
+ | perdigao@windsptds:/ | ||
+ | data/ | ||
+ | ├── dlr | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | ├── dtu | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | ├── inegi | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | ├── ucar | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | └── windfors | ||
+ | ├── 2017 | ||
+ | │ | ||
+ | │ | ||
+ | │ | ||
+ | └── cross | ||
+ | └── 2017 | ||
+ | |||
+ | 441 directories, | ||
+ | </ | ||
+ | |||
+ | ===== - Building the UDA ===== | ||
+ | |||
+ | Each institution, | ||
+ | |||
+ | Available exports: | ||
+ | < | ||
+ | nejoco@VIND-pNEWA04: | ||
+ | test | ||
+ | archive | ||
+ | ucar | ||
+ | dtu RSYNC DTU FILES | ||
+ | inegi RSYNC INEGI FILES | ||
+ | dlr RSYNC DLR FILES | ||
+ | windfors | ||
+ | </ | ||
+ | |||
+ | ===== - Upload DTU data ===== | ||
+ | |||
+ | UPORTO | ||
+ | |||
+ | First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job: | ||
+ | ''/ | ||
+ | |||
+ | Later the '' | ||
+ | |||
+ | <code bash crons> | ||
+ | # DTU data sync to UDA, At minute 31 past every 4th hour | ||
+ | #31 */4 * * * / | ||
+ | 31 */4 * * * / | ||
+ | </ | ||
< | < | ||
- | ssh nejoco@login.neweuropeanwindatlas.eu | + | cat ~nejoco/ |
+ | archive/ | ||
+ | data/ | ||
</ | </ | ||
- | We are automatically syncing every 4 hours the DTU data directory using the cron command: | + | ===== - Upload UCAR data ===== |
- | '' | + | |
- | '' | + | UCAR uses the UDA export |
+ | |||
+ | ===== - Upload DLR data ===== | ||
+ | |||
+ | DLR uses the UDA export '' | ||
+ | |||
+ | ===== - Upload INEGI data ===== | ||
+ | |||
+ | INEGI uses the UDA export | ||
+ | |||
+ | ===== - Upload WindsForS data ===== | ||
+ | |||
+ | WindsForS uses the UDA export '' | ||
+ | |||
+ | ===== - Mirror UCAR ftp site ===== | ||
+ | |||
+ | Preliminary data at the ftp site (ARL, Notre dame, ...) uploaded with: | ||
+ | < | ||
+ | #! /bin/sh | ||
+ | dir=arl | ||
+ | source=ftp:// | ||
+ | destination=/ | ||
+ | nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 & | ||
+ | </ | ||
+ | |||
+ | Afterwards it is verified by running | ||
+ | < | ||
+ | #! /bin/sh | ||
+ | dir=arl | ||
+ | cd < | ||
+ | export RSYNC_PASSWORD=t****YLa**** | ||
+ | archive=ucar@windsptds.fe.up.pt:: | ||
+ | rsync -avz --delete --dry-run $dir $archive | ||
+ | </ | ||
+ | |||
+ | ===== - Mirror UDA to DTU ===== | ||
+ | |||
+ | The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: '' | ||
<code bash crons> | <code bash crons> | ||
- | # DTU data sync to UDA, "At minute 31 past every 4th hour." | + | # UDA archive to DTU, At midnight every day |
- | #31 */4 * * * / | + | |
- | 31 */4 * * * / | + | |
- | # | + | |
- | # UDA archive to DTU, "At midnight every day." | + | |
0 0 * * * / | 0 0 * * * / | ||
</ | </ | ||
- | ===== - Collecting UCAR data ===== | + | <code bash sync-uda.sh> |
+ | #! /bin/sh | ||
- | UCAR uses the UDA export '' | + | # the Perdigao root at NEWA storage |
+ | perdigao=/ | ||
- | DLR | + | # the archive root |
+ | archive=$perdigao/ | ||
- | INEGI | + | # the actual size of the archive |
+ | echo "Total du of $archive:" | ||
+ | du -ks $archive | ||
- | WindsForS | + | # the UDA readonly password |
+ | export RSYNC_PASSWORD=-password- | ||
- | ===== - Mirror UDA to DTU ===== | + | # catalogues |
+ | CATALOGS="dlr inegi ucar windfors" | ||
- | The mirror | + | for c in $CATALOGS; do |
+ | # mirror | ||
+ | echo; echo "$(tr [a-z] [A-Z] <<< | ||
+ | # | ||
+ | cmd=" | ||
+ | echo " | ||
+ | # do it | ||
+ | $cmd | ||
+ | done | ||
- | To manually update the archive, you may run the shell script (see the attached log of today' | + | # catalog structure |
- | ~nejoco/ | + | echo |
+ | tree -L 2 $archive | ||
- | We are automatically syncing every 24 hours using the cron command: | + | # total space usage for each archive |
- | '' | + | echo |
+ | du -khs $archive/* | ||
+ | # the final size of the archive | ||
+ | echo | ||
+ | echo "Total du of $archive:" | ||
+ | du -ks $archive | ||
- | The DTU NEWA directory / | + | # the end |
+ | echo | ||
+ | echo " | ||
+ | </ | ||
+ | |||
+ | The DTU NEWA directory / | ||
< | < | ||
Line 90: | Line 278: | ||
===== - Current status ===== | ===== - Current status ===== | ||
- | ^ Data | + | <note warning> |
- | | **ALR** | + | |
- | | **CU** | + | {{ : |
- | | **DLR** | + | {{ : |
- | | **DTU** | + | |
- | | **ENERCON** | + | |
- | | **IPMA** | + | |
- | | **ISFS** | + | |
- | | **ISS** | + | |
- | | **Leosphere** | + | |
- | | **NCAS** | + | |
- | | **ND** | + | |
- | | **OU** | + | |
- | | **WindForS** | + | |
--- // | --- // | ||