Perdigão's datasets repository using the THREDDS Data Server (TDS)
The UPORTO Data Archive (UDA) may be accessed using the THREDDS Data Server (TDS) by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!).
WindsP App users may explore the UPORTO Data Archive (UDA) but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months).
uporto_data_archive_20171120_.pdf | uporto_data_archive_2020-05-23.pdf
Export | Size | Last 24 hours |
---|---|---|
DLR | 1.2 TiB | 2.32553 GiB |
DTU | 2.0 TiB | 0.000537872 GiB |
INEGI | 301 MiB | 0 GiB |
UCAR | 1.6 TiB | 6.24173 GiB |
WINDFORS | 3.2 GiB | 0 GiB |
Summary on 20-03-2018 00:00. For more info see the datalogs details.
perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao data/perdigao ├── dlr │ ├── HATPRO_level-1 │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── HATPRO_level-2 │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── HATPRO_surface-met │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── mcs_data │ │ ├── 20170430095732 │ │ ... │ ├── netcdf_lidar │ │ ├── DLR85 │ │ ├── DLR86 │ │ ├── DLR89 │ │ └── readme.tx │ │── raw_data │ │ ├── DLR85 │ │ ├── DLR86 │ │ └── DLR89 │ └── sound │ ├── mic1 │ ├── mic2 │ ├── mic3 │ ├── mic4 │ ├── mic5 │ └── microphone_position.txt ├── dtu │ ├── data │ │ ├── DTU_Leica_Scanning │ │ ├── DTU_Mast_Data │ │ └── DTU_WindScanner │ ├── docs │ ├── landscape │ ├── photos │ └── plots │ └── DTU_WindScanner ├── inegi │ ├── EnerconWindTurbine │ ├── LeosphereWindcube │ │ └── 01_RawData │ └── LidarAerialSurvey_RawData │ ├── Images │ ├── PerdigaoTurbineTopView.pdf │ ├── PointCloud │ └── Portugal Laserscanning Report.pdf ├── ucar │ ├── arl │ │ ├── ARL_Scanning_Lidar_George_Site │ │ ├── ARL_Scanning_Lidar_Lionstail_Site │ │ └── ARL_Scintillometer │ ├── colorado │ │ └── CU_Lidar │ ├── eol │ │ └── WV-DIAL │ ├── isfs │ │ ├── hr_noqc_geo │ │ └── noqc_geo_notiltcor │ ├── ncas │ │ └── NCAS_profiler │ ├── notredame │ │ ├── UND_Ceilometer │ │ ├── UND_Radiosonde │ │ ├── UND_Scanning_Lidar_Lionshead_Site │ │ ├── UND_Scanning_Lidar_MI6_Site │ │ ├── UND_Scanning_Lidar_Orange_Site │ │ └── UND_SODAR_RASS │ └── oklahoma │ ├── CLAMPS_AERI │ ├── CLAMPS_MWR │ └── CLAMPS_Scanning_Lidar └── windfors ├── 2017 │ ├── 201704 │ ├── 201705 │ └── 201706 └── cross └── 2017 441 directories, 8 files
Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in their catalogue in UDA (rsync exports).
Available exports:
nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt test RSYNC test archive RSYNC UDA FILES (read only) ucar RSYNC UCAR FILES dtu RSYNC DTU FILES inegi RSYNC INEGI FILES dlr RSYNC DLR FILES windfors RSYNC WindForS FILES
UPORTO (as nejoco@login.neweuropeanwindatlas.eu
) uses the UDA export dtu@windsptds.fe.up.pt::dtu
to sync data collected by DTU.
First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job:
/usr/bin/rsync -az –delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu
.
Later the –delete
option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.
# DTU data sync to UDA, At minute 31 past every 4th hour #31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1 31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
cat ~nejoco/sync-exclude-list archive/ data/DLR_WindScanner/
UCAR uses the UDA export ucar@windsptds.fe.up.pt::ucar
to copy NCAR/EOL ISFS data.
DLR uses the UDA export dlr@windsptds.fe.up.pt::dlr
to maintain the DLR data.
INEGI uses the UDA export inegi@windsptds.fe.up.pt::inegi
to maintain the ENERCON data and “Lidar Aerial Survey Data”.
WindsForS uses the UDA export windfors@windsptds.fe.up.pt::windfors
to maintain the WindForS data.
Preliminary data at the ftp site (ARL, Notre dame, …) uploaded with:
#! /bin/sh dir=arl source=ftp://ftp.eol.ucar.edu/pub/data/incoming/perdigao/uda/$dir destination=/data/perdigao/ucar nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 &
Afterwards it is verified by running in the ftp site:
#! /bin/sh dir=arl cd <ftp-root>/incoming/perdigao/uda export RSYNC_PASSWORD=t****YLa**** archive=ucar@windsptds.fe.up.pt::ucar/$dir rsync -avz --delete --dry-run $dir $archive
The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: uda@windsptds.fe.up.pt::archive/
, using a cron job.
# UDA archive to DTU, At midnight every day 0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1
#! /bin/sh # the Perdigao root at NEWA storage perdigao=/newa/WP2/PERDIGAO # the archive root archive=$perdigao/archive # the actual size of the archive echo "Total du of $archive:" du -ks $archive # the UDA readonly password export RSYNC_PASSWORD=-password- # catalogues to sync CATALOGS="dlr inegi ucar windfors" for c in $CATALOGS; do # mirror catalog from the version at UDA (UPORTO) echo; echo "$(tr [a-z] [A-Z] <<< "$c"):" #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/" cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/" echo "$cmd..." # do it $cmd done # catalog structure echo tree -L 2 $archive # total space usage for each archive echo du -khs $archive/* # the final size of the archive echo echo "Total du of $archive:" du -ks $archive # the end echo echo "Done."
The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 2.0 TiB of storage).
/newa/WP2/PERDIGAO/archive ├── dlr │ ├── HATPRO_level-1 │ ├── HATPRO_level-2 │ ├── HATPRO_surface-met │ ├── mcs_data │ ├── netcdf_lidar │ └── raw_data ├── dtu │ ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning │ ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data │ └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner ├── inegi │ ├── EnerconWindTurbine │ ├── LeosphereWindcube │ └── LidarAerialSurvey_RawData ├── ucar │ ├── isfs │ ├── iss │ └── ncas └── windfors ├── 2017 └── cross
<note warning>There is a collaborative version of this table being updated at Google docs.</note>
snap at 22/12/2017 | snap at 20/03/2018 | snap at 23/05/2020
— Correia Lopes 2017/11/17 11:10