Perdigão's datasets repository using the THREDDS Data Server (TDS)

The UPORTO Data Archive (UDA) can be accessed using the THREDDS Data Server (TDS) at https://windsptds.fe.up.pt/thredds/catalog_perdigao.html by providing the same credentials as in the UCAR ftp site (perdigao / -password-).

WindsP App users can also explore the UPORTO Data Archive (UDA) at https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml and they have to provide TDS credentials, when they request access to data or meta-data that is in the Data Archive (outside WindsP App and during the embargoed period of 12 months).

Summary on 19-11-2017 22:19. For more info see the datalogs details.

perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao
data/perdigao
├── dlr
│   ├── HATPRO_level-1
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_level-2
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_surface-met
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── mcs_data
│   │   ├── 20170430095732
│   │   ...
│   ├── netcdf_lidar
│   │   ├── DLR85
│   │   ├── DLR86
│   │   └── DLR89
│   └── raw_data
│       ├── DLR85
│       ├── DLR86
│       └── DLR89
├── dtu
│   ├── data
│   │   ├── DTU_Leica_Scanning
│   │   ├── DTU_Mast_Data
│   │   └── DTU_WindScanner
│   ├── docs
│   ├── photos
│   └── plots
│       └── DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   │   └── 01_RawData
│   └── LidarAerialSurvey_RawData
│       ├── Images
│       └── PointCloud
├── ucar
│   ├── isfs
│   │   ├── hr_noqc_geo
│   │   └── noqc_geo_notiltcor
│   ├── iss
│   │   ├── sodar-rass-site
│   │   ├── upper-orange-grove-site
│   │   └── west-profiler-site
│   └── ncas
└── windfors
    ├── 2017
    │   ├── 201704
    │   ├── 201705
    │   └── 201706
    └── cross
        └── 2017

Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in their catalogue in UDA (rsync exports).

Available exports:

nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt
test           	RSYNC test
archive        	RSYNC UDA FILES (read only)
ucar           	RSYNC UCAR FILES
dtu            	RSYNC DTU FILES
inegi          	RSYNC INEGI FILES
dlr            	RSYNC DLR FILES
windfors       	RSYNC WindForS FILES

UPORTO (as nejoco@login.neweuropeanwindatlas.eu) uses the UDA export dtu@windsptds.fe.up.pt::dtu to sync data collected by DTU.

First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job: /usr/bin/rsync -az –delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu.

Later the –delete option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.

crons

# DTU data sync to UDA, At minute 31 past every 4th hour
31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1

cat ~nejoco/sync-exclude-list
archive/
data/DLT_WindScanner/

UCAR uses the UDA export ucar@windsptds.fe.up.pt::ucar to copy NCAR/EOL ISFS data.

DLR uses the UDA export dlr@windsptds.fe.up.pt::dlr to maintain the DLR data.

INEGI uses the UDA export inegi@windsptds.fe.up.pt::inegi to maintain the ENERCON data and “Lidar Aerial Survey Data”.

WindsForS uses the UDA export windfors@windsptds.fe.up.pt::windfors to maintain the WindForS data.

The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: uda@windsptds.fe.up.pt::archive/, using a cron job.

crons

# UDA archive to DTU, At midnight every day
0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1

sync-uda.sh

#! /bin/sh
 
# the Perdigao root at NEWA storage
perdigao=/newa/WP2/PERDIGAO
 
# the archive root
archive=$perdigao/archive
 
# the actual size of the archive
echo "Total du of $archive:"
du -ks $archive
 
# the UDA readonly password
export RSYNC_PASSWORD=-password-
 
# catalogues to sync
CATALOGS="dlr inegi ucar windfors"
 
for c in $CATALOGS; do
    # mirror catalog from the version at UDA (UPORTO)
    echo; echo "$(tr [a-z] [A-Z] <<< "$c"):"
    #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
    cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
    echo "$cmd..."
    # do it
    $cmd
done
 
# catalog structure
echo
tree -L 2 $archive
 
# total space usage for each archive
echo
du -khs $archive/*
 
# the final size of the archive
echo
echo "Total du of $archive:"
du -ks $archive
 
# the end
echo
echo "Done."

The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 1.8 TiB of storage).

/newa/WP2/PERDIGAO/archive
├── dlr
│   ├── HATPRO_level-1
│   ├── HATPRO_level-2
│   ├── HATPRO_surface-met
│   ├── mcs_data
│   ├── netcdf_lidar
│   └── raw_data
├── dtu
│   ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning
│   ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data
│   └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   └── LidarAerialSurvey_RawData
├── ucar
│   ├── isfs
│   ├── iss
│   └── ncas
└── windfors
    ├── 2017
    └── cross

— Correia Lopes 2017/11/17 11:10