Perdigão's datasets repository using the THREDDS Data Server (TDS)

The UPORTO Data Archive (UDA) can be accessed using the THREDDS Data Server (TDS) at https://windsptds.fe.up.pt/thredds/catalog_perdigao.html by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxxx).

WindsP App users can also explore the UPORTO Data Archive (UDA) at https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml and they have to provide TDS credentials, when they request access to data or meta-data that is in the Data Archive (outside WindsP App and during the embargoed period of 12 months).

perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao
data/perdigao
├── dlr
│   ├── HATPRO_level-1
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_level-2
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_surface-met
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── mcs_data
│   │   ├── 20170430095732
│   │   ...
│   ├── netcdf_lidar
│   │   ├── DLR85
│   │   ├── DLR86
│   │   └── DLR89
│   └── raw_data
│       ├── DLR85
│       ├── DLR86
│       └── DLR89
├── dtu
│   ├── data
│   │   ├── DTU_Leica_Scanning
│   │   ├── DTU_Mast_Data
│   │   └── DTU_WindScanner
│   ├── docs
│   ├── photos
│   └── plots
│       └── DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   │   └── 01_RawData
│   └── LidarAerialSurvey_RawData
│       ├── Images
│       └── PointCloud
├── ucar
│   ├── isfs
│   │   ├── hr_noqc_geo
│   │   └── noqc_geo_notiltcor
│   ├── iss
│   │   ├── sodar-rass-site
│   │   ├── upper-orange-grove-site
│   │   └── west-profiler-site
│   └── ncas
└── windfors
    ├── 2017
    │   ├── 201704
    │   ├── 201705
    │   └── 201706
    └── cross
        └── 2017

Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in their catalogue in UDA.

Available exports:

nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt
test           	RSYNC test
archive        	RSYNC UDA FILES (read only)
ucar           	RSYNC UCAR FILES
dtu            	RSYNC DTU FILES
inegi          	RSYNC INEGI FILES
dlr            	RSYNC DLR FILES
windfors       	RSYNC WindForS FILES

UPORTO uses the UDA export dtu@windsptds.fe.up.pt::dtu to sync data collected by DTU.

ssh nejoco@login.neweuropeanwindatlas.eu

First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using the cron command: rsync -az –delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu

Later the –delete option was removed and some diretories excluded to achieve the Perdigão Data Archive at UDA.

cat ~nejoco/sync-excluded-list
archive/
data/DLT_WindScanner/

crons

# DTU data sync to UDA, "At minute 31 past every 4th hour."
31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
#
# UDA archive to DTU, "At midnight every day."
0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1

sync-uda.sh

#! /bin/sh
 
# the Perdigao root at NEWA storage
perdigao=/newa/WP2/PERDIGAO
 
# the archive root
archive=$perdigao/archive
 
# the actual size of the archive
echo "Total du of $archive:"
du -ks $archive
 
# the UDA readonly password
export RSYNC_PASSWORD=t286rFTrktHn
 
# catalogues to sync
CATALOGS="dlr inegi ucar windfors"
 
for c in $CATALOGS; do
    # mirror catalog from the version at UDA (UPORTO)
    echo; echo "$(tr [a-z] [A-Z] <<< "$c"):"
    #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
    cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
    echo "$cmd..."
    # do it
    $cmd
done
 
# catalog structure
echo
tree -L 2 $archive
 
# total space usage for each archive
echo
du -khs $archive/*
 
# the final size of the archive
echo
echo "Total du of $archive:"
du -ks $archive
 
# the end
echo
echo "Done."

UCAR uses the UDA export ucar@windsptds.fe.up.pt::ucar

DLR

INEGI

WindsForS

The mirror of UPORTO Data Archive (UDA) to the NEWA storage is now complete.

To manually update the archive, you may run the shell script (see the attached log of today's execution ): ~nejoco/sync-uda.sh

We are automatically syncing every 24 hours using the cron command: rsync -az /newa/WP2/PERDIGAO/data/ dtu@windsptds.fe.up.pt::dtu/data/

The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to void using more 2.9 TiB of storage).

/newa/WP2/PERDIGAO/archive
├── dlr
│   ├── HATPRO_level-1
│   ├── HATPRO_level-2
│   ├── HATPRO_surface-met
│   ├── mcs_data
│   ├── netcdf_lidar
│   └── raw_data
├── dtu
│   ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning
│   ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data
│   └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   └── LidarAerialSurvey_RawData
├── ucar
│   ├── isfs
│   ├── iss
│   └── ncas
└── windfors
    ├── 2017
    └── cross

— Correia Lopes 2017/11/17 11:10