This is an old revision of the document!
Perdigão's datasets repository using the THREDDS Data Server (TDS)
The UPORTO Data Archive (UDA) may be accessed using the THREDDS Data Server (TDS) by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!).
WindsP App users may explore the UPORTO Data Archive (UDA) but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months).
Export | Size | Last 24 hours |
---|---|---|
DTU | 1.8 TiB | 4.06844 GiB |
UCAR | 1.2 TiB | 0 GiB |
DLR | 1.2 TiB | 0 GiB |
INEGI | 301 MiB | 0 GiB |
WINDFORS | 3.2 GiB | 0 GiB |
Summary on 19-11-2017 22:19. For more info see the datalogs details.
perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao data/perdigao ├── dlr │ ├── HATPRO_level-1 │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── HATPRO_level-2 │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── HATPRO_surface-met │ │ ├── 201704 │ │ ├── 201704_quicklook │ │ ... │ ├── mcs_data │ │ ├── 20170430095732 │ │ ... │ ├── netcdf_lidar │ │ ├── DLR85 │ │ ├── DLR86 │ │ └── DLR89 │ └── raw_data │ ├── DLR85 │ ├── DLR86 │ └── DLR89 ├── dtu │ ├── data │ │ ├── DTU_Leica_Scanning │ │ ├── DTU_Mast_Data │ │ └── DTU_WindScanner │ ├── docs │ ├── photos │ └── plots │ └── DTU_WindScanner ├── inegi │ ├── EnerconWindTurbine │ ├── LeosphereWindcube │ │ └── 01_RawData │ └── LidarAerialSurvey_RawData │ ├── Images │ └── PointCloud ├── ucar │ ├── isfs │ │ ├── hr_noqc_geo │ │ └── noqc_geo_notiltcor │ ├── iss │ │ ├── sodar-rass-site │ │ ├── upper-orange-grove-site │ │ └── west-profiler-site │ └── ncas └── windfors ├── 2017 │ ├── 201704 │ ├── 201705 │ └── 201706 └── cross └── 2017
Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in their catalogue in UDA (rsync exports).
Available exports:
nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt test RSYNC test archive RSYNC UDA FILES (read only) ucar RSYNC UCAR FILES dtu RSYNC DTU FILES inegi RSYNC INEGI FILES dlr RSYNC DLR FILES windfors RSYNC WindForS FILES
UPORTO (as nejoco@login.neweuropeanwindatlas.eu
) uses the UDA export dtu@windsptds.fe.up.pt::dtu
to sync data collected by DTU.
First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job:
/usr/bin/rsync -az –delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu
.
Later the –delete
option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.
# DTU data sync to UDA, At minute 31 past every 4th hour #31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1 31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
cat ~nejoco/sync-exclude-list archive/ data/DLR_WindScanner/
UCAR uses the UDA export ucar@windsptds.fe.up.pt::ucar
to copy NCAR/EOL ISFS data.
DLR uses the UDA export dlr@windsptds.fe.up.pt::dlr
to maintain the DLR data.
INEGI uses the UDA export inegi@windsptds.fe.up.pt::inegi
to maintain the ENERCON data and “Lidar Aerial Survey Data”.
WindsForS uses the UDA export windfors@windsptds.fe.up.pt::windfors
to maintain the WindForS data.
The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: uda@windsptds.fe.up.pt::archive/
, using a cron job.
# UDA archive to DTU, At midnight every day 0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1
#! /bin/sh # the Perdigao root at NEWA storage perdigao=/newa/WP2/PERDIGAO # the archive root archive=$perdigao/archive # the actual size of the archive echo "Total du of $archive:" du -ks $archive # the UDA readonly password export RSYNC_PASSWORD=-password- # catalogues to sync CATALOGS="dlr inegi ucar windfors" for c in $CATALOGS; do # mirror catalog from the version at UDA (UPORTO) echo; echo "$(tr [a-z] [A-Z] <<< "$c"):" #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/" cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/" echo "$cmd..." # do it $cmd done # catalog structure echo tree -L 2 $archive # total space usage for each archive echo du -khs $archive/* # the final size of the archive echo echo "Total du of $archive:" du -ks $archive # the end echo echo "Done."
The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 1.8 TiB of storage).
/newa/WP2/PERDIGAO/archive ├── dlr │ ├── HATPRO_level-1 │ ├── HATPRO_level-2 │ ├── HATPRO_surface-met │ ├── mcs_data │ ├── netcdf_lidar │ └── raw_data ├── dtu │ ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning │ ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data │ └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner ├── inegi │ ├── EnerconWindTurbine │ ├── LeosphereWindcube │ └── LidarAerialSurvey_RawData ├── ucar │ ├── isfs │ ├── iss │ └── ncas └── windfors ├── 2017 └── cross
<note warning>There is a collaborative version of this table being updated at Google docs.</note>
Institution | Data | Responsable | DTU | UPORTO | UCAR |
---|---|---|---|---|---|
ALR | Scintillometer Data | R. Krishnamurthy | ✘ | ✘ | ✔ |
CO2/H2O Gas Analyzer Data | C. Hocut | ✘ | ✘ | ✘ | |
Tower Data | C. Hocut | ✘ | ✘ | ✘ | |
CU | High Frequency Seismometer Data | ? | ✘ | ✘ | ✘ |
Cornell Tower Data | R. Barthelmie | ✘ | ✘ | ✘ | |
Cornell University Microbarometer Data | R. Barthelmie | ✘ | ✘ | ✘ | |
DLR | Acoustic Measurement Data | A. Schady | ✔ | ✔ | ✘ |
HATPRO Surface Meteorological Data | M. Hagen | ✔ | ✔ | ✘ | |
DTU | Tower Data | J. Mann | ✔ | ✔ | ✘ |
ENERCON | ENERCON Tower Data | ? | ✔ | ✔ | ✘ |
ENERCON Wind Turbine SCADA Data | ? | ✘ | ✘ | ✘ | |
IPMA | Portugal Surface Meteorological Data | ? | ✘ | ✘ | ✘ |
Portugal Lightning Detection Network (LDN) Data | S. Correia and V. Prior | ✘ | ✘ | ✔ | |
INEGI | Lidar Aerial Survey Data | ? | ✔ | ✔ | ✘ |
UCAR/EOL | Preliminary 5 minute ISFS data, geographic coordinate | ? | ✔ | ✔ | ✔ |
Preliminary 5 minute ISFS data, instrument coordinates | ? | ✔ | ✔ | ✔ | |
Preliminary High Rate ISFS data | ? | ✔ | ✔ | ✔ | |
GTS LDM Surface Hourly Observations | Global, GEMPAK | ✔ | |||
GTS LDM Surface Synoptic Observations | Global, GEMPAK | ✔ | |||
LDM Surface METAR Data | METAR format | ✔ | |||
ISS Preliminary Surface Meteorology Data | Sodar-RASS Site | ✔ | |||
ISS Preliminary Surface Meteorology Data | Sounding Site at the upper orange grove | ✔ | |||
ISS Preliminary Surface Meteorology Data | West Profiler Site | ✔ | |||
Perdigao Field Catalog Surface Products | ✔ | ||||
NCAS | ✔ | ✔ | ✔ | ||
NOAA | MADIS GTS METAR Data | NOAA/ESRL | ✘ | ✘ | ✔ |
ND | ✘ | ✘ | |||
OU | OU Tower Data | P. Klein | ✘ | ✘ | |
NSSL/OU CLAMPS Surface Meteorological Data | P. Klein | ✘ | ✘ |
— Correia Lopes 2017/11/17 11:10