WindsPT@U.PORTO

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== UPORTO DATA ARCHIVE (UDA) ======

Perdigão's datasets repository using the [[http://www.unidata.ucar.edu/software/thredds/current/tds/|THREDDS Data Server (TDS)]]

  * **most recent info at the [[https://docs.google.com/document/d/1qBJk-eVFiO_A1f6n3IEd99q7GxZJtlKHM5LAOZU4-Do|docs.google.com/document]]**
  * [2020.05.23] TDS password removed
  * [2020.05.23] cron jobs removed and the mirroring process stopped {{closing_history.tgz|DTU account closing status}}

===== - Using the UDA =====

The UPORTO Data Archive (UDA) may be [[https://windsptds.fe.up.pt/thredds/catalog_perdigao.html|accessed using the THREDDS Data Server (TDS)]] by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!).

WindsP App users may [[https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml|explore the UPORTO Data Archive (UDA)]] but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months). 

{{uporto_data_archive_20171120_.pdf|}} | {{ :uda:uporto_data_archive_2020-05-23.pdf |}}


===== - UDA contents =====

^ Export  ^ Size ^ Last 24 hours  ^
| DLR  | 1.2 TiB | 2.32553 GiB  |
| DTU  | 2.0 TiB | 0.000537872 GiB  |
| INEGI  | 301 MiB | 0 GiB  |
| UCAR  | 1.6 TiB | 6.24173 GiB  |
| WINDFORS  | 3.2 GiB | 0 GiB  |

Summary on 20-03-2018 00:00. 
For more info see the [[http://winds.fe.up.pt/datalogs/?C=N;O=D|datalogs details]].

<file>
perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao
data/perdigao
├── dlr
│   ├── HATPRO_level-1
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_level-2
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_surface-met
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── mcs_data
│   │   ├── 20170430095732
│   │   ...
│   ├── netcdf_lidar
│   │   ├── DLR85
│   │   ├── DLR86
│   │   ├── DLR89
│   │   └── readme.tx
│   │── raw_data
│   │   ├── DLR85
│   │   ├── DLR86
│   │   └── DLR89
│   └── sound
│       ├── mic1
│       ├── mic2
│       ├── mic3
│       ├── mic4
│       ├── mic5
│       └── microphone_position.txt
├── dtu
│   ├── data
│   │   ├── DTU_Leica_Scanning
│   │   ├── DTU_Mast_Data
│   │   └── DTU_WindScanner
│   ├── docs
│   ├── landscape
│   ├── photos
│   └── plots
│       └── DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   │   └── 01_RawData
│   └── LidarAerialSurvey_RawData
│       ├── Images
│       ├── PerdigaoTurbineTopView.pdf
│       ├── PointCloud
│       └── Portugal Laserscanning Report.pdf
├── ucar
│   ├── arl
│   │   ├── ARL_Scanning_Lidar_George_Site
│   │   ├── ARL_Scanning_Lidar_Lionstail_Site
│   │   └── ARL_Scintillometer
│   ├── colorado
│   │   └── CU_Lidar
│   ├── eol
│   │   └── WV-DIAL
│   ├── isfs
│   │   ├── hr_noqc_geo
│   │   └── noqc_geo_notiltcor
│   ├── ncas
│   │   └── NCAS_profiler
│   ├── notredame
│   │   ├── UND_Ceilometer
│   │   ├── UND_Radiosonde
│   │   ├── UND_Scanning_Lidar_Lionshead_Site
│   │   ├── UND_Scanning_Lidar_MI6_Site
│   │   ├── UND_Scanning_Lidar_Orange_Site
│   │   └── UND_SODAR_RASS
│   └── oklahoma
│       ├── CLAMPS_AERI
│       ├── CLAMPS_MWR
│       └── CLAMPS_Scanning_Lidar
└── windfors
    ├── 2017
    │   ├── 201704
    │   ├── 201705
    │   └── 201706
    └── cross
        └── 2017
        
441 directories, 8 files
</file>

===== - Building the UDA =====

Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in  their catalogue in UDA (rsync exports).

Available exports:
<file>
nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt
test           	RSYNC test
archive        	RSYNC UDA FILES (read only)
ucar           	RSYNC UCAR FILES
dtu            	RSYNC DTU FILES
inegi          	RSYNC INEGI FILES
dlr            	RSYNC DLR FILES
windfors       	RSYNC WindForS FILES
</file>

===== - Upload DTU data =====

UPORTO (as ''nejoco@login.neweuropeanwindatlas.eu'') uses the UDA export ''dtu@windsptds.fe.up.pt::dtu'' to sync data collected by DTU.

First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job: 
''/usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu''.

Later the ''--delete'' option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.

<code bash crons>
# DTU data sync to UDA, At minute 31 past every 4th hour
#31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
</code>

<code>
cat ~nejoco/sync-exclude-list
archive/
data/DLR_WindScanner/
</code>

===== - Upload UCAR data =====

UCAR uses the UDA export ''ucar@windsptds.fe.up.pt::ucar'' to copy NCAR/EOL ISFS data.

===== - Upload DLR data =====

DLR uses the UDA export ''dlr@windsptds.fe.up.pt::dlr'' to maintain the DLR data. 

===== - Upload INEGI data =====

INEGI uses the UDA export ''inegi@windsptds.fe.up.pt::inegi'' to maintain the ENERCON data and "Lidar Aerial Survey Data".

===== - Upload WindsForS data =====

WindsForS uses the UDA export ''windfors@windsptds.fe.up.pt::windfors'' to maintain the WindForS data.

===== - Mirror UCAR ftp site =====

Preliminary data at the ftp site (ARL, Notre dame, ...) uploaded with:
<code>
#! /bin/sh
dir=arl
source=ftp://ftp.eol.ucar.edu/pub/data/incoming/perdigao/uda/$dir
destination=/data/perdigao/ucar
nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 &
</code>

Afterwards it is verified by running in the ftp site:
<code>
#! /bin/sh
dir=arl
cd <ftp-root>/incoming/perdigao/uda
export RSYNC_PASSWORD=t****YLa****
archive=ucar@windsptds.fe.up.pt::ucar/$dir
rsync -avz --delete --dry-run $dir $archive
</code>

===== - Mirror UDA to DTU =====

The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: ''uda@windsptds.fe.up.pt::archive/'', using a cron job.

<code bash crons>
# UDA archive to DTU, At midnight every day
0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1
</code>

<code bash sync-uda.sh>
#! /bin/sh

# the Perdigao root at NEWA storage
perdigao=/newa/WP2/PERDIGAO

# the archive root
archive=$perdigao/archive

# the actual size of the archive
echo "Total du of $archive:"
du -ks $archive

# the UDA readonly password
export RSYNC_PASSWORD=-password-

# catalogues to sync
CATALOGS="dlr inegi ucar windfors"

for c in $CATALOGS; do
    # mirror catalog from the version at UDA (UPORTO)
    echo; echo "$(tr [a-z] [A-Z] <<< "$c"):"
    #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
    cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
    echo "$cmd..."
    # do it
    $cmd
done

# catalog structure
echo
tree -L 2 $archive

# total space usage for each archive
echo
du -khs $archive/*

# the final size of the archive
echo
echo "Total du of $archive:"
du -ks $archive

# the end
echo
echo "Done."
</code>

The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 2.0 TiB of storage).

<file>
/newa/WP2/PERDIGAO/archive
├── dlr
│   ├── HATPRO_level-1
│   ├── HATPRO_level-2
│   ├── HATPRO_surface-met
│   ├── mcs_data
│   ├── netcdf_lidar
│   └── raw_data
├── dtu
│   ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning
│   ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data
│   └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   └── LidarAerialSurvey_RawData
├── ucar
│   ├── isfs
│   ├── iss
│   └── ncas
└── windfors
    ├── 2017
    └── cross
</file>

===== - Current status =====

<note warning>There is a collaborative version of this table being updated at Google docs.</note>

{{ :uda:20171120_perdigao_archiving.pdf |snap at 22/12/2017}} | 
{{ :uda:20180320_perdigao-archives.pdf |snap at 20/03/2018}} | {{ :uda:2020-05-23_perdigao-archives.pdf |snap at 23/05/2020}}


 --- //[[jlopes@fe.up.pt|Correia Lopes]] 2017/11/17 11:10//
WindsPT@U.PORTO

User Tools

Site Tools

Page Tools
