Differences

This shows you the differences between two versions of the page.

--- uda:index [2017/11/19 13:29]
Correia Lopes [2. Collecting DTU data]
+++ uda:index [2020/06/03 10:39] (current)
Correia Lopes [UPORTO DATA ARCHIVE (UDA)]
@@ Line 2: / Line 2: @@
 Perdigão's datasets repository using the [[http://www.unidata.ucar.edu/software/thredds/current/tds/|THREDDS Data Server (TDS)]]
+  * **most recent info at the [[https://docs.google.com/document/d/1qBJk-eVFiO_A1f6n3IEd99q7GxZJtlKHM5LAOZU4-Do|docs.google.com/document]]**
+  * [2020.05.23] TDS password removed
+  * [2020.05.23] cron jobs removed and the mirroring process stopped {{closing_history.tgz|DTU account closing status}}
 ===== - Using the UDA =====
-The UDA can be accessed using the TDS at  https://windsptds.fe.up.pt/thredds/catalog_perdigao.html
+The UPORTO Data Archive (UDA) may be [[https://windsptds.fe.up.pt/thredds/catalog_perdigao.html|accessed using the THREDDS Data Server (TDS)]] by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!).
-The UDA can be explored using the WindsP App at https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml
+WindsP App users may [[https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml|explore the UPORTO Data Archive (UDA)]] but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months).
-As a WindsP user you get credentials to the Web App.
+{{uporto_data_archive_20171120_.pdf|}} | {{ :uda:uporto_data_archive_2020-05-23.pdf |}}
-When you request access to data that is in the Data Archive (outside WindsP and embargoed during 12 months) you must provide the TDS credentials:
-perdigao / Bl@yer! (the same as in the UCAR ftp site).
-Each institution also owns credentials to their catalogue in our Data Server, to upload and maintain their data.
+===== - UDA contents =====
+^ Export  ^ Size ^ Last 24 hours  ^
+| DLR  | 1.2 TiB | 2.32553 GiB  |
+| DTU  | 2.0 TiB | 0.000537872 GiB  |
+| INEGI  | 301 MiB | 0 GiB  |
+| UCAR  | 1.6 TiB | 6.24173 GiB  |
+| WINDFORS  | 3.2 GiB | 0 GiB  |
-===== - Collecting DTU data =====
+Summary on 20-03-2018 00:00.
+For more info see the [[http://winds.fe.up.pt/datalogs/?C=N;O=D|datalogs details]].
-UPORTO uses the UDA export ''dtu@windsptds.fe.up.pt::dtu'' to sync data collected by DTU.
+<file>
+perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao
+data/perdigao
+├── dlr
+│   ├── HATPRO_level-1
+│   │   ├── 201704
+│   │   ├── 201704_quicklook
+│   │   ...
+│   ├── HATPRO_level-2
+│   │   ├── 201704
+│   │   ├── 201704_quicklook
+│   │   ...
+│   ├── HATPRO_surface-met
+│   │   ├── 201704
+│   │   ├── 201704_quicklook
+│   │   ...
+│   ├── mcs_data
+│   │   ├── 20170430095732
+│   │   ...
+│   ├── netcdf_lidar
+│   │   ├── DLR85
+│   │   ├── DLR86
+│   │   ├── DLR89
+│   │   └── readme.tx
+│   │── raw_data
+│   │   ├── DLR85
+│   │   ├── DLR86
+│   │   └── DLR89
+│   └── sound
+│       ├── mic1
+│       ├── mic2
+│       ├── mic3
+│       ├── mic4
+│       ├── mic5
+│       └── microphone_position.txt
+├── dtu
+│   ├── data
+│   │   ├── DTU_Leica_Scanning
+│   │   ├── DTU_Mast_Data
+│   │   └── DTU_WindScanner
+│   ├── docs
+│   ├── landscape
+│   ├── photos
+│   └── plots
+│       └── DTU_WindScanner
+├── inegi
+│   ├── EnerconWindTurbine
+│   ├── LeosphereWindcube
+│   │   └── 01_RawData
+│   └── LidarAerialSurvey_RawData
+│       ├── Images
+│       ├── PerdigaoTurbineTopView.pdf
+│       ├── PointCloud
+│       └── Portugal Laserscanning Report.pdf
+├── ucar
+│   ├── arl
+│   │   ├── ARL_Scanning_Lidar_George_Site
+│   │   ├── ARL_Scanning_Lidar_Lionstail_Site
+│   │   └── ARL_Scintillometer
+│   ├── colorado
+│   │   └── CU_Lidar
+│   ├── eol
+│   │   └── WV-DIAL
+│   ├── isfs
+│   │   ├── hr_noqc_geo
+│   │   └── noqc_geo_notiltcor
+│   ├── ncas
+│   │   └── NCAS_profiler
+│   ├── notredame
+│   │   ├── UND_Ceilometer
+│   │   ├── UND_Radiosonde
+│   │   ├── UND_Scanning_Lidar_Lionshead_Site
+│   │   ├── UND_Scanning_Lidar_MI6_Site
+│   │   ├── UND_Scanning_Lidar_Orange_Site
+│   │   └── UND_SODAR_RASS
+│   └── oklahoma
+│       ├── CLAMPS_AERI
+│       ├── CLAMPS_MWR
+│       └── CLAMPS_Scanning_Lidar
+└── windfors
+    ├── 2017
+    │   ├── 201704
+    │   ├── 201705
+    │   └── 201706
+    └── cross
+        └── 2017
+directories, 8 files
+</file>
+===== - Building the UDA =====
+Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in  their catalogue in UDA (rsync exports).
+Available exports:
+<file>
+nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt
+test           	RSYNC test
+archive        	RSYNC UDA FILES (read only)
+ucar           	RSYNC UCAR FILES
+dtu            	RSYNC DTU FILES
+inegi          	RSYNC INEGI FILES
+dlr            	RSYNC DLR FILES
+windfors       	RSYNC WindForS FILES
+</file>
+===== - Upload DTU data =====
+UPORTO (as ''nejoco@login.neweuropeanwindatlas.eu'') uses the UDA export ''dtu@windsptds.fe.up.pt::dtu'' to sync data collected by DTU.
+First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job:
+''/usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu''.
+Later the ''--delete'' option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.
+<code bash crons>
+# DTU data sync to UDA, At minute 31 past every 4th hour
+#31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
+*/4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
+</code>
 <code>
-ssh nejoco@login.neweuropeanwindatlas.eu
+cat ~nejoco/sync-exclude-list
+archive/
+data/DLR_WindScanner/
 </code>
-First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using the cron command:
+===== - Upload UCAR data =====
-''rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu''
+UCAR uses the UDA export ''ucar@windsptds.fe.up.pt::ucar'' to copy NCAR/EOL ISFS data.
+===== - Upload DLR data =====
+DLR uses the UDA export ''dlr@windsptds.fe.up.pt::dlr'' to maintain the DLR data.
+===== - Upload INEGI data =====
+INEGI uses the UDA export ''inegi@windsptds.fe.up.pt::inegi'' to maintain the ENERCON data and "Lidar Aerial Survey Data".
+===== - Upload WindsForS data =====
+WindsForS uses the UDA export ''windfors@windsptds.fe.up.pt::windfors'' to maintain the WindForS data.
-Later the ''--delete'' option was removed and some diretories excluded to achieve the Perdigão Data Archive at UDA.
+===== - Mirror UCAR ftp site =====
+Preliminary data at the ftp site (ARL, Notre dame, ...) uploaded with:
 <code>
-cat ~nejoco/sync-excluded-list
+#! /bin/sh
-archive/
+dir=arl
-data/DLT_WindScanner/
+source=ftp://ftp.eol.ucar.edu/pub/data/incoming/perdigao/uda/$dir
->/code>
+destination=/data/perdigao/ucar
+nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 &
+</code>
+Afterwards it is verified by running in the ftp site:
+<code>
+#! /bin/sh
+dir=arl
+cd <ftp-root>/incoming/perdigao/uda
+export RSYNC_PASSWORD=t****YLa****
+archive=ucar@windsptds.fe.up.pt::ucar/$dir
+rsync -avz --delete --dry-run $dir $archive
+</code>
+===== - Mirror UDA to DTU =====
+The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: ''uda@windsptds.fe.up.pt::archive/'', using a cron job.
 <code bash crons>
-# DTU data sync to UDA, "At minute 31 past every 4th hour."
+# UDA archive to DTU, At midnight every day
-*/4 * * * /usr/bin/rsync -az --exclude-from '~nejoco/sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1
-#
-# UDA archive to DTU, "At midnight every day."
 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1
 </code>
-===== - Collecting UCAR data =====
+<code bash sync-uda.sh>
+#! /bin/sh
-UCAR uses the UDA export ''ucar@windsptds.fe.up.pt::ucar''
+# the Perdigao root at NEWA storage
+perdigao=/newa/WP2/PERDIGAO
-DLR
+# the archive root
+archive=$perdigao/archive
-INEGI
+# the actual size of the archive
+echo "Total du of $archive:"
+du -ks $archive
-WindsForS
+# the UDA readonly password
+export RSYNC_PASSWORD=-password-
-===== - Mirror UDA to DTU =====
+# catalogues to sync
+CATALOGS="dlr inegi ucar windfors"
-The mirror of UPORTO Data Archive (UDA) to the NEWA storage is now complete.
+for c in $CATALOGS; do
+    # mirror catalog from the version at UDA (UPORTO)
+    echo; echo "$(tr [a-z] [A-Z] <<< "$c"):"
+    #cmd="rsync -avz uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
+    cmd="rsync -avz --delete uda@windsptds.fe.up.pt::archive/$c/ $archive/$c/"
+    echo "$cmd..."
+    # do it
+    $cmd
+done
-To manually update the archive, you may run the shell script (see the attached log of today's execution ):
+# catalog structure
-~nejoco/sync-uda.sh
+echo
+tree -L 2 $archive
-We are automatically syncing every 24 hours using the cron command:
+# total space usage for each archive
-''rsync -az /newa/WP2/PERDIGAO/data/ dtu@windsptds.fe.up.pt::dtu/data/''
+echo
+du -khs $archive/*
+# the final size of the archive
+echo
+echo "Total du of $archive:"
+du -ks $archive
-The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to void using more 2.9 TiB of storage).
+# the end
+echo
+echo "Done."
+</code>
+The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 2.0 TiB of storage).
 <file>
@@ Line 95: / Line 278: @@
 ===== - Current status =====
-^ Data           ^  DTU  ^  UPORTO  ^  UCAR  ^
+<note warning>There is a collaborative version of this table being updated at Google docs.</note>
-| **ALR**        |  ✘    |  ✘       |  ✔     |
-| **CU**         |  ✘    |  ✘       |  ✘     |
+{{ :uda:20171120_perdigao_archiving.pdf |snap at 22/12/2017}} |
-| **DLR**        |  ✔    |  ✔       |  ✘     |
+{{ :uda:20180320_perdigao-archives.pdf |snap at 20/03/2018}} | {{ :uda:2020-05-23_perdigao-archives.pdf |snap at 23/05/2020}}
-| **DTU**        |  ✔    |  ✔       |  ✘     |
-| **ENERCON**    |  ✔    |  ✔       |  ✘     |
-| **IPMA**       |  ✘    |  ✘       |  ✘     |
-| **ISFS**       |  ✔    |  ✔       |  ✔     |
-| **ISS**        |  ✔    |  ✔       |  ✔     |
-| **Leosphere**  |  ✔    |  ✔       |  ✔     |
-| **NCAS**       |  ✔    |  ✔       |  ✔     |
-| **ND**         |  ✘    |  ✘       |  ✔     |
-| **OU**         |  ✘    |  ✘       |  ✘     |
-| **WindForS**   |  ✔    |  ✔       |  ✘     |
  --- //[[jlopes@fe.up.pt|Correia Lopes]] 2017/11/17 11:10//

WindsPT@U.PORTO

User Tools

Site Tools

Differences

Page Tools