WindsPT@U.PORTO

windscanner.eu & NEWA

User Tools

Site Tools

You are here: Winds @ FEUP » UPORTO DATA ARCHIVE (UDA)
uda:index

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
uda:index [2017/11/19 14:33]
Correia Lopes [3. Building the UDA] 		uda:index [2020/06/03 10:39] (current)
Correia Lopes [UPORTO DATA ARCHIVE (UDA)]
Line 2: Line 2:
  
 Perdigão's datasets repository using the [[http://www.unidata.ucar.edu/software/thredds/current/tds/|THREDDS Data Server (TDS)]] Perdigão's datasets repository using the [[http://www.unidata.ucar.edu/software/thredds/current/tds/|THREDDS Data Server (TDS)]]
 +
 +  * **most recent info at the [[https://docs.google.com/document/d/1qBJk-eVFiO_A1f6n3IEd99q7GxZJtlKHM5LAOZU4-Do|docs.google.com/document]]**
 +  * [2020.05.23] TDS password removed
 +  * [2020.05.23] cron jobs removed and the mirroring process stopped {{closing_history.tgz|DTU account closing status}}
  
 ===== - Using the UDA ===== ===== - Using the UDA =====
  
-The UPORTO Data Archive (UDA) can be accessed using the THREDDS Data Server (TDS) at https://windsptds.fe.up.pt/thredds/catalog_perdigao.html by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxxx).+The UPORTO Data Archive (UDA) may be [[https://windsptds.fe.up.pt/thredds/catalog_perdigao.html|accessed using the THREDDS Data Server (TDS)]] by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!). 
 + 
 +WindsP App users may [[https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml|explore the UPORTO Data Archive (UDA)]] but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months).  
 + 
 +{{uporto_data_archive_20171120_.pdf|}} | {{ :uda:uporto_data_archive_2020-05-23.pdf |}}
  
-WindsP App users can also explore the UPORTO Data Archive (UDA) at https://windsp.fe.up.pt/experiments/3/datasets/~2Fthredds~2Fcatalog_perdigao.xml and they have to provide TDS credentials, when they request access to data or meta-data that is in the Data Archive (outside WindsP App and during the embargoed period of 12 months).  
  
 ===== - UDA contents ===== ===== - UDA contents =====
 +
 +^ Export  ^ Size ^ Last 24 hours  ^
 +| DLR  | 1.2 TiB | 2.32553 GiB  |
 +| DTU  | 2.0 TiB | 0.000537872 GiB  |
 +| INEGI  | 301 MiB | 0 GiB  |
 +| UCAR  | 1.6 TiB | 6.24173 GiB  |
 +| WINDFORS  | 3.2 GiB | 0 GiB  |
 +
 +Summary on 20-03-2018 00:00. 
 +For more info see the [[http://winds.fe.up.pt/datalogs/?C=N;O=D|datalogs details]].
  
 <file> <file>
Line 31: Line 48:
 │   │   ... │   │   ...
 │   ├── netcdf_lidar │   ├── netcdf_lidar
 +│   │   ├── DLR85
 +│   │   ├── DLR86
 +│   │   ├── DLR89
 +│   │   └── readme.tx
 +│   │── raw_data
 │   │   ├── DLR85 │   │   ├── DLR85
 │   │   ├── DLR86 │   │   ├── DLR86
 │   │   └── DLR89 │   │   └── DLR89
-│   └── raw_data +│   └── sound 
-│       ├── DLR85 +│       ├── mic1 
-│       ├── DLR86 +│       ├── mic2 
-│       └── DLR89+│       ├── mic3 
 +│       ├── mic4 
 +│       ├── mic5 
 +│       └── microphone_position.txt
 ├── dtu ├── dtu
 │   ├── data │   ├── data
Line 44: Line 69:
 │   │   └── DTU_WindScanner │   │   └── DTU_WindScanner
 │   ├── docs │   ├── docs
 +│   ├── landscape
 │   ├── photos │   ├── photos
 │   └── plots │   └── plots
Line 53: Line 79:
 │   └── LidarAerialSurvey_RawData │   └── LidarAerialSurvey_RawData
 │       ├── Images │       ├── Images
-│       └── PointCloud+│       ├── PerdigaoTurbineTopView.pdf 
 +│       ├── PointCloud 
 +│       └── Portugal Laserscanning Report.pdf
 ├── ucar ├── ucar
 +│   ├── arl
 +│   │   ├── ARL_Scanning_Lidar_George_Site
 +│   │   ├── ARL_Scanning_Lidar_Lionstail_Site
 +│   │   └── ARL_Scintillometer
 +│   ├── colorado
 +│   │   └── CU_Lidar
 +│   ├── eol
 +│   │   └── WV-DIAL
 │   ├── isfs │   ├── isfs
 │   │   ├── hr_noqc_geo │   │   ├── hr_noqc_geo
 │   │   └── noqc_geo_notiltcor │   │   └── noqc_geo_notiltcor
-│   ├── iss +│   ├── ncas 
-│   │   ├── sodar-rass-site +│   │   └── NCAS_profiler 
-│   │   ├── upper-orange-grove-site +│   ├── notredame 
-│   │   └── west-profiler-site +│   │   ├── UND_Ceilometer 
-│   └── ncas+│   │   ├── UND_Radiosonde 
 +│   │   ├── UND_Scanning_Lidar_Lionshead_Site 
 +│   │   ├── UND_Scanning_Lidar_MI6_Site 
 +│   │   ├── UND_Scanning_Lidar_Orange_Site 
 +│   │   └── UND_SODAR_RASS 
 +│   └── oklahoma 
 +│       ├── CLAMPS_AERI 
 +│       ├── CLAMPS_MWR 
 +│       └── CLAMPS_Scanning_Lidar
 └── windfors └── windfors
     ├── 2017     ├── 2017
Line 70: Line 114:
     └── cross     └── cross
         └── 2017         └── 2017
 +        
 +441 directories, 8 files
 </file> </file>
  
 ===== - Building the UDA ===== ===== - Building the UDA =====
  
-Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in  their catalogue in UDA.+Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in  their catalogue in UDA (rsync exports).
  
 Available exports: Available exports:
Line 90: Line 136:
 ===== - Upload DTU data ===== ===== - Upload DTU data =====
  
-UPORTO uses the UDA export ''dtu@windsptds.fe.up.pt::dtu'' to sync data collected by DTU.+UPORTO (as ''nejoco@login.neweuropeanwindatlas.eu'') uses the UDA export ''dtu@windsptds.fe.up.pt::dtu'' to sync data collected by DTU. 
 + 
 +First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job:  
 +''/usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu''. 
 + 
 +Later the ''--delete'' option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA. 
 + 
 +<code bash crons> 
 +# DTU data sync to UDA, At minute 31 past every 4th hour 
 +#31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1 
 +31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1 
 +</code>
  
 <code> <code>
-ssh nejoco@login.neweuropeanwindatlas.eu+cat ~nejoco/sync-exclude-list 
 +archive/ 
 +data/DLR_WindScanner/
 </code> </code>
  
-First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using the cron command: +===== - Upload UCAR data =====
-''rsync -az --delete /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu'' +
  
-Later the ''--delete'' option was removed and some diretories excluded to achieve the Perdigão Data Archive at UDA.+UCAR uses the UDA export ''ucar@windsptds.fe.up.pt::ucar'' to copy NCAR/EOL ISFS data.
  
 +===== - Upload DLR data =====
 +
 +DLR uses the UDA export ''dlr@windsptds.fe.up.pt::dlr'' to maintain the DLR data. 
 +
 +===== - Upload INEGI data =====
 +
 +INEGI uses the UDA export ''inegi@windsptds.fe.up.pt::inegi'' to maintain the ENERCON data and "Lidar Aerial Survey Data".
 +
 +===== - Upload WindsForS data =====
 +
 +WindsForS uses the UDA export ''windfors@windsptds.fe.up.pt::windfors'' to maintain the WindForS data.
 +
 +===== - Mirror UCAR ftp site =====
 +
 +Preliminary data at the ftp site (ARL, Notre dame, ...) uploaded with:
 <code> <code>
-cat ~nejoco/sync-excluded-list +#! /bin/sh 
-archive/ +dir=arl 
-data/DLT_WindScanner/+source=ftp://ftp.eol.ucar.edu/pub/data/incoming/perdigao/uda/$dir 
 +destination=/data/perdigao/ucar 
 +nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 &
 </code> </code>
 +
 +Afterwards it is verified by running in the ftp site:
 +<code>
 +#! /bin/sh
 +dir=arl
 +cd <ftp-root>/incoming/perdigao/uda
 +export RSYNC_PASSWORD=t****YLa****
 +archive=ucar@windsptds.fe.up.pt::ucar/$dir
 +rsync -avz --delete --dry-run $dir $archive
 +</code>
 +
 +===== - Mirror UDA to DTU =====
 +
 +The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: ''uda@windsptds.fe.up.pt::archive/'', using a cron job.
  
 <code bash crons> <code bash crons>
-# DTU data sync to UDA, "At minute 31 past every 4th hour." +# UDA archive to DTU, At midnight every day
-31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1 +
-# +
-# UDA archive to DTU, "At midnight every day."+
 0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1 0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1
 </code> </code>
Line 129: Line 215:
  
 # the UDA readonly password # the UDA readonly password
-export RSYNC_PASSWORD=t286rFTrktHn+export RSYNC_PASSWORD=-password-
  
 # catalogues to sync # catalogues to sync
Line 162: Line 248:
 </code> </code>
  
-===== - Collecting UCAR data ===== +The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 2.0 TiB of storage).
- +
-UCAR uses the UDA export ''ucar@windsptds.fe.up.pt::ucar'' +
- +
-DLR +
- +
-INEGI +
- +
-WindsForS +
- +
-===== - Mirror UDA to DTU ===== +
- +
-The mirror of UPORTO Data Archive (UDA) to the NEWA storage is now complete. +
- +
-To manually update the archive, you may run the shell script (see the attached log of today's execution ): +
-~nejoco/sync-uda.sh +
- +
-We are automatically syncing every 24 hours using the cron command: +
-''rsync -az /newa/WP2/PERDIGAO/data/ dtu@windsptds.fe.up.pt::dtu/data/''  +
- +
- +
-The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to void using more 2.9 TiB of storage).+
  
 <file> <file>
Line 213: Line 278:
 ===== - Current status ===== ===== - Current status =====
  
-^ Data           ^  DTU  ^  UPORTO  ^  UCAR  ^ +<note warning>There is a collaborative version of this table being updated at Google docs.</note> 
-| **ALR**        |  ✘    |  ✘       |  ✔     | + 
-| **CU**         |  ✘    |  ✘       |  ✘     | +{{ :uda:20171120_perdigao_archiving.pdf |snap at 22/12/2017}} |  
-| **DLR**        |  ✔    |  ✔       |  ✘     | +{{ :uda:20180320_perdigao-archives.pdf |snap at 20/03/2018}} | {{ :uda:2020-05-23_perdigao-archives.pdf |snap at 23/05/2020}}
-| **DTU**        |  ✔    |  ✔       |  ✘     | +
-| **ENERCON**    |  ✔    |  ✔       |  ✘     | +
-| **IPMA**       |  ✘    |  ✘       |  ✘     | +
-| **ISFS**       |  ✔    |  ✔       |  ✔     | +
-| **ISS**        |  ✔    |  ✔       |  ✔     | +
-| **Leosphere**  |  ✔    |  ✔       |  ✔     | +
-| **NCAS**       |  ✔    |  ✔       |  ✔     | +
-| **ND**         |  ✘    |  ✘       |  ✔     | +
-| **OU**         |  ✘    |  ✘       |  ✘     | +
-| **WindForS**   |  ✔    |  ✔       |  ✘     |+
  
  
  --- //[[jlopes@fe.up.pt|Correia Lopes]] 2017/11/17 11:10//  --- //[[jlopes@fe.up.pt|Correia Lopes]] 2017/11/17 11:10//
  
uda/index.1511098401.txt.gz · Last modified: 2017/11/19 14:33 by Correia Lopes

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki