Perdigão's datasets repository using the THREDDS Data Server (TDS)

• most recent info at the docs.google.com/document
• [2020.05.23] TDS password removed
• [2020.05.23] cron jobs removed and the mirroring process stopped DTU account closing status

The UPORTO Data Archive (UDA) may be accessed using the THREDDS Data Server (TDS) by providing the same credentials as in the UCAR ftp site (perdigao / Bxxxxx!).

WindsP App users may explore the UPORTO Data Archive (UDA) but, when they request access to data or meta-data that is in the Data Archive, they have to provide the TDS credentials (during the embargoed period of 12 months).

uporto_data_archive_20171120_.pdf | uporto_data_archive_2020-05-23.pdf

Summary on 20-03-2018 00:00. For more info see the datalogs details.

perdigao@windsptds:/data$ tree -d -L 3 /data/perdigao
data/perdigao
├── dlr
│   ├── HATPRO_level-1
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_level-2
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── HATPRO_surface-met
│   │   ├── 201704
│   │   ├── 201704_quicklook
│   │   ...
│   ├── mcs_data
│   │   ├── 20170430095732
│   │   ...
│   ├── netcdf_lidar
│   │   ├── DLR85
│   │   ├── DLR86
│   │   ├── DLR89
│   │   └── readme.tx
│   │── raw_data
│   │   ├── DLR85
│   │   ├── DLR86
│   │   └── DLR89
│   └── sound
│       ├── mic1
│       ├── mic2
│       ├── mic3
│       ├── mic4
│       ├── mic5
│       └── microphone_position.txt
├── dtu
│   ├── data
│   │   ├── DTU_Leica_Scanning
│   │   ├── DTU_Mast_Data
│   │   └── DTU_WindScanner
│   ├── docs
│   ├── landscape
│   ├── photos
│   └── plots
│       └── DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   │   └── 01_RawData
│   └── LidarAerialSurvey_RawData
│       ├── Images
│       ├── PerdigaoTurbineTopView.pdf
│       ├── PointCloud
│       └── Portugal Laserscanning Report.pdf
├── ucar
│   ├── arl
│   │   ├── ARL_Scanning_Lidar_George_Site
│   │   ├── ARL_Scanning_Lidar_Lionstail_Site
│   │   └── ARL_Scintillometer
│   ├── colorado
│   │   └── CU_Lidar
│   ├── eol
│   │   └── WV-DIAL
│   ├── isfs
│   │   ├── hr_noqc_geo
│   │   └── noqc_geo_notiltcor
│   ├── ncas
│   │   └── NCAS_profiler
│   ├── notredame
│   │   ├── UND_Ceilometer
│   │   ├── UND_Radiosonde
│   │   ├── UND_Scanning_Lidar_Lionshead_Site
│   │   ├── UND_Scanning_Lidar_MI6_Site
│   │   ├── UND_Scanning_Lidar_Orange_Site
│   │   └── UND_SODAR_RASS
│   └── oklahoma
│       ├── CLAMPS_AERI
│       ├── CLAMPS_MWR
│       └── CLAMPS_Scanning_Lidar
└── windfors
    ├── 2017
    │   ├── 201704
    │   ├── 201705
    │   └── 201706
    └── cross
        └── 2017
        
441 directories, 8 files

Each institution, collecting data in the Perdigão experiment, also owns credentials to upload and maintain their data in their catalogue in UDA (rsync exports).

Available exports:

nejoco@VIND-pNEWA04:~> rsync -rdt rsync://windsptds.fe.up.pt
test           	RSYNC test
archive        	RSYNC UDA FILES (read only)
ucar           	RSYNC UCAR FILES
dtu            	RSYNC DTU FILES
inegi          	RSYNC INEGI FILES
dlr            	RSYNC DLR FILES
windfors       	RSYNC WindForS FILES

UPORTO (as [email protected]) uses the UDA export [email protected]::dtu to sync data collected by DTU.

First a complete mirror was in place, by automatically syncing every 4 hours the DTU data directory using a cron job: /usr/bin/rsync -az –delete /newa/WP2/PERDIGAO/ [email protected]::dtu.

Later the –delete option was removed and some directories excluded to achieve the Perdigão Data Archive at UDA.

crons

# DTU data sync to UDA, At minute 31 past every 4th hour
#31 */4 * * * /usr/bin/rsync -az --delete /newa/WP2/PERDIGAO/ [email protected]::dtu > /dev/null 2>&1
31 */4 * * * /usr/bin/rsync -az --exclude-from 'sync-exclude-list' /newa/WP2/PERDIGAO/ dtu@windsptds.fe.up.pt::dtu > /dev/null 2>&1

cat ~nejoco/sync-exclude-list
archive/
data/DLR_WindScanner/

UCAR uses the UDA export [email protected]::ucar to copy NCAR/EOL ISFS data.

DLR uses the UDA export [email protected]::dlr to maintain the DLR data.

INEGI uses the UDA export [email protected]::inegi to maintain the ENERCON data and "Lidar Aerial Survey Data".

WindsForS uses the UDA export [email protected]::windfors to maintain the WindForS data.

Preliminary data at the ftp site (ARL, Notre dame, …) uploaded with:

#! /bin/sh
dir=arl
source=ftp://ftp.eol.ucar.edu/pub/data/incoming/perdigao/uda/$dir
destination=/data/perdigao/ucar
nohup wget -m -nH --cut-dirs=5 -P $destination $source >| /dev/null 2>&1 &

Afterwards it is verified by running in the ftp site:

#! /bin/sh
dir=arl
cd <ftp-root>/incoming/perdigao/uda
export RSYNC_PASSWORD=t****YLa****
[email protected]::ucar/$dir
rsync -avz --delete --dry-run $dir $archive

The UPORTO Data Archive (UDA) is automatically synced to the DTU, every 24 hours, using the UDA read only export: [email protected]::archive/, using a cron job.

crons

# UDA archive to DTU, At midnight every day
0 0 * * * /home/nejoco/sync-uda.sh >| sync-uda_last.log 2>&1

sync-uda.sh

#! /bin/sh
 
# the Perdigao root at NEWA storage
perdigao=/newa/WP2/PERDIGAO
 
# the archive root
archive=$perdigao/archive
 
# the actual size of the archive
echo "Total du of $archive:"
du -ks $archive
 
# the UDA readonly password
export RSYNC_PASSWORD=-password-
 
# catalogues to sync
CATALOGS="dlr inegi ucar windfors"
 
for c in $CATALOGS; do
    # mirror catalog from the version at UDA (UPORTO)
    echo; echo "$(tr [a-z] [A-Z] <<< "$c"):"
    #cmd="rsync -avz [email protected]::archive/$c/ $archive/$c/"
    cmd="rsync -avz --delete [email protected]::archive/$c/ $archive/$c/"
    echo "$cmd..."
    # do it
    $cmd
done
 
# catalog structure
echo
tree -L 2 $archive
 
# total space usage for each archive
echo
du -khs $archive/*
 
# the final size of the archive
echo
echo "Total du of $archive:"
du -ks $archive
 
# the end
echo
echo "Done."

The DTU NEWA directory /newa/WP2/PERDIGAO/archive/ contains an exact copy of UDA, except for the DTU data that are links to existing NEWA directories (in order to avoid using a duplication 2.0 TiB of storage).

/newa/WP2/PERDIGAO/archive
├── dlr
│   ├── HATPRO_level-1
│   ├── HATPRO_level-2
│   ├── HATPRO_surface-met
│   ├── mcs_data
│   ├── netcdf_lidar
│   └── raw_data
├── dtu
│   ├── DTU_Leica_Scanning -> /newa/WP2/PERDIGAO/data/DTU_Leica_Scanning
│   ├── DTU_Mast_Data -> /newa/WP2/PERDIGAO/data/DTU_Mast_Data
│   └── DTU_WindScanner -> /newa/WP2/PERDIGAO/data/DTU_WindScanner
├── inegi
│   ├── EnerconWindTurbine
│   ├── LeosphereWindcube
│   └── LidarAerialSurvey_RawData
├── ucar
│   ├── isfs
│   ├── iss
│   └── ncas
└── windfors
    ├── 2017
    └── cross

snap at 22/12/2017 | snap at 20/03/2018 | snap at 23/05/2020

— Correia Lopes 2017/11/17 11:10