Data Deliverables Summary for AUV Sentry

1. Cruise Structure Organization

The Sentry cruise structure is a set of nested file-system directories, the top level composed of the following:

<CruiseID>

• dives docs
• engineering planning planning-bathy
• plots
• products
• raw-usbl
• svp

dives- All raw and processed data from individual dives organized by dive

docs- Documents pertaining to the cruise such as launch positions and dive statistic summaries

engineering- All raw and processed data from decktests and dive debugging

planning- Files pertaining to mission planning. These are not generally needed by science

planning-bathy- This is the bathy provided by science or generated during the cruise for planning purposes

plots – auto-generated plots from the post processing pipeline

products- The best at sea derived data products from the cruise organized by dive number. May also contain extra data products specific to the cruise, such as multiple dive combined bathymetry.

raw-usbl- Log and configuration files from the Sonardyne USBL system

svp- Sound velocity profiles used during the cruise

Of these directories, dives and products hold all of the normal data products generated during the cruise. Cruise-specific data products are outlined in the At-Sea Processed Data Products section.

2. At-Sea Processed Data Products: <CruiseID>/products

The products directory contains a directory for each dive. Most data products include a time and date stamp in the file name. For photos, that is the time the image was taken, for all other products that is the time of the renavigation process and can be matched to other files created with the same navigation.

Most, if not all, of the files present in the products directory for a dive are copies from another location, most often from the corresponding <CruiseID>/dives/sentryXXX directory. The main purpose of the products structure is to separate a final product (such as a multi- beam grid file) from the raw and intermediate data (such raw multibeam swath data). Some data is not duplicated here due to file size considerations, the most notable example being the processed photos which are only present in <CruiseID>/dives/sentryXXX/photos/proc. Within each dive directory the following directories are included:

products

• sentryXXX
  • blueview (processed blueview data, usually empty)
  • hf-sss (processed high-frequency 540kHz data from the Edgetek sidescan)
  • lf-sss (processed low-frequency 120kHz data from the Edgetek sidescan)
  • sbp (processed sub-bottom profiler data data from the Edgetek sidescan)
  • multibeam (processed data from the Kongsberg EM2040 multibeam)
  • photos (processed digital camera still photos)
  • plots (Plots of timeseries data generated during dive)
  • scc (Navigation and post-processed data for selected instruments)

2.1 Blueview Forward-looking Sonar: <CruiseID>/products/sentryXXX/blueview

This contains any data products created from the blueview sonar. The blueview sonar is a forward looking sonar, operating at 900KHz with a 90 degree field of view. Products are not normally created even if that sonar is installed and this directory is often empty.

2.2 High-frequency Sidescan Sonar: <CruiseID>/products/sentryXXX/hf-sss

This directory contains data products generated from the 540kHz sidescan sonar system. hf- sss stands for high frequency side scan sonar. Note that for a particular survey it is typical to have only HF or LF products, not both. Data in this folder is a subset of the files in <CruiseID>/dives/sentryXXX/sss-sbp/hf-sss

hf-sss

• geotiffs (Processed single-file sidescan data in geotiff format)
• mosaics (Mosaic surveys, combining multiple files and look-directions)

2.3 Low-frequncy Sidescan Sonar: <CruiseID>/products/sentryXXX/lf-sss

This directory contains data products generated from the 120kHz or 230kHz sidescan sonar system. lf-sss stands for low frequency side scan sonar Note that for a particular survey it is typical to have only HF or LF products, not both. Please see the associated cruise report for information on whether the 120kHz or 230kHz frequency was used. Data in this folder is a subset of the files in <CruiseID>/dives/sentryXXX/sss-sbp/lf-sss

lf-sss

• geotiffs (Processed single-file sidescan data in geotiff format)
• mosaics (Mosaic surveys, combining multiple files and look-directions)

2.4 Sub-bottom Profiler Sonar: <CruiseID>/products/sentryXXX/sbp

This directory contains SEGY, seismic-unix, and image files derived from the Edgetech sub-bottom profiler. Data in this folder is a subset of the files in <CruiseID>/dives/sentryXXX/sss-sbp/sbp

sbp

• images (PDFs of trace plots for each SEG-Y file) segy (Extracted sub-bottom data in SEG-Y format)
• su (SEG-Y data converted to Seismic Unix format)

2.5 Kongsberg EM2040 Multibeam: <CruiseID>/products/sentryXXX/multibeam

This directory contains the data products from Sentry’s Kongsberg EM2040 multibeam sonar including grd and pdf files. The multibeam sonar operates at 400KHz with an additional 200KHz option that is rarely used.

sentryxxx_yyyymmdd_hhmm_tide_XxX_BVXX.grd

where X is the grid size. If _nav_ is included in the file name this means that mbnavad- just was applied. This is not common but if available these files are probably preferred to others. Data in this folder is a subset of the files in <CruiseID>/dives/sentryXXX/multibeam

multibeam

• images (PDFs, EPSs, PNGs of gridded bathymetry)
• grids (Processed gridded bathymetry)

2.6 Digital Still Photos: <CruiseID>/products/sentryXXX/photos

This directory contains thumbnails and movies of the photos collected by Sentry. Full resolu- tion photos can be found in the dives directory. The movies created by post processing is the collection of photos combined to make a movie. Sentry does not take continous video during the dive. The photo thumbnails are created to give an alternative to the larger ˜24Mbyte .tif files with a smaller easier to load ˜200kbyte .jpg image file. Data in this folder is a subset of the files in <CruiseID>/dives/sentryXXX/photos

photos

• movies (Video of processed images)
• thumbnails (Smaller sized thumbnail images of processed images)

2.7 SCC <CruiseID>/products/sentryXXX/scc

SCCs are 1Hz ASCII files containing post processed navigation and selected other science data. The timestamps on the SCCs can be matched to other data products. This flat ASCII file contains the date, time, latitude, longitude, depth, pressure, heading, altitude, obs- optitcal back scatter, eh, conductivity, temperature, sound speed, ORP sensor (if installed) and data from all three magnetometers (if installed). The file name contains both the dive number and the date on which the scc file was generated. If there are multiple scc files for a single dive, use the file with the most recent date. All fields in the scc file have been interpolated onto a 1 second time base. Users wanting to load the data into Matlab should use the mat files in the nav-sci directory. Data in this folder is a subset of the files in <CruiseID>/dives/sentryXXX/nav-sci/proc

3 Raw and Intermediate Data: <CruiseID>/dives

The <CruiseID>/dives directory contains the raw and intermediate data for each dive. Within the dives directory there will be a directory for each dive labeled as sentryxxx. Typi- cally there will also be a directory labeled decktest that contains assorted data from tests conducted prior to the first dive, or during days at-sea without dives.

3.1 Blueview Forward-looking Sonar: <CruiseID>/dives/sentryXXX/blueview

This directory contains the raw data from the blueview sonar if installed and active. The blueview is stored in the following directory structure:

blueview (data files produced by driver)

• son (raw data files produced by sonar)

3.2 Instrument Metadata: <CruiseID>/dives/sentryXXX/metadata

This directory contains plain-text files with sensor-specific metadata (e.g. serial numbers, configuration parameters, etc.). These files are generated during Sentry’s normal pre-dive process and are mostly of use to the Sentry team to ensure the sensors are set up correctly before the dive. However, some instruments may report information useful for science (such as instrument calibration coefficients). The list of sensors covered by this procedure is not a complete inventory of the sensors on board Sentry for the dive. The following sensors report information:

• APS1540 Magnetometers
• Seabird SBE49 CTC
• Sonardyne Avtrak USBL beacon
• RDI DVL
• Sentry’s acoustic trigger
• Sentry’s analog instrument digitizers (a2d2 pods)

3.3 Kongsberg EM2040 Multibeam: <CruiseID>/dives/sentryXXX/multibeam

We provide the raw and processed data in several formats. The most common products used by our scientific collaborators are 2D plots (pdf, ps, png) and gridded bathymetry in GMT- compatible grd files and asc files for input into other GIS tools. The gridded data has been edited using our automated scripts and the soundings geolocated using our post-processed navigation. MB-system or Caris tools can be used by the client to hand-edit soundings, however our automated scripts work well. We also provide the edited data in fbt format, which can be imported directly to Fledermaus for griding and display.

The multibeam is stored in the following directory structure:

multibeam

• raw (raw data files from the multibeam sonar)
• log (log files from the reson driver that configures the reson subsea) timing test (separate directory used to compute or check timing offsets)
• proc (all intermediate MB-system files, parameter files used during processing, and 2D and gridded multibeam products)
• nav (vehicle navigation data)
• extra folders may exist with additional processing

3.4 Sentry time-series data: <CruiseID>/dives/sentryXXX/nav-sci

This directory contains all of the navigation, science, and engineering data logged by the vehicle during the dive. Most of this data is provided for archival purposes only (the scc files provide all standard sensor and vehicle navigation data. Users wishing to load data into Matlab, can use the mat files in /proc.

The navigation and science files are stored in the following directory structure:

nav-sci

• proc (Processed navigation and science files)
• raw (Raw navigation and science files)
  • mc (mission controller files)
  • rosbag (ROS data logs, these is the ‘‘raw’’ data)
  • topside-nav (Filtered ship-board navigation data)
  • roslog (ROS status logs)

3.4.1 Renavigated Vehicle Data: <CruiseID>/dives/sentryXXX/nav-sci/proc

The final vehicle data product (e.g. vehicle position, non-sonar instrument data, etc.) is a large number of Matlab data structures in the <CruiseID>/dives/sentryxxx/nav-sci/proc folder. These files can be grouped into two classes: Raw data converted from ROS “.bag” files and “renav” files that combine Sentrys “renavigated” position at sensor-specific timestamps (e.g. Sentrys interpolated lat/lon position for each CTD reading). Typical filenames for each group are listed below.

• Vehicle (Engineering) Data:
  • sentry123 mat: Data transmitted/received acoustically
  • sentry123 mat: Vehicle flight controller data
  • sentry123 mat: Vehicle actuator data (thrusters, servos)
  • sentry123 mat: Misc. sensor metadata (e.g. serial numbers)
  • sentry123 mat: Dead-reckoning DVL (vehicle XYZ, altitude, attitude, depth)
  • sentry123 mat: Global vehicle status
  • sentry123 mat: Mission controller goals
  • sentry123 hotel.mat: Sentry hotel data (e.g. batteries, housing temperatures, )
  • sentry123 mat: Sentry deadreckoning navigation
  • sentry123 navpp mat: Parameters applied to Dead-reckoning/USBL filter.
  • sentry123 mat: Sentry position based on USBL fixes
  • sentry123 mat: Topside ship position
• Timestamped Science Data:
  • sentry123 mat: Dive metadata (name, start and end times, etc.)
  • sentry123 mat: CTD data
  • sentry123 mat: depth data
  • sentry123 mat: Attitude (heading, pitch, roll) in radians
  • sentry123 mat: Sound velocity
  • sentry123 mat: Mission block metadata, including timestamps.
  • sentry123 mat: Dead-reckoning position, velocities, and accelerations
  • sentry123 mat: Optical Backscatter
  • sentry123 mat: Oxygen Concentration
  • sentry123 mat: The latitude and longitude for the dive origin
  • sentry123 mat: Oxygen Reduction Potential
  • sentry123 sensors.mat: Raw sensor data, including magnetometers, ctd, orp, obs,
• Navigated Data (vehicle position and attitude interpolated to sensor timestamp):
  • sentry123 YYYYMMDD HHMM mag mat: Magnetometer data
  • sentry123 YYYYMMDD HHMM obs mat: Optical Backscatter
  • sentry123 YYYYMMDD HHMM optode renav.mat: Oxygen Concentration
  • sentry123 YYYYMMDD HHMM orp mat: Oxygen Reduction Potential
  • sentry123 YYYYMMDD HHMM mat: Gyro (attitude) with position.
  • sentry123 YYYYMMDD HHMM sbe49 mat: CTD
  • sentry123 YYYYMMDD HHMM svp mat
• Miscellaneous Data:
  • sentry123 YYYYMMDD HHMM.scc: Many data streams interpolated to com- mon 1Hz clock in a plain text format. This file is also copied into the scc folder in the products directory
  • sentry123 YYYYMMDD HHMM scc.mat: Same data as the .scc file, but in a Matlab structure
  • sentry123 ascent svp.pro: Sonardyne Ranger compatable sound velocity profile generated during vehicle descent.
  • sentry123 descent svp.pro: Sonardyne Ranger compatable sound velocity profile generated during vehicle ascent.

3.4.2 Common Matlab Structure Fields

Many of the Matlab structures in the above files have fields names by convention (times- tamps, for example). Below is a list of the most common:

• hdr t: Sensor data timestamp in POSIX time (seconds since 01-01-1970 UTC).
• hdr datetime: Same time as hdr t, but converted to a Matlab datetime
• bag t: Timestamp data was saved to file (may be different if sensor provides its own timestamp)
• bag datetime: Same as bag t, but converted to a Matlab datetime t: same as hdr t

3.5 Digital Still Photos: <CruiseID>/dives/sentryXXX/photos

We provide images in several formats with different levels of processing. These include the raw bayer encoded (color) tif files directly from the camera real-time software should users choose to reprocess those images. We also provide automated processing for color compensation and equalization. Filenames include date and time and can be used in conjunction with the SCC to obtain information on vehicle state and scientific sensors.

The photos are stored in the following directory structure:

photos

• raw (Bayer encoded original images) proc (color corrected color tif photos)
• movie (contains movies in .mp4 formats)
• thumbnails (reduced resolution jpgs of each processed image)

Presently, Sentry takes photos during the planned camera surveys and in the event that the dive ends with a photo survey, also during the ascent. Thus there will be photos of the water column.

3.6 Time series and Navigated plots: <CruiseID>/dives/sentryXXX/plots

Much of the scalar (e.g. non-sonar, non-image) timeseries data recorded and processed during a dive is plotted during normal post-dive processing for basic Q/A purposes. A smaller section of data is also plotted against vehicle position, giving a rough estimate of spatial values during a dive. This directory contains a mixture of both vehicle engineering plots, (e.g. thruster output current vs. time), and science plots (e.g. CTD Temperature vs time). These plots can serve as a high-level overview for the dive data and can help narrow down what time ranges warrant further investigation.

3.7 Sidescan and Sub-bottom Profiler: <CruiseID>/dives/sentryXXX/sss-sbp

All of the data from Sentry’s sub-bottom profiler and sidescan sonar We provide the raw and processed Edgetech sonar data. This data is processed using MB-System. For each dive, there is a folder containing the raw data (jsf) files, a nav directory for navigation the jsf files, and lf-sss=120kHz, hf-sss=230kHz, vhf-sss=540kHz, and sbp=Chirp Subbottom data processing directories.

The side scan data is stored in the following directory structure:

• sss-sbp
  • hf-sss (processes high frequency sidescan
    • geotiff (raw grd tiff files images with turns and not merged into full length track lines
    • images (hf image files)
      • flt (Raw data images after removing turns but before merging the data into full length track lines)
      • raw (raw data images before removing turns and merging the data into full length track lines)
      • tracklines (postscripts and pdfs of merged tracklines with turns removed)
        • raw (postscripts and pdfs of merged tracklines with turns included)
• lf-sss (processes low frequency sidescan)
  • geotiff (raw grd tiff files images with turns and not merged into full length track lines
  • images (hf image files)
    • flt (Raw data images after removing turns but before merging the data into full length track lines)
    • raw (raw data images before removing turns and merging the data into full length track lines)
    • tracklines (postscripts and pdfs of merged tracklines with turns removed)  raw (postscripts and pdfs of merged tracklines with turns included)
• nav (processed nav data)
• proc (common processed data and config) raw (raw sidescan and sub bottom data)
• sbp (Processed sub bottom data)
• sonarwiz (Processed sidescan data from sonarwiz program, Refer to SonarWiz user guide for additional details, See sonarwiz folder outline on the next page)

3.7.1 Sonarwiz project structure

The sonarwiz data is stored in the following directory structure:

sonarwiz

• 3DView (The default location of the files created in the 3D export process)
• Backup (Contains backup files of project data that are automatically created by SonarWiz)
• CSF (Contains SonarWiz Common Sensor Format (CSF) files that are created either in recording raw data or when importing sonar data files into the program)
• DEP (Empty, not typically used)
• GeoTiff (GeoTiff image files generated by Save Project as GeoTiff function)
  • Tiles (prj tfw and tif files)
• Images (Default directory for Batch Image Export from Sonar File Manager)
• jsf (Navigation injected jsf files used for processing, created by Sentry post processing)
• MAG (Contains CMF magnetometers files)
• Misc (Not currently used by sonarwiz)
• Rasters (Not currently used by sonarwiz)
• SBP (Empty, not typically used)
• SHP (Default directory for exported shp files from the Grid and Contour Utility)
• Targets (Contains data and image files for all digitized targets)
• Tmp (Used by some of the utility programs to store temporary files)
• Web (Contains the html and associated image files generated by the Save Project as Website function)
• XTF (Contains raw sonar files created during Real-time data recording)
• sentryXXX mml (Sonarwiz project file)

3.8 Vehicle Acoustic Communication:
<CruiseID>/dives/sentryXXX/subsea-acomm

This folder contains logging output from the driver for Sentry’s Micro-Modem acoustic mo- dem. It is not generally of importance to science.

3.9 Topside (ship) Communications:
<CruiseID>/dives/sentryXXX/topside-comms

This folder contains logging output from a number of software drivers running on the ship that monitor communication pathways with Sentry while underway. It is not generally of importance to science.

topside-comms

•  acomm (ship-side Micro-Modem acoustic modem logs)
• iridium (ship-side iridium beacon logs)
• sdyne (ship-side Sonardyne SMS logs)

4 Cruise-specific Data Products

4.1 HD Video: <CruiseID>/products/sentryXXX/media/

This directory contains video files collected by the onboard MISO camera system.

media

• <CruiseID> MISO-data-summary (MISO videos summary)
• <CruiseID>-DXXX-MISO-video (MISO video directory)

A Required Software

Sentry produces a large amount of data for each dive from a wide variety of instruments. Most of the processed data is viewable by common, free programs. However, the raw data formats may require proprietary software to view. Usually this software, (or another with similar capabilities) is freely distributed by the hardware vendor upon request.

• blueview: The ProViewer program available from BlueView and can replay raw sonar
• multibeam: The Kongsberg multibeam produces .kmall files that can be processed using MBSystem or other commercial software Sentry processes multibeam data at sea with MBSystem and the resulting .grd files are compatible with many GIS programs such as ArcGIS, QGIS, etc.
• sidescan: The Edgetech sidescan produces .jsf files that can be processed using Sonar- wiz or other commerical software packages. Sentry processes sidescan data with Sonar- wiz at sea, producing GeoTIFF images that can be opened in any image
• raw vehicle data: Sentry uses the ROS software framework and logs all raw vehicle data in ROS .bag The raw data is not typically used by Science.
• processed vehicle data: Sentrys post-dive processing of vehicle data creates a large number of Matlab structures, each of which is saved in its own file in the <CruiseID>/dives/sentryXX/nav-sci/proc folder. For Sentry’s chemical sensors (e.g. ORP, Optode, CTD) these files are the main processed data product.

B Coordinate origins

The vehicle’s control system uses simple equidistant coordinates. This system uses an origin, defined in terms of latitude and longitude (WGS84), and a fixed scaling between meters displacement from the origin. We use the identical routines that have been used by the NDSF assets Alvin and Jason for decades. These simple coordinates have several advantages for realtime control of a vehicle. Unlike UTM grid coordinates, the x and y axes intersect at right angles and align with true east and north respectively at the origin. These coordinates distort quickly as one moves away from the origin, but we solve that problem by putting the origin close to the operating area. We almost always report our results in latitude/longitude, so most users need not be aware of these details.

Individual Origins for each dive can be found in the dive summaries appended to the cruise report.

C Data Receipt

Prior to departure from <Port Name>, the Sentry team delivered one copy of the entire raw and processed data to <Chief Scientist> of <Institution>.

This data delivery comprises the final deliverable of both raw and processed data from the Sentry AUV. The data structure on the dive is described in detail in this report. Additional copies of the data will be transported back to WHOI for archiving.

D Cruise Data Totals

Table 1: Tally of Sentry data obtained on <CruiseID>

Data	Number of Files	Size (bytes)
Total	105097	2.0T
svp	0	4.1K
docs	58	972.3K
planning	108	6.9M
planning-bathy	11	1.8G
products	5321	711.4G
raw-usbl	212	2.8G
dives	80745	1.2T
engineering	18642	111.1G