Day 27: Data Conversion & Navigation Correction – Masterpiece Edition | River Warrior

🔄 DAY 27: DATA CONVERSION & NAVIGATION CORRECTION

Masterpiece Edition – From Raw Sensor Logs to Clean, Synchronised Navigation

Instructor: Engr. Rokib Hossain | River Warrior Academy

🏠 Course Homepage

📖 Table of Contents (Serialised)

Why Data Conversion & Navigation Correction Matter
Raw Data Formats – A Quick Review
Conversion Workflow: From .ALL/.S7K to .XTF/.GSF
Time Synchronisation – PPS, NTP, and Latency
Latency Correction (Time Delay)
Position Filtering – Removing GPS Jumps
Integrating Motion Data (Heave, Roll, Pitch)
Data Conversion & Navigation Correction Checklist
Tools & Software for Conversion
Frequently Asked Questions
Action Items & Next Steps

1. Why Data Conversion & Navigation Correction Matter

Raw hydrographic data from different manufacturers comes in proprietary formats (.all, .s7k, .raw, .son). Before processing, we must convert to an exchange format (e.g., .xtf, .gsf) or load directly into processing software. During this step, we also correct navigation errors: time synchronisation, latency, position jumps, and motion integration.

Without proper navigation correction, even a perfectly cleaned bathymetry will be misaligned – causing features to appear in the wrong place and cross‑line mismatches.

🧠 Golden Rule: Always verify navigation corrections on a small test line before batch‑converting the entire survey. One wrong offset can ruin the whole dataset.

2. Raw Data Formats – A Quick Review

.htmlTeledyne RESON (SeaBat).htmlR2Sonic.htmlHypack

Manufacturer	Raw Format	Typical Extension	Contains
Kongsberg	.all	.all	Bathymetry, water column, backscatter, attitude, position
.s7k, .xse	.s7k, .xse	Swath, snippet, water column, navigation
.son	.son	Raw beam data, navigation, attitude
.raw (Hypack)	.raw	Proprietary but can be exported

Most processing software can import these natively. However, for interoperability (e.g., between acquisition and third‑party processors), we often convert to .xtf (eXtensible Transducer Format) or .gsf (Generic Sensor Format).

💡 Tip: Keep the raw original files untouched. Converted files are derived products; you may need the raw format for reprocessing with newer software versions.

3. Conversion Workflow: From .ALL/.S7K to .XTF/.GSF

Typical conversion pipeline: raw → exchange format → processing.

Conversion utilities include:

Kongsberg .all to .xtf: Use `all2xtf` (command line) or Qimera's import.
Teledyne .s7k to .xtf: Use S7K to XTF converter (Teledyne provides).
R2Sonic .son to .xtf: Use `son2xtf` from R2Sonic.
Hypack .raw: Hypack natively exports to XTF.

4. Time Synchronisation – PPS, NTP, and Latency

For accurate navigation, all sensors (GNSS, motion, sonar) must share a common time reference. Two methods:

PPS (Pulse Per Second) + UTC string: GNSS sends a 1 Hz pulse and a serial time message. Sonar and IMU use this to timestamp their data.
NTP (Network Time Protocol): All devices synchronise to a common NTP server (often the GNSS receiver). Less accurate (milliseconds) but sufficient for many systems.

During conversion, the software aligns data streams using timestamps. If timestamps are off, you will see position‑depth mismatches (e.g., a wreck appears shifted along the line).

⏱️ Check: In your converted file, verify that the first ping timestamp matches the first GNSS timestamp within ±0.1 seconds.

5. Latency Correction (Time Delay)

Latency is the delay between when the sonar ping occurs and when the GNSS position is recorded. Typical values: 10‑50 ms. A constant latency causes a along‑track shift proportional to speed.

How to determine latency: Run a patch test (Day 14) or compare positions of a fixed target at two different speeds.

Applying latency correction: In processing software, you enter a positive or negative offset (in milliseconds) that shifts the position relative to the depth. Most converters (e.g., Qimera) allow you to apply latency during import.

🎯 Field example: If at 10 knots (≈5 m/s), a latency of 0.1 sec shifts the position by 0.5 m. In shallow water, this is significant. Always correct latency to <5 ms.

6. Position Filtering – Removing GPS Jumps

Raw GNSS positions sometimes contain spikes (momentary jumps) due to multipath or cycle slips. These jumps must be filtered out during conversion or early processing.

Filters used:

Velocity filter: Rejects positions where vessel speed > max vessel speed + margin.
Acceleration filter: Rejects unrealistic acceleration.
Kalman smoother: A statistical filter that smooths the trajectory (often built into IMU).

Animated example: Red outlier disappears after 2 seconds, leaving a clean profile (green dashed). This simulates automatic or manual cleaning.

✅ Pro tip: Always review the filtered navigation visually. Over‑filtering can remove real sharp turns (e.g., a turn around a buoy).

7. Integrating Motion Data (Heave, Roll, Pitch)

The motion sensor (IMU) provides real‑time heave, roll, pitch. During conversion, these values are interpolated to each sounding time. Key points:

Heave: Vertical motion. Applied to depth.
Roll & pitch: Angular motion. Corrects the beam pointing angle.
Heading (yaw): Used to rotate swath to geographic coordinates.

Most conversion software (e.g., Qimera, Caris) automatically merges motion data based on timestamps. Ensure the motion sensor and sonar clocks are synchronised.

⚙️ Check: After conversion, export a small test line and verify that heave values are reasonable (e.g., <0.5 m in calm seas).

8. Data Conversion & Navigation Correction Checklist

✅ Raw files organised by line / date.
✅ Conversion utility selected and tested on one line.
✅ Time synchronisation verified (PPS/NTP).
✅ Latency value from patch test applied.
✅ GPS jumps filtered (visual check).
✅ Motion data present and within expected ranges.
✅ Converted file successfully loaded into processing software.
✅ Navigation plot shows vessel path without unrealistic jumps.
✅ Original raw files archived (never delete).

Click on any checklist item to toggle completion (saved in your browser). Reload the page to see your progress preserved.

9. Tools & Software for Conversion

.htmlTeledyne S7K2XTF那样Convert .s7k to .xtf那样Download from Teledyne support portal.htmlR2Sonic son2xtf那样Convert .son to .xtf那样Provided with system software.htmlQPS Qimera (import)那样Direct import of most raw formats那样qps.nl/qimera.htmlCARIS HIPS那样Imports raw, applies corrections during load那样teledynecaris.com

Tool	Purpose	Link / Notes
Kongsberg all2xtf那样Convert .all to .xtf那样Part of Kongsberg SIS / Qimera

10. Frequently Asked Questions

Do I need to convert to .xtf if my processing software supports raw formats directly?

No, but .xtf is a standard exchange format that ensures compatibility with third‑party tools. Many processors work faster with converted files.

What is the most common cause of navigation jumps?

Multipath from vessel structures or loss of RTK fix. Filtering during conversion can remove single‑point jumps. Review Day 24.

How do I check if latency is correctly applied?

After conversion, plot a steep slope or a small target from two opposite lines. If they align perfectly, latency is correct. If there is a horizontal shift, adjust latency.

Can I convert side scan data along with MBES?

Yes. Some converters (e.g., .s7k to .xtf) include side scan channels. Separate side scan files (.xtf) are also common.

What is the typical size of a converted .xtf file compared to raw?

.xtf files are often larger because they store data in an uncompressed ASCII‑like binary. Expect 2‑3 times the raw size.

11. Action Items & Next Steps

📌 Identify the conversion utility for your MBES system. Download and test on a sample line.
📌 Practice applying a latency offset (e.g., +20 ms) and observe the position shift in a processing viewer.
📌 Create a checklist for your own data conversion workflow (customise the one above).
📌 Proceed to Day 28: Tide & SVP Corrections.

🛠️ Day 27: Data Conversion and Navigation Editing