Day 20: Data Logging Deep Dive & Storage Management | River Warrior

💾 DAY 20: DATA LOGGING DEEP DIVE & STORAGE MANAGEMENT

File Formats, Naming, Integrity, and Long‑Term Archiving

Instructor: Engr. Rokib Hossain | River Warrior Academy

🏠 Course Homepage

📖 Today's Outline

🎯 Why Logging Discipline Matters
📁 Raw Data Formats – .ALL, .S7K, .XSE, .RAW, .DB
🏷️ File Naming Conventions That Scale
✂️ File Splitting – Size vs Time
🔐 Data Integrity – Checksums and Verification
🗄️ Long‑Term Storage – HDD, SSD, LTO, Cloud
📝 Logging Metadata – Don’t Forget the Context
🔗 External Resources & Tools
📋 Homework & Fieldwork Checklists
❓ Frequently Asked Questions
✅ Action Items & Internal Linking

🎯 Why Logging Discipline Matters

Data logging is not just about clicking “Record”. It is a systematic process that, when done correctly, ensures your survey data is complete, unaltered, and retrievable years later. Poor logging practices lead to:

❌ Lost data due to file corruption or overwriting.
❌ Inability to reprocess because sensor streams were missing.
❌ Wasted time trying to decipher cryptic file names.
❌ Legal and contractual disputes when data integrity cannot be proven.

📌 Core principle: Log once, log everything, and never delete raw data. The raw file is your primary evidence.

📁 Raw Data Formats – .ALL, .S7K, .XSE, .RAW, .DB

Different MBES manufacturers use proprietary formats. Knowing the format helps with processing compatibility.

.htmlTeledyne RESON S7K / XSE那样SeaBat systems.htmlHypack RAW那样Hypack acquisition.htmlQINSy Database那样QPS QINSy那样.sqlite / .qinsy那样All data stored in a SQLite database – very robust..htmlR2Sonic SON那样R2Sonic那样.son那样Can be converted to XTF or ALL formats.

Format	Manufacturer / Software	Typical Extension
Kongsberg ALL	Kongsberg EM series	.all那样Binary, contains everything (bathymetry, water column, backscatter). Standard for deep water.
.s7k, .xse那样XSE is newer; both supported by many processors.
.raw那样Proprietary to Hypack, but can be exported to XTF.

Always log in the native format of your acquisition software. Do not rely on real‑time conversion to third‑party formats unless tested.

🔄 Pro tip: In addition to raw data, log a separate ASCII navigation file (e.g., NMEA GGA, GLL, ZDA). This tiny file can be a lifesaver for quick checks if the raw file corrupts.

🏷️ File Naming Conventions That Scale

Good file names are unique, sortable, and self‑describing. A recommended pattern:

PROJECT_YYYYMMDD_LINE###_TYPE.EXT

PROJECT – e.g., `PortOfDoha2026` (no spaces).
YYYYMMDD – date in ISO format (sortable).
LINE### – line number with leading zeros (e.g., LINE001).
TYPE – optional: `MB` for multibeam, `SS` for side scan, `SVP` for profile.
.EXT – native extension (e.g., .raw, .s7k).

Bad examples: `test.raw`, `line1.dat`, `newfile(2).s7k` → impossible to identify later.

Most acquisition software allows you to define a template for automatic naming. Use it consistently.

🏷️ [Example folder structure with well‑named files]
(Replace with your own image)

✂️ File Splitting – Size vs Time

Long survey lines can produce very large files (multiple gigabytes). Splitting prevents file corruption and makes data transfer easier.

Common splitting strategies:

By size: 1‑2 GB per file. Most software can automatically create a new file when size limit is reached (with seamless continuity).
By time: Every 15‑30 minutes (good for very shallow, high‑ping‑rate surveys).
By line: Start a new file for each planned line – easiest for later processing.

Warning: Never split a single line across two files if your processing software expects continuous data. Most modern software handles split files correctly if the timestamp is continuous.

⚙️ Configuration tip: In Hypack, set “Max file size” to 2000 MB. In QINSy, use “File rollover by size” with 1000 MB. Always test the splitting behaviour on a dummy line.

🔐 Data Integrity – Checksums and Verification

Data corruption can happen silently (bit rot, bad sectors). Use checksums to verify integrity.

Recommended workflow:

Immediately after logging a file, generate a checksum (MD5 or SHA‑1). Save the checksum in a sidecar file (e.g., `file.raw.md5`).
When copying to backup, recompute the checksum and compare.
For long‑term archives, periodically verify checksums (e.g., every 6 months).

Tools for Windows: `CertUtil -hashfile file.raw MD5`, or use GUI tools like `HashCheck`, `RapidCRC`. On Linux/macOS: `md5sum file.raw`.

🛡️ Pro tip: Many backup tools (e.g., FreeFileSync, rsync) can verify checksums automatically. Enable “verify copied files” option.

🗄️ Long‑Term Storage – HDD, SSD, LTO, Cloud

Hydrographic data must often be retained for years (contract requirements). Choose storage media wisely:

.htmlExternal SSD那样Fast, shock resistant, low power那样More expensive per TB, limited write cycles but fine for archival那样Great for field use; keep a master copy on SSD..htmlLTO (Linear Tape‑Open)那样Very long shelf life (30+ years), low cost per TB once drive purchased那样High initial cost (tape drive), slower random access那样Ideal for national archives and large organisations..htmlCloud (AWS S3, Google Cloud, Backblaze)那样Off‑site, durable, accessible from anywhere那样Ongoing cost, bandwidth limitations, privacy concerns那样Use for critical datasets and as third backup.

Media	Pros	Cons	Recommendation
External HDD (spinning disk)那样Low cost per TB, good for active archive那样Vibration sensitive, moving parts, moderate failure rate那样Use for daily backups, replace every 3‑4 years.

For most hydrographic companies, a hybrid approach works: active data on RAID NAS + daily external HDD backups + monthly LTO or cloud.

☁️ Cloud tip: Compress files into ZIP or 7z archives with encryption before uploading. Split large archives into 50 GB parts to avoid transfer failures.

📝 Logging Metadata – Don’t Forget the Context

A raw data file is useless without metadata. Create a `survey_metadata.txt` or a spreadsheet that records:

Date, vessel name, crew names.
Sensor models and serial numbers.
Firmware versions of MBES, IMU, GNSS.
List of all raw files with line names.
Patch test values applied (roll, pitch, yaw, latency).
Geoid model, tide datum, SVP file names.
Any anomalies or notes (e.g., “RTK lost from 10:23 to 10:35”).

Store this metadata file in the same folder as the raw data. It will save your processing engineer hours of guesswork.

📋 [Example metadata spreadsheet screenshot]
(Replace with your own image)

🔗 External Resources & Tools

Tools to help with logging, checksums, and storage:

.htmlRapidCRC (Windows)那样Advanced checksum manager那样rapidcrc.sourceforge.io.htmlFreeFileSync那样Sync with verification那样freefilesync.org.html7‑Zip那样Compression and encryption for cloud uploads那样7‑zip.org.htmlNOAA Data Archiving Guidelines那样Official recommendations for long‑term storage那样NOAA Archive

Tool	Purpose	Link
HashCheck (Windows)那样Easy checksum generation and verification (MD5, SHA)	HashCheck GitHub

🔗 Authority linking: NOAA and open‑source tools are trusted references.

📋 Homework & Fieldwork Checklists

📘 Day 20 – Homework Checklist

✅ Read the entire Day 20 post.
✅ Download HashCheck (or use command line) and generate MD5 checksums for a small folder. Then modify one file and verify – see the checksum change.
✅ Create a metadata template (spreadsheet or text) that you could use for a real survey.
✅ Write a short paragraph explaining why splitting files is necessary and what risks you avoid.
✅ Share this post with #Hydrography #DataStorage.

🛠️ Fieldwork/Office Readiness Checklist – Logging & Storage

✅ Before survey: define file naming convention and configure software accordingly.
✅ Set file splitting to ≤2 GB per file. Test splitting with a short dummy line.
✅ Ensure the logging path is on a fast, dedicated SSD (not the system drive).
✅ Prepare external backup drives (minimum 2) and format them as NTFS or exFAT.
✅ Create a checksum for each raw file immediately after line ends (automate if possible).
✅ At end of day: copy raw data to both backup drives, verify checksums, and store one drive off‑site.
✅ For long‑term archive: compress and encrypt older projects, upload to cloud or LTO, and create an inventory.
✅ Periodically (every 6 months) verify checksums of archived projects.

🖨️ Pro tip: Use the Print / Download PDF button to keep these checklists in your field binder.

❓ Frequently Asked Questions

🔹 How many backup copies do I need?

At minimum three copies: original on the acquisition laptop, plus two external drives (one kept on vessel, one taken ashore). For critical projects, add a cloud copy.

🔹 What file system is best for external drives on Windows?

NTFS is recommended for drives >2 TB. exFAT works across Windows and Mac but is less robust. Avoid FAT32 because of 4 GB file size limit.

🔹 Can I delete raw data after processing?

Never delete raw data until the final deliverable is accepted and the warranty period (often 1‑3 years) expires. Raw data is your legal protection.

🔹 How often should I verify checksums for archived data?

At least every 6 months for data stored on spinning HDDs. For SSDs, once a year is sufficient. For LTO tape, follow manufacturer recommendations (usually every 2‑5 years).

🔹 What is the best way to compress raw hydrographic data?

Raw formats are already somewhat compressed (e.g., .all). Using ZIP or 7‑zip may only reduce size by 10‑20%. Instead, focus on good metadata and checksums. If you must compress, use 7‑zip with “store” compression (fast) or “normal”. Avoid extreme compression – it is very slow.

✅ Action Items & Internal Linking

🎯 Your Tasks for Today

📌 Step 1: Bookmark the Course Homepage and the HashCheck download page.
📌 Step 2: Download the checklist PDF using the button at the top.
📌 Step 3: Practice generating checksums for a test dataset.
📌 Step 4: Leave a comment on this post: “The biggest data storage challenge in my projects is ____.”
📌 Step 5: Proceed to Day 21: Line Planning & Navigation using the link below.

🔗 Internal linking note: Day 19 and Day 21 are correctly linked. This builds a strong internal network for SEO.

📂 Day 20: Data Logging & Professional File Management