VulCode
Blog

Sonic: TRIGGER backup works while a store is in use

Backups in the Sonic search backend failed right after data was written, because RocksDB will not open a database that is already open in the same process. The backup now uses the open store.

DualFroz - VulCode CEODualFroz - VulCode CEO2 min read
Sonic: TRIGGER backup works while a store is in use - DualFroz - VulCode CEO
Title in the repository
fix(core): Make `TRIGGER backup` work while stores are in use
Merged
5 Oct 2026
Written up here
6 Oct 2026

The note was written 1 day after the change was merged.

Sonic
Technologies:RustRocksDB

What was broken

Sonic is a fast search backend written in Rust, which its authors describe as an alternative to Elasticsearch that runs on a few megabytes of RAM. It keeps its index in RocksDB (the KV store) and in FST files. The TRIGGER backup command was meant to copy both, but it failed in two situations described in issue #274.

The first was a backup right after writing data with PUSH. Writing opens the RocksDB database and leaves it open in the store pool, and the backup code tried to open it a second time. RocksDB refuses that, because its LOCK file is already held. The second was a fresh server where nothing had been written yet, so the store directories did not exist at all.

What we changed

If the store is already open in the pool, the backup is taken from that instance. Before the backup the data is flushed to disk, because in-memory buffers may hold writes that are not in the files yet:

rust
match pooled_store {
    Some(store) => kv_backup_engine.create_new_backup_flush(&store.database, true),
    None => {
        let origin_kv = self.open(store_id, |_| {})?;
        kv_backup_engine.create_new_backup(&origin_kv)
    }
}

When the KV or FST store directory does not exist yet, the backup simply skips it instead of failing.

How we tested it

The trigger_backup test was previously ignored with a note that the client did not support it, and its body was todo!(). We wrote it from scratch: a backup on an empty server, then a PUSH, then a backup while the store is open, and a check that the backup directory holds exactly one non-empty KV backup.

The result

The change was merged on 5 October 2026. Our second fix (#432) was accepted the same day: since an earlier change, the loop over FST buckets checked the wrong entry, so the backup reported success without writing any FST files and a restore restored nothing.

A lightweight search backend in Rust, an alternative to Elasticsearch that runs in a few megabytes of RAM.

Language
Rust
GitHub stars
21,357
Our merged changes to Sonic: 5
The project's page in Contributions

The same calls before and after the change

The cases of the note and of the test in pull request #430: what should come out, what came out before the fix and what comes out now.

CommandShould giveBefore the changeAfter the changeResult
CommandTRIGGER backupa fresh server, nothing written yet
Should givea backup without an error
Before the changean error: the store directories do not exist
After the changea missing KV or FST directory is skipped
As expected
CommandPUSH, TRIGGER backupthe RocksDB store left open in the pool after the write
Should givea backup of the open store
Before the changeRocksDB refuses a second open, its LOCK file is held
After the changea backup from the pooled instance, after flushing to disk
As expected

2 of 2 cases as expected after the change

What the test checks

A backup on an empty server, a PUSH, a backup while the store is open, and exactly one non-empty KV backup in the backup directory.

The same day

Our fix #432 was merged too: the backup reported success without writing the FST files.

What it taught us

A backup taken while running must capture a consistent state rather than lock the store or read it mid-write. It is the same rule we apply to client data backups.

Questions about change #430

Where the change is, how we tested it and who merged it.

Was this post helpful?