LanceDB table on disk: 265 versions, 13.4 GiB, one live fragment
What a LanceDB table holds on disk: 265 versions, 13.4 GiB, one live fragment. Manifests and transactions decoded on a real table.
I ran a read-only script on my own machine to look inside a LanceDB table, and an AI assistant (Claude, from Anthropic) helped me write that script and draft this article. The numbers below are the real output of that run, with LanceDB 0.41.0 and pyarrow 26.0.1. I replaced the table name and the paths, and I translated the labels of the output from French. This is the first article of a series on LanceDB seen as a DBA.
Quick answer
A LanceDB table is a directory, and nothing else: data files in data/, one manifest per version in _versions/, one transaction file per commit in _transactions/, and an _indices/ directory once you build an index. My table holds 1,130,595 rows of 1024-dimension vectors and weighs 13.4 GiB on disk, but its current version, number 265, references a single 5.8 GiB data file. The other 7.6 GiB or so belong to 259 older data files that no current version needs, and they stay on disk until someone cleans up the old versions. Every append adds a new data file and a new version, which is how a table reaches 265 versions and 260 data files, and the last version merged 226 fragments into one without removing the old files.
What is in a LanceDB table directory
The first part of my script walks the LanceDB table directory, docs_index.lance, and needs no LanceDB at all. The tree has three subdirectories. The layout documentation lists others, and two of them, _indices/ and _deletions/, are absent from my table.
| Directory | What the documentation says it holds | What I found |
|---|---|---|
data/ | data files (*.lance) | 260 files, 13.4 GiB, median 21.6 MiB, largest 5.8 GiB |
_versions/ | manifests (*.manifest) and an optional latest_version_hint.json | 265 manifests, 2.4 MiB in total, plus a 15-byte hint file |
_transactions/ | transaction files (*.txn) | 265 files, 56.7 KiB in total, median 132 bytes |
_indices/ | one folder per index | absent |
The table has 791 files in all. The statistics report num_indices: 0 and num_deleted_rows: 0, which matches the absence of those two directories: I never built an index and never deleted a row.
A version is a manifest and a transaction
The manifest is the heart of the format. The documentation describes it as one immutable description of a single version, with the schema, the list of fragments, the version number, a timestamp and a reference to the transaction that created it. My table has 265 manifests and 265 versions, so each version has exactly one manifest, and 265 transaction files, so each commit left one as well.
The names carry information. A transaction file is named {read_version}-{uuid}.txn, where the number is the version the transaction was built from, and my largest one is 263-45522501-…txn at 20.1 KiB. The manifests look odd at first, but the logic is simple: the file for version 265 is 18446744073709551350.manifest. That number is 18446744073709551615, the largest 64-bit unsigned integer, minus 265, and version 1 is 18446744073709551614.manifest. The names therefore count down as versions go up. The documentation pages I read do not explain this scheme, so my reading is only that it makes the newest version sort first in a directory listing. The latest_version_hint.json file is 15 bytes. The documentation calls it “purely an optimization”, there to speed up finding the latest version on stores where listing the directory is expensive.
Every append adds a fragment
The last five versions show how the table grew. list_versions returned these rows, which I trimmed to the interesting fields and aligned as a table:
version timestamp rows data files 261 2026-11-03 22:55:19.615921 1,123,032 224 262 2026-11-03 22:56:55.260015 1,127,160 225 263 2026-11-03 22:58:15.392524 1,130,595 226 264 2026-11-03 22:59:22.144490 1,130,595 226 265 2026-11-03 22:59:22.177593 1,130,595 1
From 261 to 263, each version added one data file and a few thousand rows: 4,128 rows for the first append and 3,435 for the second. That is the Append operation of the documentation, which adds new fragments without modifying existing data. Small appends therefore produce small data files, and a table fed in small batches accumulates fragments: mine had 226 of them, and the median data file is 21.6 MiB.
Versions 264 and 265 are 33 milliseconds apart, and the number of rows does not change. I decoded their transactions with the protobuf definitions published in the Lance repository. Version 264 is a reserve_fragments transaction of 46 bytes: it reserves a fragment id, and the manifest’s max_fragment_id goes from 258 to 259 while the 226 fragments stay the same. Version 265 is a rewrite transaction, the compaction: it replaces the 226 old fragments by one new fragment, id 259, holding all 1,130,595 rows. The documentation calls a Rewrite an operation that “reorganizes data without semantic modification”, and that is what the file shows. Versions 21 and 33 of the same table hold earlier rewrites, of 21 fragments into one and of 6 into one, and the first version is an overwrite of 329 bytes.
The current version of a LanceDB table is smaller than the table
The statistics of version 265 give the size of what the table actually needs:
total_bytes: 6,203,819,954 num_rows: 1,130,595 num_deleted_rows: 0 num_indices: 0 num_fragments: 1 num_small_fragments: 0
That is 5.8 GiB, or about 5,487 bytes per row. The vector column is 1024 floats of 4 bytes each, so 4,096 bytes per row, and the remaining 1,391 bytes per row hold the text, the title, the path and the chunk number. The data/ directory holds 13.4 GiB, which leaves about 7.6 GiB in 259 data files that version 265 does not reference. They are the fragments of the previous versions, still there because nothing removed them. In this table, merging the fragments did not shrink the directory: the new file was added and the old ones stayed in place, still referenced by the older versions.
Why the manifest of a single fragment weighs 20.5 KiB
First, the manifests grow with the number of fragments: 628 and 736 bytes for the first ones, 20.4 KiB at 226 fragments, about 90 bytes per fragment. However, version 265 has one fragment and its file is still 21,031 bytes, and I decoded it to find out why. The file starts with a 4-byte length and a copy of the transaction, then a second length and the manifest proper, then a 16-byte footer ending in the LANC marker. The manifest proper is 391 bytes. The other 20,620 bytes are the embedded transaction, the same rewrite that is stored in 263-45522501-…txn, which lists the 226 fragments it replaces, 88.6 bytes each on average. For comparison, the manifests of versions 263 and 264 are 20.8 KiB of fragment list each, plus a small embedded transaction of 136 and 50 bytes.
So the 20 KiB of the last manifest are not data about the current version. They record what the compaction removed from the table, and they sit twice on disk, in the manifest and in the transaction file. That field is called transaction_section in the protobuf. I could not find that described in the format pages I read.
What I did not test
However, I did not run any write operation on the original table, because I wanted the table untouched: no compaction, no cleanup of old versions, no index. The script only lists files and calls count_rows, schema, list_versions, list_indices and stats. I decoded manifests and transaction files only for versions 1, 21, 33 and 263 to 265, and only on a copy of the table. I also did not measure query time, which depends on the index and comes in a later article.
Official documentation
The Lance table format, its layout, versioning and transactions pages describe what I measured, and the protobuf definitions table.proto and transaction.proto give the field names. The LanceDB project documents the Python API behind the script.
