A markdown file's table of contents
markdownduck_block_utils
LOAD markdown; LOAD duck_block_utils;
SELECT level, title
FROM duck_blocks_toc_rows(
(SELECT list(b) FROM read_markdown_blocks('README.md') b)
);
┌───────┬───────────────────────────┐
│ level │ title │
├───────┼───────────────────────────┤
│ 1 │ DuckDB Markdown Extension │
│ 2 │ Features │
│ 2 │ Installation │
│ 3 │ Loading the Extension │
└───────┴───────────────────────────┘
Run in the duckdb_markdown checkout. markdown parses the file; duck_block_utils walks the structure. The reader knows nothing about tables of contents, and the walker knows nothing about markdown.
HTML, through the same type
webbedduck_block_utils
LOAD webbed; LOAD duck_block_utils;
SELECT b.element_type, b.content
FROM (SELECT unnest(
html_to_duck_blocks('<h1>Title</h1><p>Hi there</p><h2>Sub</h2>')
) AS b)
WHERE b.kind = 'block';
┌──────────────┬──────────┐
│ element_type │ content │
├──────────────┼──────────┤
│ heading │ Title │
│ paragraph │ Hi there │
│ heading │ Sub │
└──────────────┴──────────┘
Same element types, same columns, different format. A query written against one reader's output runs against the other's.
The table of contents as of any git commit
duck_tailsmarkdownduck_block_utils
LOAD duck_tails; LOAD markdown; LOAD duck_block_utils;
SELECT level, title
FROM duck_blocks_toc_rows(
(SELECT list(b) FROM read_markdown_blocks('git://README.md@HEAD') b)
);
┌───────┬───────────────────────────┐
│ level │ title │
├───────┼───────────────────────────┤
│ 1 │ DuckDB Markdown Extension │
│ 2 │ Features │
│ 2 │ Installation │
└───────┴───────────────────────────┘
duck_tails registers a git:// filesystem and markdown opens it through DuckDB's FileSystem API, so the reader needs no git support at all -- swap @HEAD for a tag and you get that release's structure. Three extensions, no dependency between them.
An offline Wikipedia article's structure
zimwebbedduck_block_utils
LOAD zim; LOAD webbed; LOAD duck_block_utils;
SELECT level, title
FROM duck_blocks_toc_rows(
(SELECT html_to_duck_blocks(decode(zim_get_content('wikipedia.zim', path)))
FROM read_zim('wikipedia.zim')
WHERE NOT is_redirect AND title = 'Amphibian' LIMIT 1)
);
┌───────┬───────────┐
│ level │ title │
├───────┼───────────┤
│ 1 │ Amphibian │
└───────┴───────────┘
zim serves the article out of the archive, webbed turns its HTML into blocks, and duck_block_utils walks them -- the same walker as the markdown example above. Run against a 100-article mini archive, which is why the outline is short.
A document with no file at all
scalarfswebbedduck_block_utils
LOAD scalarfs; LOAD webbed; LOAD duck_block_utils;
SELECT element_type, content
FROM read_html_blocks('data:,%3Ch1%3EInline%3C/h1%3E%3Cp%3ENo%20file%20needed%3C/p%3E')
WHERE kind = 'block'
ORDER BY element_order;
┌──────────────┬────────────────┐
│ element_type │ content │
├──────────────┼────────────────┤
│ heading │ Inline │
│ paragraph │ No file needed │
└──────────────┴────────────────┘
scalarfs is what serves the data: URL -- controlled by running the same query without it loaded, where it errors. webbed's reader opens whatever path DuckDB can resolve, with no check on the file extension, so a URL carrying no filename works.
A live web page, straight into blocks
httpfswebbedduck_block_utils
LOAD httpfs; LOAD webbed; LOAD duck_block_utils;
SELECT element_type, left(content, 42) AS content
FROM read_html_blocks('https://example.com/')
WHERE kind = 'block'
ORDER BY element_order;
┌──────────────┬────────────────────────────────────────────┐
│ element_type │ content │
├──────────────┼────────────────────────────────────────────┤
│ paragraph │ This domain is for use in documentation ex │
└──────────────┴────────────────────────────────────────────┘
Core httpfs fetches it, webbed parses it, duck_block_utils types it -- and the same slot takes any filesystem extension, so a caching or object-storage layer drops in without the reader knowing. Ran against the live page; example.com is one paragraph of body content, which is why the outline is a single row.
Two formats in one query, validated by a third extension
markdownwebbedduck_block_utils
LOAD markdown; LOAD webbed; LOAD duck_block_utils;
WITH md AS (SELECT list(b) AS blocks FROM read_markdown_blocks('README.md') b),
html AS (SELECT html_to_duck_blocks('<h1>Title</h1><p>Hi</p>') AS blocks)
SELECT 'markdown' AS source, len(blocks) AS blocks,
(duck_blocks_validate(blocks)).valid AS conforms FROM md
UNION ALL
SELECT 'html', len(blocks), (duck_blocks_validate(blocks)).valid FROM html;
┌──────────┬────────┬──────────┐
│ source │ blocks │ conforms │
├──────────┼────────┼──────────┤
│ markdown │ 1339 │ true │
│ html │ 2 │ true │
└──────────┴────────┴──────────┘
This is the part that makes the suite a suite. Two independently built extensions emit the identical STRUCT, a third validates both against a published spec, and a UNION ALL across them type-checks -- with no dependency between any of the three.
