Markly Package

Markdown file parsing and management with YAML and TOML frontmatter support.

Main repository: https://gitlab.com/lyoneel/markly Any other host that serves this repository is a mirror. Open issues and merge requests on GitLab.

Overview

The markly package provides tools for reading, parsing, and managing Markdown files with YAML or TOML frontmatter metadata. It supports lazy loading, dependency resolution between files, typed metadata access, and efficient batch operations using the dirly integration.

Key Features:

• Automatic YAML/TOML frontmatter extraction with line number tracking and format detection
• Lazy content loading (content loaded only when explicitly requested)
• ATX-style heading detection with line range information
• Typed metadata access via MDMetadata
• Directory-based file discovery with dependency graph resolution
• Batch read/write operations for efficiency

Supported Frontmatter Formats

The package supports both YAML and TOML frontmatter formats:

Format	Opening Delimiter	Closing Delimiter	Example
YAML	---	---	See below
TOML	+++	+++	See below

YAML Frontmatter

---
name: my-doc
tags:
  - guide
  - tutorial
active: true
---

# Document Title

TOML Frontmatter

+++
name = "my-doc"
tags = ["guide", "tutorial"]
active = true
+++

# Document Title

Note: The package automatically detects the format based on the opening delimiter. Mixed delimiters in code blocks are handled correctly - only the first frontmatter section is parsed as metadata.

Core Types

Type	Purpose
MDFile	A single Markdown file with lazy or eager loading, frontmatter parsing, and body editing
MDMetadata	Typed accessor for frontmatter data with line ranges and format type
MDContent	Parsed body with heading list and raw text
MDHeading	One ATX heading with its line range
MDFolder	Directory loader with dependency resolution, filtering, and batch operations

Basic usage:

// Lazy: metadata parsed, content not loaded
md := markly.NewMDFile("path/to/file.md")
meta := md.GetMetadata()
title := meta.GetString("title")

// Load content on demand
content, err := md.LoadContent()

// Eager: load everything immediately
md, err := markly.NewMDFileWithContent("path/to/file.md")

The complete type documentation, method tables, and loader options live in DEVELOPMENT.md. The generated reference lives on pkg.go.dev.

Dependency Graph

The MDFolder builds a directed acyclic graph (DAG) from metadata dependencies.

Files declare dependencies in frontmatter, in array or string form:

depends_on: ["basics.md", "concurrency.md"]

prerequisites: "basics.md, concurrency.md"

With a custom separator:

requires: "file1.md; file2.md"  // With markly.WithDepSeparator("; ")

Resolution runs in four phases: discovery (paths only), graph building (metadata only), cycle detection (DFS), and topological sort (Kahn's algorithm). Content loads lazily on first access. The full resolution process and performance characteristics live in DEVELOPMENT.md.

Default Dependency Keys

By default, the package checks these field names for dependencies (in order):

1. deps (short form)
2. depends
3. dependencies
4. depends_on
5. prerequisites

Override with WithDepKeys():

NewMDFolder(path, markly.WithDepKeys("my_custom_key"))

Line Number Tracking

All parsing preserves line numbers for error reporting and navigation:

md := markly.NewMDFile("file.md")

// YAML frontmatter line range
meta := md.GetMetadata()
if meta != nil {
    fmt.Printf("YAML from line %d to %d\n", meta.FromLine, meta.ToLine)
}

// Content loading and heading tracking
content, err := md.LoadContent()
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Content starts at line %d\n", md.GetContentFrom())

for _, heading := range content.Headings {
    fmt.Printf("%s at lines %d-%d: %s\n", 
        strings.Repeat("#", heading.Level),
        heading.FromLine, heading.ToLine,
        heading.Text)
}

Error Handling

All operations return errors for proper error handling:

Operation	Possible Errors
NewMDFileWithContent()	File not found (os.IsNotExist), parse errors
GetContent() / LoadContent()	File not found, YAML parse errors, scanner errors (e.g., token too long)
GetMetadata()	Returns nil if file doesn't exist or has no YAML frontmatter
GetLoadOrder()	Circular dependency detected
GetAllWithDependencies()	Circular dependency, file not found

Error Patterns:

md := markly.NewMDFile("nonexistent.md")
meta := md.GetMetadata()  // Returns nil (no error)

content, err := md.LoadContent()
if err != nil {
    if os.IsNotExist(err) {
        // Handle missing file
    } else if strings.Contains(err.Error(), "token too long") {
        // Handle scanner error for very long lines
    }
}

// Eager loading returns error immediately
md, err := markly.NewMDFileWithContent("file.md")
if err != nil {
    log.Fatal(err)
}

Document Manipulation

Beyond parsing, markly can construct documents in memory and mutate their frontmatter and body.

In-Memory Construction

md := markly.NewMDFileFromString(content)        // YAML/TOML + body parsed immediately
md := markly.NewMDFileFromBytes(data)            // same, from bytes

meta, _ := md.GetMetadata()
meta.SetString("title", "New Title")

md.SetPath("/vault/note.md")                     // give it a write target
if err := md.Save(); err != nil {                // direct write
    return err
}
if err := md.SaveAtomic(); err != nil {          // temp file + rename
    return err
}

Save() keeps the classic direct write. SaveAtomic() writes a uniquely named temp file in the target directory and renames it over the target, so a crash mid-write leaves the previous file intact.

Loading Options

All constructors accept options:

md := markly.NewMDFileFromString(content,
    markly.WithRawScalars(),          // keep dates/timestamps as original text
    markly.WithFenceAwareHeadings(),  // skip headings inside ``` / ~~~ fences
)

WithRawScalars() converts !!bool, !!int, !!float, !!null scalars to their natural types but keeps everything else (dates, timestamps, plain strings) as the exact original text, so round-trips never rewrite values.

Metadata Structure Inspection

meta, _ := md.GetMetadata()

data := meta.Data()                       // underlying map[string]any
meta.IsFlowSequence("tags")               // tags: [a, b] -> true
meta.HasBlockScalarSequence()             // tags:\n  - a -> true

Pluggable Frontmatter Serializer

YAML serialization defaults to yaml.Marshal. Register a hook to control key order, list style, and quoting:

md.SetSerializer(func(meta map[string]any) (string, error) {
    // canonical example: sorted keys, hash last
    keys := make([]string, 0, len(meta))
    for k := range meta {
        if k != "hash" {
            keys = append(keys, k)
        }
    }
    sort.Strings(keys)

    var b strings.Builder
    for _, k := range keys {
        fmt.Fprintf(&b, "%s: %v\n", k, meta[k])
    }
    if h, ok := meta["hash"]; ok {
        fmt.Fprintf(&b, "hash: %v\n", h)
    }
    return b.String(), nil
})

The hook applies to YAML documents only; TOML keeps the builtin marshaler.

Body Editing

Body mutations use absolute 1-based file line numbers, so stored anchors point at the same lines a human sees in the file.

md.AppendLine("- [ ] new item")          // append to body
md.InsertLine(5, "inserted line")        // insert before file line 5
md.RemoveLine(7)                         // remove file line 7
md.SetBody("fresh\nbody")                // replace the whole body

First-Level Headings

title, found := md.ExtractFirstH1()      // first "# Title" outside code fences
md.RemoveFirstH1()                       // strip it from the body
md.SetFirstHeading("New Title")          // replace, or insert after frontmatter

Sections

line, found := md.FindSection("my-section")      // slug lookup, file line number
body, found := md.FindSectionBody("my-section")  // slug lookup, section text
md.SetSectionHeading("my-section", "Renamed")    // rename "## My Section"
md.SetSectionBullet("my-section", "status", "done")  // replace/insert "- status: done"
md.InsertLineUnderHeading("My Section", "- [ ] moved", true)  // true: create if missing

FindSectionBody returns the section text up to the next heading of the same or higher level, with leading and trailing blank lines trimmed.

SlugHeading("My Section!") produces the stable slug my-section (lowercase, non-alphanumeric runs collapsed to one hyphen).

Frontmatter Parsing with Errors

GetMetadata treats every parse failure the same way (nil metadata). When callers need to tell the failure modes apart, use the one-shot parse:

meta, body, err := markly.ParseFrontmatter(text)
if errors.Is(err, markly.ErrNoFrontmatter) {
    // no delimiter block at all
}
var fmErr *markly.FrontmatterError
if errors.As(err, &fmErr) {
    // fmErr.Line points at the document line that broke the parse
}

Coercing accessors read YAML int-typed scalars (dates) and decimal strings without changing GetString semantics:

meta.GetCoercedString("date")  // int scalar 20260922 -> "20260922"
meta.GetCoercedInt("count")    // string "42" -> 42

Strict duplicate-key detection is opt-in:

md := markly.NewMDFileFromString(text, markly.WithStrictDuplicates())
// a repeated top-level key fails with *markly.FrontmatterError

Checkboxes

md.SetCheckboxMarker(5, 'x')             // "- [ ] task" -> "- [x] task"
md.RewriteCheckboxTitle(5, "renamed")    // keeps indent, marker, (due: ...),
                                         // and capture-timestamp suffixes

Heading Slug Helper

slug := markly.SlugHeading("Hello World!") // "hello-world"

Folder Discovery Options

folder, _ := markly.NewMDFolder("./vault",
    markly.WithSkipDotDirs(true),   // skip files inside dot-directories
)

files := folder.GetAll()
for path, err := range folder.Errors() {
    // files skipped because their frontmatter failed to parse
    fmt.Printf("%s: %v\n", path, err)
}

Metadata parse errors no longer log; they are collected and exposed through Errors(), keyed by relative path.

Testing

Run tests with race detection:

go test -race ./...

The package includes comprehensive tests covering:

• YAML frontmatter parsing (various formats, edge cases)
• Heading detection and extraction (ATX-style, all levels 1-6)
• Lazy vs eager loading behavior
• Dependency graph building and resolution
• Cycle detection in complex DAGs
• Metadata filtering by key-value pairs
• Typed accessor methods (GetString, GetInt, GetBool, etc.)
• Batch read/write operations
• Line number tracking accuracy
• Real-world scenarios (kaizen structures, boolean fields, mixed arrays)

Example: Complete Workflow

package main

import (
    "fmt"
    "log"
    
    "gitlab.com/lyoneel/markly"
)

func main() {
    // Load all markdown files from directory
    folder, err := markly.NewMDFolder("./docs")
    if err != nil {
        log.Fatal(err)
    }

    // Get files in dependency order
    orderedFiles, err := folder.GetLoadOrder()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Processing %d files in dependency order:\n", len(orderedFiles))
    
    for _, md := range orderedFiles {
        meta := md.GetMetadata()
        
        // Extract metadata
        title := meta.GetString("title")
        tags := meta.GetStringList("tags")
        
        fmt.Printf("\n%s\n", title)

        if len(tags) > 0 {
            fmt.Printf("Tags: %s\n", tags)
        }
        
        // Load content (lazy - only when needed)
        content, err := md.LoadContent()
        if err != nil {
            log.Printf("Error loading %s: %v\n", md.path, err)
            continue
        }
        
        fmt.Printf("Headings: %d\n", len(content.Headings))
        for _, h := range content.Headings {
            fmt.Printf("  - [%d] %s (lines %d-%d)\n", 
                h.Level, h.Text, h.FromLine, h.ToLine)
        }
    }
}

Comparison: Lazy vs Eager Loading

Aspect	Lazy Loading (NewMDFile)	Eager Loading (NewMDFileWithContent)
Metadata parsing	Yes (on GetMetadata() call)	Yes (immediate)
Content loading	No (explicit via LoadContent())	Yes (immediate)
First access cost	Low (metadata only)	High (full file parse)
Memory usage	Minimal until content loaded	Full file in memory immediately
Use case	Directory scanning, metadata filtering	Single-file processing, immediate content needs

Recommendation: Use lazy loading for directory operations (MDFolder), eager loading when you need content immediately.

Schema Validation

The validate sub-package checks frontmatter against a schema file: a markdown document whose frontmatter carries a fields map and an optional config map of format constants.

fields:
  title: {required: true, shape: string}
  status: {required: true, shape: single, values: [draft, done]}
  tags: {required: false, shape: list, from: tag-vocabulary}
  sources: {required: true, shape: objects, keys: [name, url]}
  created: {required: true, shape: date}

schema, err := validate.LoadSchema("schema.md")
if err != nil {
    return err
}
issues := schema.ValidateDocument("note.md", data)   // one document
issues, err = schema.ValidateDir("docs/")            // file, directory, or glob

Each issue carries File, Field, and Reason. Generic shapes cover string, single, list, objects, object, case, date, number, and boolean. from references resolve at load time to the controlled vocabulary files next to the schema. Domain-specific shapes register through ShapeFunc and receive the schema's Ctx value, so domain semantics stay caller-side:

schema.Ctx = myIndex
schema.RegisterShape("game", func(ctx any, field validate.Field, value any) []validate.Issue {
    // custom validation over ctx
    return nil
})

Inline Style Rules

A field declaring inline: true pins the YAML style of its value. ValidateDocument sees only the decoded data, where style is lost, so the style checks run through ValidateMetadata on the parsed metadata:

Shape	inline: true means
list, objects	the sequence must be a flow list: tags: [a, b]
object, case	the mapping must be a flow map: meta: {id: 1}
string	no block scalars: | and > are rejected

meta, _, err := markly.ParseFrontmatter(text)
if err != nil {
    return err
}
issues := schema.ValidateMetadata("note.md", meta)

The style checks apply to YAML frontmatter only; TOML has no block style and always satisfies the rule.

License

MIT License - See LICENSE file for details.