Part 21: Data formats

Sooner or later a program has to talk to something that is not itself — a web service, a configuration file, another program written in another language. This part reads and writes the two text formats you will meet most: JSON, for exchanging data, and TOML, for configuration. Both packages work the same way: text becomes a tree of values, a program asks each value what it is before taking it apart, and a document that is wrong is refused with the place it went wrong.

Three more lessons cover what travels inside and around such documents: bytes written as text with Base64 and its relatives, the URLs that point at everything, and compressed data in the gzip and zlib formats.

What you will learn

• Parsing JSON into a tree of six kinds of value, and reading typed values out of it through pointers and out-parameters.
• Building a JSON tree by hand, writing it compact or pretty, and checking a round trip.
• Reading JSON as a stream of events, for documents too large to hold at once.
• Reading TOML, whose values have real types: integers, floats, dates and tables.
• Changing a TOML document and writing it back — and what a round trip keeps and loses.
• Reporting a bad document by byte, or by line and column, so a person can find the mistake.
• Writing bytes as hexadecimal, Base64, Base32 or Base58 text, and decoding them back strictly.
• Parsing, normalizing and resolving URLs, and reading and building their queries.
• Compressing data in memory or as a stream, in the raw, zlib and gzip formats.

The part at a glance

flowchart LR
    jt(["JSON text"]) -- "`Parse`" --> jv["`JsonValue` tree"]
    jv -- "`WriteValue`" --> jt
    jt -- "`JsonEventReader`" --> ev["Events, one at a time"]
    tt(["TOML text"]) -- "`TomlParse`" --> tv["`TomlValue` tree"]
    tv -- "`TomlWriteDocument`" --> tt
    jt -- "malformed" --> err["Refused, with<br/>where and why"]
    tt -- "malformed" --> err

Lessons

Before you start

Documents are held in memory from an allocator, and values are read through pointers and out-parameters, all from Part 15: Memory. Text is handled with the String, StringView and StringBuilder of Part 14: Text, trees are move-only values as in Part 11: Ownership, and failures come in error sums from Part 9. The TOML lesson also reads a Date from Part 20: Utilities. The base encodings write into buffers you provide, as writable slices, and Compress streams into a file through the Writer and Reader of Part 19: Files. Each lesson's package is in the Examples repository's DataFormats/ folder:

cd Examples/DataFormats/Json
rux run

After this part

Part 22: Packages steps back from single programs to how code is organised and shared — modules, packages, libraries and the Rux.toml that describes them, which you can now read as the TOML it is.

Before moving on, try the checkpoint project Notes: a to-do list kept as JSON in a file, saved, loaded back, added to, and protected against a damaged or missing file. It puts this part together with Part 19: Files.

For the language rules this part relies on, see Pointers, null pointers and Interfaces in the Rux Reference, and the manifest for the TOML file every package has.