I hate YAML. Instead of writing an essay on why I hate YAML, I can just link to noyaml.com. In my personal projects I will never use it, preferring either JSON, TOML or even plain old INI files depending on the use case. However the ship has sailed already, there are tons of projects everywhere that uses YAML: from most CI systems (GitHub Actions, CircleCI, Travis, et tu builds.sr.ht), to Kubernetes, or in almost every Rails application.
One way to avoid at least some issues with the language is to write YAML in another language. I will show my solution in one of my personal repositories, writing Nix to generate GitHub Actions configuration files. Bonus points for validating the result against the schema of GitHub Actions, so the famous "this is supposed to be string instead of a list of strings" is gone.
Let's start with the basics: YAML is supposed to be a superset of JSON. What that means is that a JSON file can be parsed by a YAML parser. And Nix itself generates JSON natively, after all, Nix can be imagined as "JSON with functions".
To make things easier, I will assume that you have the nix-commands and
flakes enabled as experimental-features in your Nix configuration. If not,
go here.
Using the nix eval command, we can generate a JSON expression from Nix by:
$ nix eval --expr '{ foo = "bar"; }' --json
{"foo":"bar"}
However, typing long excerpts of Nix code inside the console would be
impractical. We can write the following code inside a foo.nix file instead:
{
foo = "bar";
}
And:
$ nix eval --file foo.nix --json
{"foo":"bar"}
While you can use a JSON output as an input for YAML parsers, it is probably
not the best idea. Sadly (or
maybe not), Nix has no native functionality to export data to YAML. However,
since we are using Nix, it is trivial to use nixpkgs to use some program to
convert from JSON to YAML.
To start, let's create a new directory, move our foo.nix file to it, create a
new flake.nix file and put the following contents:
{
description = "Generate YAML files with Nix";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
};
outputs = { nixpkgs, ... }:
{
packages.x86_64-linux =
let
inherit (nixpkgs) lib;
pkgs = import nixpkgs { system = "x86_64-linux"; };
in
{
toYAML = pkgs.runCommand "toYAML" {
buildInputs = with pkgs; [ yj ];
json = builtins.toJSON (import ./go.nix);
passAsFile = [ "json" ]; # will be available as `$jsonPath`
} ''
mkdir -p $out
yj -jy < "$jsonPath" > $out/go.yaml
'';
};
};
}
We are loading the ./foo.nix as a Nix file, converting it to JSON with
builtins.toJSON function, and finally, using pkgs.runCommand and its
passAsFile option to load the contents of the JSON file into
yj, that converts between serialisation
formats (-jy flag means "JSON to YAML"). The reason I choose yj is mostly
because it is a single binary Go program, but you can use whatever you prefer.
By the way, there is a
lib.generators.toYAML
inside nixpkgs.lib, but as of the day of this post it only calls
lib.strings.toJSON (that in turn, calls builtins.toJSON). So it doesn't
really help here. Another option would be pkgs.formats.yaml.generate, that
converts between formats, but it calls
remarshal
(in Python), so not my favorite choice.
If we run the following commands, we can see the result:
$ nix build .#packages.x86_64-linux.toYAML
$ cat result/foo.yaml
foo: bar
That is the basic idea. To have a more realistic example, let's convert the
go.yml,
that builds this blog, to Nix:
{
name = "Go";
on.push.branches = [ "main" ];
jobs = {
build = {
runs-on = "ubuntu-latest";
permissions.contents = "write";
steps = [
{ uses = "actions/checkout@v4"; }
{
name = "Set up Go";
uses = "actions/checkout@v4";
"with".go-version = "1.21";
}
{
name = "Update";
run = "make";
}
{
name = "Publish";
run = "make publish";
env.MATAROA_TOKEN = ''''${{ secrets.MATAROA_TOKEN }}'';
}
{
name = "Commit";
uses = "stefanzweifel/git-auto-commit-action@v5";
"with".commit_message = "README/rss:update";
}
];
};
};
}
Some interesting things to highlight: with is a reserved word in Nix, so we
need to quote it. Not a problem, but something to be aware. And the template
string in GitHub Actions uses the same ${} that Nix uses, so we need to
escape.
And after running the following commands:
$ nix build .#packages.x86_64-linux.toYAML
$ cat result/go.yaml
jobs:
build:
permissions:
contents: write
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Go
uses: actions/checkout@v4
with:
go-version: "1.21"
- name: Update
run: make
- env:
MATAROA_TOKEN: ${{ secrets.MATAROA_TOKEN }}
name: Publish
run: make publish
- name: Commit
uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: README/rss:update
name: Go
"on":
push:
branches:
- main
Yes, the keys are not in the same order as we defined, since Nix, like most programming languages (with the exception of Python), do not guarantee the insertion order in maps/dicts/attrsets/whatever. But I really hope whatever is consuming your YAML is not relying in the order the keys are defined (this would be more cursed than YAML already is).
So that is basically it. For the bonus points that I talked at the start of the
post, we can modify pkgs.runCommand to run some kind of validator. I use
action-validator, one that I
particularly packaged in
nixpkgs to use in those cases.
But you could use e.g.: a validator of Kubernetes YAML. Or a generic YAML lint
like this one. The possibilities are
endless.
Let's modify our flake.nix to add the validation:
{
# ...
outputs = { nixpkgs, ... }:
{
packages.x86_64-linux =
let
inherit (nixpkgs) lib;
pkgs = import nixpkgs { system = "x86_64-linux"; };
in
{
toYAML = pkgs.runCommand "toYAML" {
buildInputs = with pkgs; [ action-validator yj ];
json = builtins.toJSON (import ./go.nix);
passAsFile = [ "json" ];
} ''
mkdir -p $out
yj -jy < "$jsonPath" > $out/go.yaml
action-validator -v $out/go.yaml
'';
};
};
}
And let's add an error in our go.nix file:
diff --git a/go.nix b/go.nix
index 25e0596..8c00033 100644
--- a/go.nix
+++ b/go.nix
@@ -5,7 +5,7 @@
jobs = {
build = {
runs-on = "ubuntu-latest";
- permissions.contents = "write";
+ permissions.contents = [ "write" ];
steps = [
{ uses = "actions/checkout@v4"; }
{
Finally, let's try to build our YAML file again:
$ nix build .#packages.x86_64-linux.toYAML
error: builder for '/nix/store/j8wr6j1pvyf986sf74hqw8k31lvlzac5-toYAML.drv' failed with exit code 1;
last 25 log lines:
> "Additional property 'runs-on' is not allowed",
> ),
> path: "/jobs/build",
> title: "Property conditions are not met",
> },
> Properties {
> code: "properties",
> detail: Some(
> "Additional property 'steps' is not allowed",
> ),
> path: "/jobs/build",
> title: "Property conditions are not met",
> },
> Required {
> code: "required",
> detail: None,
> path: "/jobs/build/uses",
> title: "This property is required",
> },
> ],
> },
> ],
> },
> ],
> }
For full logs, run 'nix log /nix/store/j8wr6j1pvyf986sf74hqw8k31lvlzac5-toYAML.drv'.
Yes, the output of action-validator is awfully verbose, but it is still
better than making "8 commits/push in one
hour".
If you are interested in how a more advantage usage of this technique is,
including usage of functions and constants to share common steps between
different actions, please take a look at the
actions
(permalink)
in my nix-config repository.